This article is a continuation of the previous Transition to OpenSource. In the previous publication, we explored our transition from Graylog to OpenSearch, highlighting the reasons behind this transition. In this article, we will further delve into the capabilities of OpenSearch, specifically focusing on its ability to integrate local or remote machine learning systems using the OpenSearch Assistant Module. This evolution represents a significant step in the journey towards advanced data management and offers opportunities to enhance analysis and information extraction process from our logs.

What it is

Generative AI is transforming how users interact with and derive insights from their data.

OpenSearch has supported the k-NN vector database since its inception, which is crucial in a wide range of applications, including similarity search, classification, clustering, and time series analysis.

For example, it allows:

• Conducting similarity searches, such as finding documents with content similar to a specific document
• Automatically classifying documents based on their vector contents
• Grouping documents based on their vector similarity (clustering) to simplify data analysis and understanding
• Analyzing time series and identifying trends and patterns in data

OpenSearch has also introduced the concept of Piped Processing Language (PPL), which allows users to concatenate various operations to manipulate data to obtain complex results.

The OpenSearch Assistant Module is a framework to orchestrate these different technologies to create AI assistants directly within OpenSearch Dashboards. It includes “skills” for specific tasks, an ML framework to integrate AI models, and UI components for conversational interactions.

These skills can be connected to an LLM to generate summaries from query results, while ensuring an intuitive and interactive user experience of OpenSearch Dashboards was as simple as incorporating the UI search bar component into our log exploration interface.

Supported models can be local or external. In the case of local models, you can use pre-trained models or customize them to fit your use case, considering local hardware limitations. However, you can also opt for external models, such as ChatGPT 3.5, or others.

For details on the integration of OpenSearch Assistant: RFC – OpenSearch Assistant Toolkit

How it works

When details become too complex, images of monkeys with tambourines spinning in circles form in our minds. Imagine being able to interact with data posing questions in natural language instead of regex. Thanks to LLMs, it is possible to simplify the analysis process, make it as interactive as we’ve been accustomed to since elementary school, and focus on results.

LLMs represent a turning point in the evolution of machines towards greater flexibility. Although they don’t always produce the perfect answer, this imprecision encourages users to interact and refine requests. it is an iterative process that sometimes takes time but ultimately yields more relevant and useful results.

Among the dedicated “skills” that can be combined, one is particularly focused on converting the user-provided natural language prompt into a PPL query. Thanks to this, the days of mastering regex to manipulate data may be over with the advent of LLMs. Now, you can ask them to automatically generate the necessary formulas to filter and synthesize data. For the user, the OpenSearch Assistant module appears as a field where any question about the data can be entered. In the background, questions are interpreted and converted into PPL, allowing not only for answers but also useful suggestions to further refine queries.

Let’s see some simple examples, but the advice is to download a docker-compose.yml file and run the cluster in your infrastructure with your data.

Show me possible problems.

We can start with a generic request to show the issues. Determining what to classify as a problem is the task of generic or customized “skills.”

Charting

Similarly, we can request to visualize the data.

Enriching

The most interesting and promising aspect, considering the possible integrations that can be made, is the ability to add context to our data.

Summary

We examined OpenSearch’s evolution towards advanced data management and analysis, focusing on integrating machine learning capabilities. We explored how OpenSearch’s support for k-NN vector databases and the introduction of Piped Processing Language (PPL) enhance data manipulation efficiency.

The result is the OpenSearch Assistant Module, which facilitates the creation of AI assistants within OpenSearch Dashboards and the integration of local or external AI models. We also explored examples of Language Model Models (LLMs) simplifying data analysis through natural language interactions, despite occasional imprecision.

The apparent simplicity of a prompt masks the complex integration of various technologies and methodologies since the inception of OpenSearch, including support for vector databases and the implementation of straightforward yet interconnectable “skills”. Moreover, the integration of increasingly refined and compact pre-trained LLM models adds depth to the development of the OpenSearch Assistant Module.

About the Author

Rocco Gagliardi has been working in IT since the 1980s and specialized in IT security in the 1990s. His main focus lies in security frameworks, network routing, firewalling and log management.

Links

• https://github.com/opensearch-project/dashboards-assistant/issues/18