Create Sherlock example for LLM agents pipeline

[mdemoret-nv]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

As part of the Sherlock work, an example showing how to use Morpheus to execute multiple LLM agents which are able to use tools and execute multiple LLM queries in order to solve a complex task.

Describe your ideal solution

Purpose

The purpose of this example is to illustrate how a user could build a pipeline that will integrate an LLM agent into a Morpheus pipeline. This example builds on the previous two examples, #1298 and #1306, by adding the ability to execute multiple LLM queries as part of a chain and also to augment those queries with external information if needed.

Using LLM agents allows the LLMs to be able to execute "tools" that can help them solve their task. For example, if your question was "What is Leonardo DiCaprio's girlfriends age raised to the 0.43 power?", the LLM might not know who Leonardo DiCaprio's girlfriend is, but it will know it can find that out from a quick search. If you perform the internet search for the LLM and ask it the question again with the result from the internet search, it will have the information necessary to properly respond.

Scenario

This example will show two different implementations of an agent pipeline but the pipeline and components could be used in many scenarios with different requirements. At a high level, the following illustrates different customization points for this pipeline and the specific choices made for this example:

• LLM Service
  • This pipeline could support any type of LLM service which is compatible with our LLMService interface.
  • Such services include OpenAI, NeMo, or even running locally using llama-cpp-python
  • For this example, we will focus on using OpenAI as the LLM service. This service has been chosen to be different from the previous example which uses NeMo. Additionally, the models in OpenAI have been trained to give responses that work well in the ReAct agent architecture.
• Agent type
  • There are many different styles of agents that are available today. Each one of these agent types alters the pattern for asking the LLM what the next task is, running that operation, and then asking the LLM what to do next. That repeats until the LLM is satisfied with the output.
  • For this example, we will be using the ReAct agent type. This type of agent is the most popular and has proven to work well and be reliable.
• Agent tools
  • Depending on the problem trying to be solved, there are many different tools that can be made available to the LLM agent such as an internet search, VDB retrievers, calculators, Wikipedia, etc.
  • For this example, we will be using the internet search tool and an llm-math tool. This allows the LLM agent to execute arbitrary Google searches and to perform math equations if necessary
• LLM Library
  • Most of the popular LLM libraries, Langchain, llama-index, and Haystack, have the ability to run LLM agents to execute these higher level tasks.
  • For this example, we will be using the Langchain library to provide all of our LLM agents. This allows us to run agents directly in a Morpheus pipeline reducing the overhead of migrating an existing system to Morpheus, and also eliminates the need to replicate the work already done by these libraries.

Implementation

This example will be composed of two different click commands.

Simple Morpheus agent pipeline

The simple Morpheus pipeline is built using the following components:

1. An InMemorySourceStage to hold the LLM queries in a DataFrame
2. A DeserializationStage to convert the MessageMeta objects into ControlMessages needed by the LLMEngine
   1. New functionality was added to the DeserializeStage to support ControlMessages and add a default task to each message.
3. A LLMEngineStage then wraps the core LLMEngine functionality
   1. An ExtracterNode pulls the questions out of the DataFrame
   2. A LangChainAgentNode runs the Langchain agent executor for all provided input. This node will utilize the agents arun interface to run the agents asynchronously.
   3. Finally, the responses are put back into the ControlMessage using a SimpleTaskHandler
4. The pipeline concludes with an InMemorySink stage to store the results.

Kafka Morpheus pipeline

The Kafka Morpheus agent pipeline is functionally similar to the simple agent pipeline, however it uses a Kafka message bus to send messages to the pipeline instead of an in memory source. The reason we are using a Kafka message bus is to show the capabilities of the async LLMEngine functions running inside of a Morpheus pipeline. Normally, when running Langchain agents using the arun interface, they will all run simultaneously. But the Langchain system cannot handle new requests that come in while the current batch is running. In Morpheus, when new messages are sent to the pipeline while existing ones are processing, they will immediately begin processing increasing the parallelism. This helps show why running these LLM agents in a pipeline is advantageous over a normal python library.

The major differences between the simple Morpheus agent pipeline and the Kafka Morpheus agent pipeline are:

• The source for the pipeline is a KafkaSourceStage to pull messages off of the bus as they are created
• Message responses are either printed directly to the console or output to another Kakfa topic.

Completion Criteria

The following items need to be satisfied to consider this issue complete:

• A README.md containing information on the following:
  • Background information about the problem at hand
  • Information about the specific implementation and reasoning behind design decisions (use content from this issue)
  • Step-by-step instructions for running the following:
    • How to run the Simple Morpheus agent pipeline
      • Including Information on how to get an OpenAI API key and what environment variable to set.
      • Including information on how to get a Serpapi.com API key and what environment variable to set
      • The command to run the Morpheus pipeline
    • How to run the Kafka Morpheus agent pipeline
      • Including instructions on how to set up the Kafka cluster for sending messages
      • Including Information on how to get an OpenAI API key and what environment variable to set.
      • Including information on how to get a Serpapi.com API key and what environment variable to set
      • Including instructions on how to view the output Kafka topic where responses will be placed
      • The command to run the Morpheus pipeline
  • The README.md should be linked in the developer docs
• A functioning Simple Morpheus agent pipeline command which satisfies the following:
  • Should run without error using all default arguments
  • Correctly run the LLM agents using the OpenAI chat model
  • Provide information about the success or failure of the pipeline. Including the number of queries run, throughput and total runtime.
• A functioning Kafka Morpheus agent pipeline command which satisfies the following:
  • Should run without error using all default arguments
  • Correctly run the LLM agents using the OpenAI chat model
  • Output the responses to the console or to a separate output Kafka topic
  • Provide information about the success or failure of the pipeline. Including the number of queries run, throughput and total runtime.
• Tests should be added which include the following
  • Test successfully running the Simple Morpheus agent pipeline
  • Test successfully running the Kafka Morpheus agent pipeline

Dependent Issues

The following issues should be resolved before this can be completed:

Tasks

Beta Give feedback

1. [FEA]: Create LLM Engine Core Functionality #1178
   feature request sherlock +2
2. [FEA]: Create Templating Prompt Generator for the LLM Engine #1179
   feature request sherlock +2
   
3. Add documentation and tests for the ExtractorNode #1277
   4 of 4
   sherlock +1
   
4. Add documentation and tests for the LLMEngineStage #1278
   4 of 4
   sherlock +1
   
5. Add documentation and tests for the LangChainAgentNode #1299
   4 of 4
   sherlock +1
   
6. Add documentation and tests for the SimpleTaskHandler #1302
   4 of 4
   sherlock +1
   
7. Add documentation and tests for the LLMService #1303
   4 of 4
   sherlock +1
   

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[jarmak-nv]

Closing as it's complete - I think the auto-close PR linkage stuff only works when it's going into the default branch.