Swarm: Support for nested chat as a hand off

[marklysze]

Why are these changes needed?

Extending the Swarm functionality by allowing a hand off to run a nested chat (a series of sequential chats, as per the existing nested chat functionality). The result of the nested chat will be a string (from the output of the last chat within the nested chat)

To add a nested chat hand off to an agent:

my_swarm_agent.register_hand_off(
    hand_to=[
        ON_CONDITION(
            target={
                "chat_queue":[nested_chat_one, nested_chat_two],
                "reply_func_from_nested_chats": None,
                "config": Any,
                "use_async": False},
            condition="After customer verification, transfer to refund specialist."
        )
    ]
)

Where within the nested_chat dictionary:

Key	Description
chat_queue	list of chats, as per standard nested chats
reply_func_from_nested_chats	optional Callable to be run instead of ConversableAgent.summary_from_nested_chats
config	same as you would pass in to register_nested_chats
use_async	same as you would pass in to register_nested_chats

If reply_func_from_nested_chats is a Callable, it must have a signature of:
def my_reply_func(chat_queue: List[Dict[str, Any]], recipient: Agent, messages: Union[str, Callable], sender: Agent, config: Any) -> Tuple[bool, Union[str, None]]:

Carryover
Additionally, you can push some context from the swarm's chat into the nested chat by way of incorporating a summary of the chat into the first nested chat's message. The configuration is similar to what is used when registering a nested chat for summarising the chat.

This is done via a carryover_config that you add to the first nested chat, e.g.:

carryover_summarize_chat_config = {
    "summary_method": "reflection_with_llm", # "all", "last_msg", "reflection_with_llm", Callable
    "summary_args": None
    }

nested_chat_list = [
    {
        "recipient": nested_agent_one,
        "message": "Summarise the conversation into a few key words",
        "max_turns": 1,
        "summary_method": "last_msg",
        "carryover_config": carryover_summarize_chat_config,
    },
    {
        "recipient": nested_agent_two,
        "message": "Write a poem about it.",
        "max_turns": 1,
        "summary_method": "last_msg",
    },
]

Where within the carryover_config dictionary:

Key	Description
summary_method	"last_msg", "all", "reflection_with_llm", or Callable.
summary_args	Dictionary of arguments to be passed to the summary_method when it is "reflection_with_llm" or a callable. For example, you can use "summary_prompt" to set the prompt for the LLM.

The Callable must have a signature of:
def my_summary_method_func(agent: ConversableAgent, messages: List[Dict[str, Any]], summary_args: Dict[str, Any]) -> str:

The end result of the carryover is that the starting nested chat will have a message that's a concatenation of the message set in the first chat and the swarm's chat. For example the final first message in the example code above could be:
Summarise the conversation into a few key words\nContext:\nDucks are yellow\nDogs are blue\nCats are green.

If no carryover_config is set in the first nested chat, the nested chats will not carry through any messages from the swarm chat.

Examples:

# Sample Carryover summary method
def my_summary_method_func(agent: ConversableAgent, messages: List[Dict[str, Any]], summary_args: Dict) -> str:
    return "Ducks are yellow\nDogs are blue\nCats are green."

# Sample carryover_config using the Callable
my_carryover_config = {
    "summary_method": my_summary_method_func,
    }

# ... or with reflection
my_carryover_config = {
    "summary_method": "reflection_with_llm",
    "summary_args": {
        "summary_prompt": "Summarise the chat into one paragraph.,
        },
    }

# ... or by concatenating message history content
my_carryover_config = {
    "summary_method": "all",  # "<first nested chat "message">\nContext:\n<first swarm chat message>\n<second...
    }

# ... or by concatenating just the last message content
my_carryover_config = {
    "summary_method": "all",  # "<first nested chat "message">\nContext:\n<last swarm chat message>
    }

# Setup nested chat list with the only difference being the addition of "carryover_config"
nested_chat_list = [
    {
        "recipient": nested_agent_one,
        "clear_history": False,
        "message": "Do something important with the following context.",
        "max_turns": 1,
        "summary_method": "reflection_with_llm",
        "summary_prompt": "Summarise the conversation into 3 key words.",
        "carryover_config": my_carryover_config
    },
    {
        "recipient": nested_agent_two,
        "message": "Write a poem about it.",
        "max_turns": 1,
        "summary_method": "last_msg",
    },
]

# Finally, register it in a hand off

customer_service.register_hand_off(
    hand_to=[
        ON_CONDITION(
            target={
                "chat_queue":[nested_chat_one, nested_chat_two],
                "reply_func_from_nested_chats": None,
                "config": Any,
                "use_async": False},
            condition="After customer verification, transfer to refund specialist."
        ),
        ...
    ]
)

Related issue number

#26 New interface

Checks

• I've included any doc changes needed for https://ag2ai.github.io/ag2/. See https://ag2ai.github.io/ag2/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[sonichi]

Alternative proposal:

summarize_chat_config = {"summary_method": ..., "summary_args": ...)
chat1 = {"carryover": summarize_chat_config}
nested_chat_list = [chat1, chat2, ...]

Then, modify _summary_from_nested_chats to support "carryover" as a dict for the first chat in the list, in which case the _summarize_chat method will be called with the given config as the new carryover before initiate_chats.
This will introduce least amount of change I think, and make the existing nested chat to have same capability of reusing the same summary methods we already support.

[marklysze]

Alternative proposal:

summarize_chat_config = {"summary_method": ..., "summary_args": ...)
chat1 = {"carryover": summarize_chat_config}
nested_chat_list = [chat1, chat2, ...]

Then, modify _summary_from_nested_chats to support "carryover" as a dict for the first chat in the list, in which case the _summarize_chat method will be called with the given config as the new carryover before initiate_chats. This will introduce least amount of change I think, and make the existing nested chat to have same capability of reusing the same summary methods we already support.

For swarm, the callable method (starting_message_method=my_function) takes in the context_variables, which I don't think we can handle from outside swarm at the moment?

[sonichi]

Alternative proposal:

summarize_chat_config = {"summary_method": ..., "summary_args": ...)
chat1 = {"carryover": summarize_chat_config}
nested_chat_list = [chat1, chat2, ...]

Then, modify _summary_from_nested_chats to support "carryover" as a dict for the first chat in the list, in which case the _summarize_chat method will be called with the given config as the new carryover before initiate_chats. This will introduce least amount of change I think, and make the existing nested chat to have same capability of reusing the same summary methods we already support.

For swarm, the callable method (starting_message_method=my_function) takes in the context_variables, which I don't think we can handle from outside swarm at the moment?

Good point. Can we then override the _summary_from_nested_chats for SwarmAgent for now? In future we can upstream this change to ConversableAgent.

[marklysze]

For swarm, the callable method (starting_message_method=my_function) takes in the context_variables, which I don't think we can handle from outside swarm at the moment?

Good point. Can we then override the _summary_from_nested_chats for SwarmAgent for now? In future we can upstream this change to ConversableAgent.

Okay, I'll have a look into that!

[marklysze]

Have added to documentation

[marklysze]

@yiranwu0, @sonichi - are we good to go?

[qingyun-wu]

Thanks @marklysze! It seems swarm test is not added to github workflow. We may want to add it later!