Python: GPT4o function calling occasionally buggy?

[bbence84]

Describe the bug
Using GPT4 model version 2024-05-13 on Azure (Sweden) with Semantic Kernel, Python.
I have noticed that sometimes it seems that GPT4o (os is it in SK?) gets the toolname wrong, e.g. I am getting error:

{'role': 'tool', 'content': 'The tool call with name cc_code.MigrateCodeToCalc is not part of the provided tools, please try again with a supplied tool call name and make sure to validate the name.'

However in fact in the tools list in the JSON I have the function:
'function': {'name': 'cc_code-MigrateCodeToCalc', ...

Is it that GPT4o sometimes puts a dot instead of a dash in the function name?

Interestingly, after such an error, I am getting the below error message too:
"Invalid \'messages[5].tool_calls[0].function.name\': string does not match pattern. Expected a string that matches the pattern \'^[a-zA-Z0-9_-]+$\'.", \'type\': \'invalid_request_error\', \'param\': \'messages[5].tool_calls[0].function.name\', \'code\': \'invalid_value\'

Seems that this happens more often with a specific function name, but not 100% of the time, but most of the time.

To Reproduce

Expected behavior
Function call works all the time.

Platform
Python, SK 1.5.1

[moonbox3]

@bbence84, thanks for reporting. This is model related. We've tried to put some guardrails in place inside of SK to help prevent this type of issue (as you can see, we're telling the model to try again with the correct tool name...). We have heard of this a bit more these days. We'll have another look to see if there are some further improvements we can make to make tool calling more robust.

[moonbox3]

related to #7968

[bbence84]

Thanks! I am not sure anymore, if it's model related, at least now I have switched to GPT4 32K (also on Azure, Sweden), and I am still getting the error randomly. Actually it seems to be slightly different, based on the error message:

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[5].tool_calls[0].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.", 'type': 'invalid_request_error', 'param': 'messages[5].tool_calls[0].function.name', 'code': 'invalid_value'}}

Where does the tool name "messages[5].tool_calls[0].function.name" come from? Maybe something is not getting replaced properly inside SK?

Thanks!

[moonbox3]

Thanks! I am not sure anymore, if it's model related, at least now I have switched to GPT4 32K (also on Azure, Sweden), and I am still getting the error randomly. Actually it seems to be slightly different, based on the error message:

openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[5].tool_calls[0].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.", 'type': 'invalid_request_error', 'param': 'messages[5].tool_calls[0].function.name', 'code': 'invalid_value'}}

Where does the tool name "messages[5].tool_calls[0].function.name" come from? Maybe something is not getting replaced properly inside SK?

Thanks!

That's the same error. The model is sending back the function.name like plugin.function instead of plugin-function.

[bbence84]

Is there any update on this? I mean is this an LLM or a framework (SK) issue? It did not seem to happen in the past, so either a recent SK version changed something or Microsoft / OpenAI did something on their side to provoke this error.

[moonbox3]

Is there any update on this? I mean is this an LLM or a framework (SK) issue? It did not seem to happen in the past, so either a recent SK version changed something or Microsoft / OpenAI did something on their side to provoke this error.

This is not entirely an SK issue. It's most likely a model issue where OpenAI is continuing to train with a . delimiter, and so it more often produces a result with something like plugin.function instead of plugin-function. We're still tracking this on our end to make improvements, but it may take a little longer as these improvements should be reflected across all three SK languages.

[yashworlikar]

Facing the same issue in the dotnet version, Was there any reason behind choosing - instead of . as the delimiter?

[Cobra86]

I'm having the same issue with new dotnet SK 1.21

[tfsjohan]

Same issue for me in the dotnet SDK. Very random. Both with 4o and 4o-mini.

[moonbox3]

Hi there. We've been seeing this issue pop up more and more recently. It appears to be a bug in the model. A way to mitigate this is to set parallel_tool_calls = False. We've released a SK Python version 1.13.0 that allows one to configure this attribute on the OpenAIChatPromptExecution/AzureChatPromptExecution settings.

The ability to do toggle this setting in .Net will land soon.

[moonbox3]

As mentioned above, this should be handled by toggling parallel_tool_calls = False.

[bbence84]

@moonbox3 Hm, this issue seems to be back even with parallel_tool_calls off. :(
I am using Azure, API version 2024-02-15-preview, gpt-4o, model version 2024-05-13, swedencentral.

With the latest Python SDK (1.14.0), I am still getting:

2024-11-13 09:18:00 - HTTP Request: POST https://ibp-aoai.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-02-01 "HTTP/1.1 400 model_error"
2024-11-13 09:18:00 - Function failed. Error: Error occurred while invoking function chat: ("<class 'semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion.AzureChatCompletion'> service failed to complete the prompt", BadRequestError('Error code: 400 - {\'error\': {\'message\': "Invalid \'messages[5].tool_calls[0].function.name\': string does not match pattern. Expected a string that matches the pattern \'^[a-zA-Z0-9_-]+$\'.", \'type\': \'invalid_request_error\', \'param\': \'messages[5].tool_calls[0].function.name\', \'code\': \'invalid_value\'}}'))
2024-11-13 09:18:00 - Function completed. Duration: 2.647342s
2024-11-13 09:18:00 - Something went wrong in function invocation. During function invocation: 'chat-chat'. Error description: 'Error occurred while invoking function chat: ("<class 'semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion.AzureChatCompletion'> service failed to complete the prompt", BadRequestError('Error code: 400 - {\'error\': {\'message\': "Invalid \'messages[5].tool_calls[0].function.name\': string does not match pattern. Expected a string that matches the pattern \'^[a-zA-Z0-9_-]+$\'.", \'type\': \'invalid_request_error\', \'param\': \'messages[5].tool_calls[0].function.name\', \'code\': \'invalid_value\'}}'))'
2024-11-13 09:18:00 - Error occurred

This is how I init the kernel, maybe it's wrong?

    kernel = Kernel(logger=logger)  

    service_id = "assistant-chat"
    chat_service = AzureChatCompletion(
        service_id=service_id, env_file_path=".env"
    )

    kernel.add_service(chat_service)

    req_settings = kernel.get_prompt_execution_settings_from_service_id(service_id=service_id)
    req_settings.max_tokens = 4096
    req_settings.temperature = 0.0
    req_settings.function_choice_behavior = FunctionChoiceBehavior.Auto(
        filters={"excluded_plugins": ["chat"]}
    )
    req_settings.parallel_tool_calls = False

    chat_function = kernel.add_function(
        prompt="""{{$chat_history}}{{$user_input}}""",
        function_name="chat",
        plugin_name="chat",
        description="Main chat function",
        prompt_execution_settings=req_settings,
    )