.Net: Bug: Token usage missing for Azure OpenAI Streaming Completion

[lukasz-appstream]

Bug summary
When I use Azure OpenAI I don't get Usage in metadata any of the chunks.

Technical Details
SemanticKernel: 1.24.1 (.NET)
Model: gpt-4o
Azure OpenAI Api version: 2024-10-01-PREVIEW

Bug Description
I checked any chunk and I haven't got Usage in any of them when using chat completion with streaming and from what I've read - Usage should be there for Azure OpenAI Service in this API.

[moonbox3]

Hi @lukasz-appstream, thanks for filing the issue. For the streaming API, the usage is included in the last chunk. @markwallace-microsoft / @dmytrostruk for visibility, just in case.

[lukasz-appstream]

Thanks @moonbox3 for the response! Yeah, I thought so, I updated the library, I updated the api version, I saved the last message and it still had metadata?["Usage"] == null... Does it actually work with Azure OpenAI Connector, or for now only for OpenAI Connector?

[JadynWong]

While the new api version supports stream token usage, the azure openai sdk still does not support it at this time. openai sdk has always supported it for raw openai, but SK has not implemented it yet.
Streaming token usage added to OpenAI in #9022

see #9005

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk%2Fopenai%2FAzure.AI.OpenAI%2Fsrc%2FCustom%2FChat%2FAzureChatClient.cs#L82

[RogerBarreto]

@JadynWong thanks for adding the context to this issue.

This issue will be blocked until Azure SDK provides support for getting the usage.

For context we are tracking those issues in Azure SDK repo

• [BUG] StreamOptions is always set to null - it makes it impossible to get token usage with streaming mode Azure/azure-sdk-for-net#46407
• [FEATURE REQ] Azure.AI.OpenAI include_usage in chat completion options Azure/azure-sdk-for-net#44237

[JadynWong]

2.1.0-beta.2 has been released, but with some limitations.

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk%2Fopenai%2FAzure.AI.OpenAI%2Fsrc%2FCustom%2FChat%2FAzureChatClient.cs#L82-L89

openai/openai-dotnet#266

[lukasz-appstream]

@RogerBarreto I'm not sure I follow. So you are saying that this is now completed and we can get token usage for streaming?

[lukasz-appstream]

@RogerBarreto nevermind ;) tested it, works, last chunk has the usage, thanks!

[DIDPT]

@RogerBarreto

Is there a Timeline for a solution that the Token Usage can be reported when using Streaming and OnYourData?

/**
* As of 2024-09-01-preview, stream_options support for include_usage (which reports token usage while streaming)
* is conditionally supported:
* - When using On Your Data (non-null data_sources), stream_options is not considered valid
* - When using image input (any content part of "image" type), stream_options is not considered valid
* - Otherwise, stream_options can be defaulted to enabled per parity surface.
*/
private static void PostfixClearStreamOptions(IEnumerable messages, ref ChatCompletionOptions options)
{
if (AdditionalPropertyHelpers
.GetAdditionalListProperty(options?.SerializedAdditionalRawData, "data_sources")?.Count > 0
|| messages?.Any(
message => message?.Content?.Any(
contentPart => contentPart?.Kind == ChatMessageContentPartKind.Image) == true)
== true)
{
options ??= new();
options.StreamOptions = null;
}
}