feat: Add function to Service to allow stream the results back from a…

[qbm5]

Add function to Service to allow stream the results back from a completion request

[kayhantolga]

@qbm5 when I test the method, it fetches the data as a stream but it waits until the stream completes. And it processes all of it at once. So I can't get the advantage of streaming data. Does it work for you too like that?

[qbm5]

I might not be understanding the stream correctly. When I walk through it piece by piece (see attached) it seems to be working correctly, but you are right it appears to come all at once when letting it run without break points.

[kayhantolga]

Yes, I had the same experience. The issue is HTTP client is waiting until Stream completes. I will search a bit for samples. without yield return, the method will be useless.

[qbm5]

I just had success with doing it this way. It is sloppy, but it is showing promise. I can work on getting it in the pr a bit later.

` public async IAsyncEnumerable CreateCompletionAsStream(CompletionCreateRequest createCompletionRequest, string? modelId = null)
{
// Mark the request as streaming and include a logit bias
createCompletionRequest.Stream = true;

    // Send the request to the CompletionCreate endpoint
    createCompletionRequest.ProcessModelId(modelId, _defaultModelId);

    var ser = new JsonSerializerOptions()
    {
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingDefault
    };
    JsonContent content = JsonContent.Create(createCompletionRequest, mediaType: null, ser);

    using HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, _endpointProvider.CompletionCreate());
    request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("text/event-stream"));
    request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
    request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
    request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("br"));
    request.Content = content;


    // Read the response as a stream
    ;
    using (HttpResponseMessage response = _httpClient.Send(request, HttpCompletionOption.ResponseHeadersRead))
    using (var stream = response.Content.ReadAsStream())
    {
        

        using var reader = new StreamReader(stream);
        // Continuously read the stream until the end of it
        while (!reader.EndOfStream)
        {
            var line = await reader.ReadLineAsync();
            // Skip empty lines
            if (string.IsNullOrEmpty(line)) continue;

            line = line.Replace("data: ", string.Empty);

            // Exit the loop if the stream is done
            if (line.Contains("[DONE]")) break;

            CompletionCreateResponse? block;
            try
            {
                // When the response is good, each line is a serializable CompletionCreateRequest
                block = JsonSerializer.Deserialize<CompletionCreateResponse>(line);
            }
            catch (Exception)
            {
                // When the API returns an error, it does not come back as a block, it returns a single character of text ("{").
                // In this instance, read through the rest of the response, which should be a complete object to parse.
                line += await reader.ReadToEndAsync();
                block = JsonSerializer.Deserialize<CompletionCreateResponse>(line);
            }
            if (null != block) yield return block;
        }

    }`

[qbm5]

I have pushed the update

[kayhantolga]

Awesome! I pushed some changes too, If you approve I am going to merge it.