Missing stream=True Parameter in _StreamGenerateContent Class of GenerativeServiceRestTransport Causes Non-Streaming Requests

[jbndr]

Description of the bug:

In the _StreamGenerateContent class of the GenerativeServiceRestTransport implementation for the Google AI Generative Language service, the HTTP request intended for streaming does not include the stream=True parameter. This omission results in non-streaming behavior, contrary to the expected functionality.

Actual vs expected behavior:

Actual Behavior:
The _StreamGenerateContent class makes a non-streaming HTTP request using the REST transport.

Expected Behavior:
The _StreamGenerateContent class should make a streaming HTTP request using the REST transport.

Any other information you'd like to share?

Code Location:
/google/ai/generativelanguage_v1beta/services/generative_service/transports/rest.py

Current Code Snippet:

# Send the request
headers = dict(metadata)
headers["Content-Type"] = "application/json"
response = getattr(self._session, method)(
    "{host}{uri}".format(host=self._host, uri=uri),
    timeout=timeout,
    headers=headers,
    params=rest_helpers.flatten_query_params(query_params, strict=True),
    data=body,
    # Missing stream=True parameter
)

Proposed Fix:
Add the stream=True parameter to the request call:

# Send the request
headers = dict(metadata)
headers["Content-Type"] = "application/json"
response = getattr(self._session, method)(
    "{host}{uri}".format(host=self._host, uri=uri),
    timeout=timeout,
    headers=headers,
    params=rest_helpers.flatten_query_params(query_params, strict=True),
    data=body,
    stream=True # Added stream parameter
)

[jbndr]

Please let me know if there is another repository that would be more appropriate for suggesting this fix.

[MarkDaoust]

Hi,

Maybe you're misunderstanding this code. The stream=True parameter only exists in the google.generativeai SDK. It doesn't exist in the underlying client library, or in the API itself.

The SDK converts generate_content(..., stream=True) calls into stream_generate_content(...) calls.

This is working as expected:

for chunk in model.generate_content("Tell me a story abut a cat that can talk.", stream=True):
    print(chunk.text)
    print('-'*80)

If you raised this error because in Colab it waits till all the chunks are available before yielding the first one, that's an issue with Colab, not these libraries.

[jbndr]

Hi @MarkDaoust ,

thank you for your quick reply!

I am aware of your example from: https://github.com/google-gemini/cookbook/blob/main/quickstarts/Streaming.ipynb
This also works for me locally with the SDK, but the GenerativeModel class defaults to using gRPC as the transport instead of the REST I need to use.

I am referring to the stream parameter of the requests library that is used to make the call within the REST transport implementation GenerativeServiceRestTransport, more specifically the _StreamGenerateContent class within it. I understand that this is not technically part of this repository, but I have not found the corresponding repository publicly.

getattr(self._session, method) calls this method https://requests.readthedocs.io/en/latest/api/#requests.Session.request without the stream parameter, which means that the chunks are only output after everything has been loaded.

[MarkDaoust]

Oh, I got it now. Thanks for clarifying.

Yeah, you're right and that totally fixes it, thanks!

The required change is in the generated client library, google.ai.generativelanguage to really fix this we'll need to update the code-generator.

[MarkDaoust]

I can't fix this here, so I've reported the problem over on the gapic-generator-python repository that generates this code:

googleapis/gapic-generator-python#2076

[jbndr]

Thanks!

[MarkDaoust]

Okay, they've merged that into the client library generator. So this will actually be fixed with the next release of google-ai-generativelanguage, (whenever that is).

https://pypi.org/project/google-ai-generativelanguage/#history

It should work in anything after 6.10