Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming over rest doesn't actually stream. #2076

Closed
MarkDaoust opened this issue Jul 26, 2024 · 0 comments · Fixed by #2204
Closed

Streaming over rest doesn't actually stream. #2076

MarkDaoust opened this issue Jul 26, 2024 · 0 comments · Fixed by #2204
Assignees
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@MarkDaoust
Copy link

Environment details

  • google-ai-generativelanguage 0.6.6

Steps to reproduce

pip install google.ai.generativelanguage
import google.ai.generativelanguage as glm
import datetimetime
import os

client = glm.GenerativeServiceClient(transport='rest', client_options={"api_key":os.environ['GOOGLE_API_KEY']})

times= [datetime.datetime.now()]
stream = client.stream_generate_content(
    model='models/gemini-1.5-pro',
    contents=[{'parts':[{'text':'Tell me a story about a talking cat named Bob.'}]}])

for chunk in stream:
    print(chunk.candidates[0].content.parts)
    print('#'*120)
    times.append(datetime.datetime.now())

for time in times:
  print(time)

Look at the output times, it's not streaming, It waits 10s, and then outputs all the chunks at once.

This problem goes away (the times are evenly spaced because the output is actually streaming) if:

  1. You use transport=GRPC, or
  2. In google.ai.generativelanguage you set "stream=True" in the generated code here.
@vchudnov-g vchudnov-g added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants