[Feature] Add Support on Log Probability Value from Returned Response of Gemini Models.

[jacklanda]

Description of the feature request:

How about providing new support for retrieving the log probability of each predicted token by Google models like Gemini?

Something just like the same function illustrated in this OpenAI post: User Logprobs.

What problem are you trying to solve with this feature?

Getting the returned log prob of each generated token does help me (and other users maybe) to confirm the confidence of the model's prediction. Further, this feature can help users compute the perplexity of generated sentences to better understand the textual continuation quality.

Any other information you'd like to share?

No response

[jacklanda]

Any thoughts about it?

[Brian-Ckwu]

I also think it would be extremely helpful if the API could provide the top-k log probabilities of each predicted token.

[krenova]

Really need this as I found Gemini to be hallucinating quite abit on a RAG application.

Btw your title refers to claude and not gemini.
[Feature] Add Support on Log Probability Value from Returned Response of Claude Models

[jacklanda]

Really need this as I found Gemini to be hallucinating quite abit on a RAG application.

Btw your title refers to claude and not gemini. [Feature] Add Support on Log Probability Value from Returned Response of Claude Models

Thanks for reminding 🤗

[simpleusername96]

Many hallucination detection approaches rely on log probability as a key feature. It's one of the most essential elements when building a serious product with a LLM.

[anwang427]

This would be extremely helpful!

[Said-Apollo]

As all others previously mentioned, it would be great to somehow get access to the logsprobs in a similar fashion as OpenAI does so with the models. Based on that we could then e.g. calculate the perplexity score and various other evaluation metrics.

[haugmarkus]

Yep, this would make it possible to use gemini in production

[waveworks-ai]

+1 - useful also for classification tasks.

[lavanyanemani96]

+1

[luna-b20]

+1 it is critical. Chatgpt and other major models are now all supporting this feature.
please help!

[michaelgfeldman]

+1 it is critical. Chatgpt and other major models are now all supporting this feature. please help!

Can you please specify what major models support this feature? Because I'm also in search for the alternatives. Now I don't see any other players supporting this except from OpenAI. No Anthropic, No Mistral...

[MarkDaoust]

b/361194489

[MFajcik]

I also think it would be extremely helpful if the API could provide the top-k log probabilities of each predicted token.

Yes, this would allow evaluating Gemini with threshold-free evaluation metrics. That would be excellent.

[MarkDaoust]

Google's Vertex-AI Just launched this, hopefully that means it's coming soon here, but I don't have a timeline.

[michaelgfeldman]

Google's Vertex-AI Just launched this, hopefully that means it's coming soon here, but I don't have a timeline.

can you share a link?

[haugmarkus]

Google's Vertex-AI Just launched this, hopefully that means it's coming soon here, but I don't have a timeline.

can you share a link?

They have added a field 'avgLogprobs' to the response documentation in https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference#nodejs but I am unable to get a response with such field.

[MarkDaoust]

Yeah, this API is related, but separate from vertex. Hopefully this API will catch up soon.

[kauabh]

• 1

[MarkDaoust]

This is fixed in the latest version:

code: #561
tutorial: https://github.com/google-gemini/cookbook/blob/main/quickstarts/New_in_002.ipynb