[Generations]: Support n parameter

[ngamolsky]

Currently, when using the n parameter in openai completions, this parameter is not visible anywhere in the trace, and we only see the first choice output.

It should be supported in two places:

1. At the top of the trace detail where we see the other generation params (see screenshot)
2. In the output.

This is especially important when capturing generation traces as datasets, as if you are using the n parameter you likely want all the outputs in your dataset.

Im sure there are several ways of implementing the second piece, in terms of using a list of outputs or splitting out different generations (probably less correct).

Related discussions:
BerriAI/litellm#4964

[marcklingen]

Thanks for sharing! To confirm, do you use the litellm python sdk?

At the top of the trace detail where we see the other generation params (see screenshot)
In litellm, all optional_params are used as Langfuse model_parameters. If n was included in the optional_params, it should show up in Langfuse.

In the output.
I agree, all choices need to be recorded in Langfuse. I'd suggest to store them as an array of different choices on the generation output as mentioned here. This is a change that would need to be made within the litellm langfuse integration which currently logs out the first choice.

Are you interested in contributing to this? Happy to help with reviews of the PRs.

[ngamolsky]

Thanks for getting back to me!

Currently I am using the observe decorator on higher level functions together with the OpenAI integration to capture the generation.

I believe looking at the source code modelParameters is defined just as:

    modelParameters = {
        "temperature": parsed_temperature,
        "max_tokens": parsed_max_tokens,  # casing?
        "top_p": parsed_top_p,
        "frequency_penalty": parsed_frequency_penalty,
        "presence_penalty": parsed_presence_penalty,
    }
    if parsed_seed is not None:
        modelParameters["seed"] = parsed_seed

Or maybe I'm missing where those optional_params are passed?

This is how I setup the generation in case it's helpful:

from langfuse.openai import AsyncAzureOpenAI, AsyncOpenAI, AzureOpenAI, OpenAI



async def _call_openai_model_api_async(
        self,
        model: Enum,
        client: AsyncOpenAI | AsyncAzureOpenAI,
        messages: list[dict[str, str]],
        structured_output: StructuredOutputT | None = None,
        return_logprobs: bool = False,
        n_choices: int = 1,
        temperature: float = 1,
        prompt: ChatPromptClient | None = None,
    ) -> ParsedChatCompletion[StructuredOutputT]:
        model_obj = self._create_pydantic_model_from_schema(structured_output)
        llm_args = {
            "logprobs": return_logprobs,
            "messages": messages,
            "model": model.value,
            "response_format": type_to_response_format_param(model_obj),
            "n": n_choices,
            "temperature": temperature,
            "langfuse_prompt": prompt,
        }

        completion = await client.chat.completions.create(**llm_args)

        return parse_chat_completion(
            response_format=model_obj,
            chat_completion=completion,
            input_tools=[],
        )

Happy to dig into this and submit a pull request!

[marcklingen]

Thanks, sorry I assumed that you use the LiteLLM Python SDK, that's why I referred to their implementation.

I agree, that the Langfuse openai integration should capture this, creating an issue for this to track it.

[marcklingen]

Created an issue here, open for contributions on this by the way!
#4199