Session-level scores

[felixsg]

Describe the feature or potential improvement

At the moment, scores are related to traces and/or observations within a trace but cannot yet be associated with a session. However, there are cases where you might want to provide scores for entire sessions. For instance, in a mental health chatbot, you would want to evaluate whole sessions using different metrics, since individual traces may not provide sufficient context.

Additional information

No response

[marcklingen]

Thanks for sharing, super helpful and something I chatted about with a couple of teams running on langfuse already.

[gns-fynd]

Hey @marcklingen
Is there any updates on langfuse on evaluating the session level? For our use case also we need to evaluate the agents on a session level. So does the langfuse have this support or not.

[marcklingen]

Langfuse currently does not have support for adding evaluation scores to a session. Thanks for sharing that this is important to you

[malikbrh]

Hi there @marcklingen , any news on supporting scores for sessions? Would be very useful on a project I have for my bachelor degree, I would need to compute mean metrics of multiple traces.

By the way, thanks for all the hard work on the project, you guys rock !

[marcklingen]

No update on this yet, thanks for your +1 that this would be useful

[malikbrh]

What do you think would be the best solution to achieve mean results of different scores? I thought about building a custom webapp to aggregate scores from Langfuse's API, but before putting multiple hours in this I wanted to know if any other already shipped Langfuse feature would be appropriate for this.

[marcklingen]

You can also add this to the run metadata at the end of the run while you might have all evals still available locally