Can I evaluate Span using the External Evaluation Pipeline?

[ygt1qa8]

We can evaluate Trace using the External Evaluation Pipeline.
then, span can?

[dosubot[bot]]

Hey there, @ygt1qa8! 👋 I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let's tackle this together!

Yes, you can evaluate spans using the External Evaluation Pipeline in Langfuse. You can specify how Langfuse should fill the variables in the template by extracting data from trace, generations, spans, or event objects, which belong to a trace ^[1].

To continue talking to Dosu, mention @dosu.

[ygt1qa8]

@dosu
don't use LLM-as-a-judge.
I want to evaluate span with OSS Evaluation Tool(like Ragas, Deepeval etc...).

[dosubot[bot]]

You can evaluate spans using OSS Evaluation Tools like Ragas or Deepeval instead of LLM-as-a-judge. Ragas, for instance, is designed for model-based evaluation of RAG pipelines and can perform reference-free evaluations without needing ground-truth data. It can assess aspects like faithfulness, answer relevancy, and context precision. Langfuse supports integrating these tools to score your traces and spans, providing insights into your RAG pipelines' performance. You can set up external evaluation pipelines in Langfuse to use these tools for custom evaluations ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[marcklingen]

Yes, you can also build an external evaluation pipeline which evaluates single spans. Have a look at the api which also exposes all observations which can be filtered by name and other attributes.

[ygt1qa8]

@dosu
Please show me sample code in Python.

[dosubot[bot]]

To create an external evaluation pipeline for evaluating single spans in Langfuse using Python, you can follow these steps. Here's a sample code snippet to get you started:

1. Install Required Packages:

   pip install langfuse openai deepeval --upgrade

2. Set Up Environment Variables:

   import os
   
   os.environ["LANGFUSE_PUBLIC_KEY"] = "<your_public_key>"
   os.environ["LANGFUSE_SECRET_KEY"] = "<your_secret_key>"
   os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
   os.environ["OPENAI_API_KEY"] = "<your_openai_api_key>"

3. Fetch Spans:

   from langfuse import Langfuse
   
   langfuse = Langfuse(
       secret_key=os.environ["LANGFUSE_SECRET_KEY"],
       public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
       host=os.environ["LANGFUSE_HOST"]
   )
   
   spans = langfuse.fetch_spans(limit=10, name="your_span_name").data

4. Evaluate Spans:

   • Use an OSS Evaluation Tool like Ragas or Deepeval to evaluate the spans. Here's an example using a hypothetical evaluation function:

   def evaluate_span(span):
       # Implement your evaluation logic here
       # For example, using a custom metric or tool
       score = custom_evaluation_function(span)
       return score
   
   for span in spans:
       score = evaluate_span(span)
       print(f"Span ID: {span.id}, Score: {score}")

5. Log Scores in Langfuse:

   for span in spans:
       score = evaluate_span(span)
       langfuse.score(
           span_id=span.id,
           name="custom_metric",
           value=score
       )

This code provides a basic framework for fetching spans, evaluating them using a custom evaluation function, and logging the scores back to Langfuse. You can replace custom_evaluation_function with your specific evaluation logic or tool integration.

To continue talking to Dosu, mention @dosu.