Another glimpse into prompt management in LangFuse

[barseghyanartur]

Describe the feature or potential improvement

Problem

Storing prompts in a database is helpful for experiments, but it falls short
when it comes to deployments in multiple environments.

All the hard work and optimizations can be lost, and you end up having to copy
your prompts from one environment to another. This is similar to
the "good old days" when you had to manage production databases for local
testing or set up staging environments.

The proposed solution

1. Store prompts on disk (repository level).
   • Prompts are saved on disk within the repository in a structured way
     that LangFuse understands.
   • This makes prompts available for all deployments without needing
     to copy them between environments.
2. Override prompts by unique identifier.
   • LangFuse allows overriding any prompt using its identifier (the
     relative path to the prompt serves as the prompt ID).
   • When prompt is overridden, LangFuse uses the new version for all
     experiments and when promoting to production.
   • Alongside the prompt ID, include a unique hash based on the
     original (on disk/repository) prompt's content. This ensures the
     changed version stays valid and active as long as the original
     prompt doesn't change.
3. GitHub integration for production promotion.
   • When promoting prompts to production, an optional workflow should
     integrate with GitHub (or similar, like GitLab, BitBucket, etc).
   • Once promoted, this workflow creates a pull request (PR) to the
     designated development branch specified in LangFuse settings.
   • Once the PR is merged and code is deployed, the prompt content on
     disk is updated, and LangFuse will again use the prompt from
     disk/repository instead of the database.

Implementing this solution is not simple, but I think it's the most effective
way to manage prompts efficiently across different environments and
deployments.

Additional information

No response

[marcklingen]

Thanks for sharing, when referring to "copy your prompts from one environment to another" do you refer to multiple langfuse projects?

[barseghyanartur]

I mean, multiple deployments of the same project (like DTAP).

[marcklingen]

I'd recommend to use a single Langfuse instance across all environments as this generally facilitates collaboration across teams. If that's not possible for the observability features, I've seen some teams use one Langfuse deployent for prompt management across environments.

[barseghyanartur]

I don’t want to sound insistent, but there are additional benefits to using code-based prompts.

1. Increased Resilience to Outages: The application becomes more resilient. If something goes wrong with the LangFuse platform or the self-hosted version, you still have a fallback option - the code. Application still runs fine.
2. Simplified Integration: LangFuse serves as a thinner layer within the application, reducing the risks associated with adopting or abandoning the platform. This makes it much easier to make A-level or B-level decisions.

[marcklingen]

Fully agree! I think for many prompts having them in code is a good decision as long as there is not a whole lot of collaboration necessary with non-engineering teams.

When more people are involved, some sort of prompt CMS often makes sense which usually requires moving prompts out of the git repository and into a service (such as langfuse prompt management). In Langfuse we add client-side caching and fallbacks to help with resilience and performance when using the prompt management tooling.

I understand that you'd be interested in king of a hybrid approach where there is UI and workflow for prompt management, while writing them either to disk (within the service) or to the repo itself. This is interesting! We considered build-time caching of prompts from the prompt management service to make sure prompts are always available, but then opted for optional pre-fetching for guaranteed availability as it simplifies the client a lot.

When using Langfuse for prompt management, most teams do not use multiple deployed instances of langfuse and use prompt management in each of them, but they usually use one instance as their "main" instance which then serves prompts to all environments. You can use labels on prompt versions in langfuse to the decide which version should be used in which of the environments.

Side note: Just to make sure, it is not necessary at all to use prompt management in Langfuse to use the other platform features. You can also fully manage them in code and just include metadata about the used prompt version in the observability features.