When running in relatively high amount of concurrent requests we start getting: "API error occurred: Service unavailable." from Langfuse server

[sternr]

We've a Flask LangChain based app using a locally deployed LangFuse pod and normally everything works just great.
However sometimes we've a relatively high amount of concurrent requests and we start getting: "API error occurred: Service unavailable." in the logs from the langfuse library and it also drastically hurst our app's throughput \ latency.
Is there a way to tune the client config to increase concurrency \ buffering \ make it more async so that it doesn't hurt the main app?

[dosubot[bot]]

Hey there, @sternr! I'm here to assist you with any bugs, questions, or contributions. Let's tackle this issue together!

I found a similar closed issue that might be relevant to your situation: bug: ERROR:langfuse:API error occurred:. The suggested solution in that case was to upgrade Langchain and switch to the recommended from langchain_openai import AzureChatOpenAI ^[1].

For handling high concurrency and buffering in Langfuse, you can adjust several configurations in the client library:

1. Concurrency and Buffering Settings:

   • LANGFUSE_THREADS: Increase the number of consumer threads to handle more concurrent requests. The default is 1, but you can increase it if you encounter scaling issues.
   • LANGFUSE_FLUSH_AT: Set the maximum number of events to batch before sending.
   • LANGFUSE_FLUSH_INTERVAL: Set the maximum time to wait before sending a batch (in seconds for Python, milliseconds for JavaScript).

2. Retry and Timeout Settings:

   • LANGFUSE_MAX_RETRIES: Increase the number of retries for network requests. The default is 3.
   • LANGFUSE_TIMEOUT: Adjust the timeout for network requests. The default is 20 seconds.

3. Asynchronous Operations:

   • The Langfuse client library supports asynchronous operations by queuing calls to make them non-blocking. This should help reduce latency in your application ^[2][3].

4. Error Handling:

   • The library has a built-in retry mechanism for handling transient errors like 'Service unavailable'. It retries requests a few times before giving up, which can help mitigate temporary issues ^[4].

Adjusting these settings can help optimize your application for high concurrency and improve its throughput and latency.

To continue talking to Dosu, mention @dosu.

[sternr]

This is awesome, increasing threads and buffer really helped!

[marcklingen]

awesome, have a look here for more details: https://langfuse.com/docs/tracing#queuing-batching