Langfuse retention settings

[sudhanshu746]

Describe the feature or potential improvement

Thanks for providing this great tool, a feature could be a retention settings for trace purge automatically like all traces > 30 days old.

Additional information

No response

[marcklingen]

Agree, with the migration to Langfuse v3, setting expiration dates on single rows will become simpler to handle data retention at scale in langfuse. Thanks for raising this!

[AlithAnar]

If there is a safe way to delete traces from PostgreSQL in current v2 setup could you please share from which tables can we manually delete records? are they connected with foreign keys with cascading deletetion feature or we have to manually delete some disconnected remnants?

[msabramo]

I am also interested in this. Folks in my org would like to be able to limit data storage to 24 hours to comply with regulations.

LangSmith has this feature: https://docs.smith.langchain.com/self_hosting/configuration/ttl

[marcklingen]

Quick update on this thread, retention capabilities will be added in the upcoming v3 of Langfuse as a project-level setting.

Langfuse Cloud

If you run on Langfuse Cloud, please reach out to us and we can manage retention settings for you project on our end internally.

Self-hosted

If you run self-hosted, you can safely delete data from the following tables:

DELETE FROM traces
WHERE "timestamp" < '2024-08-01'

DELETE FROM observations
WHERE "start_time" < '2024-08-01'

DELETE FROM scores
WHERE "timestamp" < '2024-08-01'

You can easily run these queries on a cron schedule. If you db is huge, please keep in mind that vacuuming might impact db performance while it is happening.

Q&A

are they connected with foreign keys with cascading deletetion feature or we have to manually delete some disconnected remnants?

No foreign key constraints are applied that would block a deletion of records. Background: Records need to be compliant with data privacy regulation as well as data needs to be ingested into langfuse without a constraint on the order of records.

[IsraelShok]

@marcklingen
Just to understand, in V3, the project-level setting will be for self-hosted also? or just to LangFuse CLoud users?
Thanks

[marcklingen]

v3 will enable more programmatic deletes at the infrastructure level. Currently, in Postgres, this can lead to unpredictable behavior due to vacuuming. We will add project-level retention settings once v3 is released.

[sudhanshu746]

Thank you, @marcklingen for sharing options.I am closing this thread for now.

[nitinkalra-bst]

are you not creating these tables partitioned by date? can't we just drop the older partitions?

[nitinkalra-bst]

@marcklingen please check this. It takes a lot of time to run delete command

[marcklingen]

Currently they are not partitioned b date but indexed by date. This will be changed in v3 as the data will be more immutable and thus easier to partition.