Freelist endgame?

[andreyvit]

If bbolt is used in write-heavy workloads, freelist can get pretty large, especially during backups or other longer-running read transactions (think reporting, although that's easy to solve with replication).

As a data point, in our workloads, I sometimes see 30 GB of free pages, with multi-megabyte freelist. Running without freelist persistence because of this, otherwise persisting ~20-30 MB of freelist on each commit tanks write performance, even if we make those commit batches pretty large (we almost never rollback).

We're working on splitting the database, mind you, but would still like Bolt to handle larger databases. So I want to open a discussion on what Bolt freelist endgame looks like. Is there interest in seeing larger freelists supported on your end as well?

I know there's work underway to refactor the two implementations we currently have, but want to discuss the step after that.

The key observation is that only a small portion of a large freelist would be altered by any given write transaction. So would be nice to reuse most of the pages instead of writing the full freelist each time. This would mean treating the freelist as (some sort of a) tree. Not sure if it needs to be a B-tree, or if a simpler two

Does anyone have thoughts on this already, or is it something I'm free to explore on our end? The timeline is 2025, not right away; just opening the discussion early.