Choose a better threshold for the soft deletion

[irevoire]

Once #557 is merged the engine will wait until 10k documents have been soft deleted before running a real deletion.
Since this number is so smol it should not cause any issue, but the bigger this number is and the bigger the perf improvement is. BUT if this number gets too big we're going to cancel indexing tasks saying there is no space left on the device when in reality there are a lot of documents waiting to be deleted.

Thus I would like to think of a better threshold, currently, my favourite idea would be to calculate an approximative document size (we take the size of the DB divided by the number of documents + soft-deleted documents) and, if the number of soft-deleted documents reaches 10% of the total space we can use then we delete them for real.
Also maybe this could be configurable. Maybe instead of 10% we should instead say something like « 100mb » (but in this case it should really be configurable).

[irevoire]

Currently, we can't use the size of the disk (or estimate the size each document takes) because we don't have this information.
The only information we have is the size of the database on disk with all freed pages, and if we use that, we're going to allow more and more soft-deleted documents for the same number of indexed documents every time we re-index.

To make this works, we would need to add a new feature in heed that multiplies the size of a memory page by the number of leaf pages. Maybe by the number of leaf pages + branch pages + overflow pages?
I can use the mdb_env_stat function to aggregate my results over all databases.

This doesn’t work.
There is also a mdb_envinfo structure in lmdb, but it’s useless as well.
Both structures do not change when we index a file which means it’s measuring useless data.

[irevoire]

The last solution I can think of is to use the MDB_env structure ourselves to compute the number of free pages x the size of a page.
That would be a cool solution if only the structure was not private.

Thus there is three options from here;

1. I missed something, @Kerollmops I tried to ping @hyc on twitter but it seems like he didn’t see my tweet. It would be cool I you could try to contact him as well. Here is what I sent him; https://twitter.com/irevoire/status/1554770698475905024
2. We fork lmdb and introduce a function that compute the free space left for us.
   -> And if @hyc is ok with this idea we try to merge that on the main branch.
3. We compute the size directly from heed with a lot of very unsafe code. That should work while we don’t update lmdb 😩

[hyc]

Sounds like the best thing to do is to migrate the freespace code from mdb_stat.c into the library. But I don't see how that is actually a useful figure for the problem you're trying to solve here.

[irevoire]

Oh, nice to see you here @hyc!

But I don't see how that is actually a useful figure for the problem you're trying to solve here.

Well, basically, we want to get an idea of how much size we’re really using to avoid freeing space when we don’t need to (=there is still a lot of available space).
But since we can’t rely on the size used by lmdb on disk, I was trying to recompute the size myself 😅

Sounds like the best thing to do is to migrate the freespace code from mdb_stat.c into the library.

Is this something doable for you? Or should we include the code in our library?
I totally missed this file when I did my tests; I’ll look into it Tuesday!