[core] Implement a vacuum strategy for the offline database

[jfirebaugh]

Enable PRAGMA auto_vacuum = INCREMENTAL, and perform a PRAGMA incremental_vacuum when deleting an offline region.

I'm targeting this for the release because enabling auto vacuum on an existing database has some pitfalls, and doing before offline sees widespread adoption will minimize the number of affected databases.

Fixes #4340.

👀 @tmpsantos @1ec5

[tmpsantos]

I think calling vacuum should be a separated API and not called automatically. I can think of 2 use cases for that:

• In your app you want to remove several regions. Each deletion would be super slow because the database would be recreated on each VACUUM. Instead, the user interface would be designed in a way you delete all regions first and at the end call OffilineDatabase::vacuum(), committing to the operation.
• This API could be used for vacuum the dangling pages from the cache.

[jfirebaugh]

Note that PRAGMA incremental_vacuum is not the same as VACUUM. In particular, my interpretation of the documentation is that PRAGMA incremental_vacuum is much cheaper: it works with a page structure that has already been prepared for incremental vacuum via PRAGMA auto_vacuum, and it doesn't compact partially filled pages of the database nor rewrite the entire database file like VACUUM does. I am not sure that a single PRAGMA incremental_vacuum after deleting several regions would be significantly cheaper than calling PRAGMA incremental_vacuum after each deletion. In fact, it might block other database commands for a longer continuous period of time.

Regarding the second point, we expect the number of free pages for the ambient cache to be low. The expected behavior is that the ambient cache fills up to the maximum allotted size, and then stays at that size. Vacuuming free pages from it would in fact be a deoptimization, because those pages would just need to be reallocated as soon as the cache filled back up to its maximum size.

I'd prefer to keep the vacuum strategy a private implementation detail for now. There are certainly some unknowns about how it will perform, but it's not clear to me that providing API control will help.

[tmpsantos]

Thanks for clarifying.

[tmpsantos]

@jfirebaugh 👍 + a comment.