Introduce experimental FIFO compaction settings for shreds in blockstore

[yhchiang-sol]

Summary of Changes

This PR introduces the experimental FIFO compaction settings for
storing shreds in the blockstore by adding related options under
BlockstoreOptions. Detailed settings will be optimized in the follow-up
PRs.

Note that this PR does not change the default compaction setting. It just
introduces a new option.

This PR is based on top of #22123, and is part of the proposal in #21358.
Once the setting is finalized and verified by the benchmark, it should help address issue #16234.

[yhchiang-sol]

Test results of using RocksFifo can be found in the draft PR #22158.

[codecov]

Codecov Report

Merging #22140 (4fb28bb) into master (47e4291) will decrease coverage by 0.0%.
The diff coverage is 16.1%.

@@            Coverage Diff            @@
##           master   #22140     +/-   ##
=========================================
- Coverage    81.2%    81.2%   -0.1%     
=========================================
  Files         564      564             
  Lines      153421   153450     +29     
=========================================
+ Hits       124640   124649      +9     
- Misses      28781    28801     +20     

[yhchiang-sol]

All tests passed in #22158 which uses FIFO as the compaction style for data_shred and code_shred column families.

[carllin]

Out of curiosity, any potential issues if a validator opens a ledger in one compaction type, then shuts down and opens the ledger with the other compaction type? i.e. are they always locked into path after making a decision?

[yhchiang-sol]

Good question. It's actually covered in the proposal #21358.

From Level to FIFO is simple, because every single file under Level compaction has a disjoint key range, which makes it a valid FIFO format.

From FIFO to Level, the DB will hit the write stall immediately because there are too many L0 files. Typically running manual compaction first before opening the DB is suggested.

[carllin]

Would like to test that forward migration path from Level to FIFO just to ensure it doesn't stall. For instance run a validator for a while with level, then switch it to FIFO. Ideally on a couple of different nodes, a couple different times make sure they don't stall.

Second rollback case is a bit tricky, I think we need to think about how to expose the details of that second case to operators.

Is there a way to tell that the db was formerly opened in FIFO and is now switching to level so we can notify the operator they should perform manual compaction?

[yhchiang-sol]

Is there a way to tell that the db was formerly opened in FIFO and is now switching to level so we can notify the operator they should perform manual compaction?

I remember RocksDB itself does not store its options. The usual way to do it is to store the options inside an options file https://github.com/facebook/rocksdb/wiki/RocksDB-Options-File. Eventually, we might want to use RocksDB-Options-file to manage RocksDB options.

For the FIFO change, what we can do is to first have the FIFO code landed, then another PR to make it configurable via the command line arguments. Then we stay at this point for a while to make sure we will never revert back to a release where we cannot configure whether we want to use FIFO or Level. Once we reach that point, we can safely assume even when we roll back. The validator will still understand the command line argument and correctly use FIFO or Level after the revert.