storage: reconsider the L0-sublevels / L0 files default limits

[nicktrav]

Admission control is currently configured with default values for L0 file count and L0 sublevels of 1k and 20, respectively.

With recent Pebble improvements (incremental L0 file addition, cockroachdb/pebble#1465), we should be able to handle more L0 files. We should consider looking into whether the 1k file limit is still a strong signal, and / or whether it could be raised?

More generally, the file count and sublevels defaults are static, and with improvements in code / hardware over time, we should consider revisiting how we set these numbers - e.g. a standard, reproducible experiment that can be run to determine a "magic number".

Jira issue: CRDB-14611

[jbowens]

I think we should still be very cautious about allowing too many files in L0. NewL0Sublevels is still slow with superlinear (I don't remember the exact order) runtime with respect to the number of files. Even if flushes are faster at updating the sublevels data structure than they were previously due to #1465, compactions that touch L0 still must go through the slow sublevels rebuild. This can still produce unhealthy nodes. While a compaction is reconstructing sublevels it holds DB.mu, preventing memtable flushes from completing, inducing memtable stalls.

From the #1406 issue:

Looking at a profile of BenchmarkL0SublevelsInit which has ~33,000 files, and runs in 44.16s on my macbook

[bananabrick]

To add to @jbowens comment, even the incremental construction of L0 still needs to iterate through ALL files in L0, and it might even be doing that many key comparisons. We'll probably have to benchmark it to determine at what point flushes become slower.

[nicktrav]

Even though still a draft, I'm linking in #79083, which has some ideas for provoking an inverted LSM.

[nicktrav]

Linking this to cockroachdb/pebble#1623.