Removal of old trie logs specified by historical block limit

[non-fungible-nelson]

How to enable this feature: https://wiki.hyperledger.org/display/BESU/Limit+Trie+Logs+for+Bonsai

There is no need to keep all the trie logs for a near-head node as the state for old blocks is rarely needed. This will place a configurable limit on how many trie logs are kept reducing storage needed for a node.

• Add configurable limit for number trie logs to be kept. This should be the same as the Bonsai read limit config
  --bonsai-historical-block-limit parameter
• Prune old trie logs that are not needed
• Trie logs are stored by block hash so there will need to be a way of determining with block hashes to remove the trie logs for
• The logic used for the block pruning which keeps a list of block hashes for each block number could possibly be reused for pruning the trie log
• This should be enabled independently of the chain data pruning feature and can be used together
• Measure storage saved by having the trie log pruning enabled
• Test that reorgs have their trie log pruned

Acceptance Criteria

• DB size of long running node should be the same before and after resync

Notes:

• The ChainDataPruner could be a helpful place to start and reuse some of the same logic

Tasks:

• First iteration of --Xbonsai-trie-log-pruning-enabled: Add trie log pruning triggered after trie log persist #6026
• Consider using safeBlock instead of 512 layers to retain
  • would be a nice strategy to prune only blocks older than RETAINED_LAYERS that have a subsequent finalized blockhash
• Consider belts and braces protection during rollback/forward in case pruned trielog may be in use
  • not sure how useful this is now we only prune finalized blocks
• DB size of long running node should be the same before and after resync -
  • without flat db https://gist.github.com/siladu/0a6fe37fe6de350cef233136e237a6e4
  • with flat db https://gist.github.com/siladu/a200ca16b5189a827e14eaa74be21ad7
  • with flat db, using debug_worldSyncState https://gist.github.com/siladu/0a6fe37fe6de350cef233136e237a6e4
• Measure storage saved by having the trie log pruning enabled - underway on *simon-6026* and *simon-flatdb* nodes

    crontab -e 
    
    0 16 * * * echo "\n\n=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n$(date)\n" >> /home/simon.dudley/rocksdb-usage.txt 2>&1
    1 16 * * * time /opt/besu/6000/bin/besu --config-file=/etc/besu/config.toml operator x-rocksdb usage >> /home/simon.dudley/rocksdb-usage.txt 2>&1
  

  • 2 week old TRIE_LOG_STORAGE saving is 6GB = 3GB per week
  • 7 week old TRIE_LOG_STORAGE saving is 22GB = 3.14GB per week
• Subcommand to one-off prune the backlog, see [WIP] Trie log pruning  #6000
  • RocksDB usage subcommand - Add RockDB Subcommand for printing usage per column family #6185
  • Subcommand to prune trie-logs "offline" - Add experimental x-trie-log subcommand for one-off backlog prune #6188 (needs testing on an old node to see how long it might take to run) Add --X-trie-log subcommand #6303
• Reuse --bonsai-historical-block-limit instead of --Xbonsai-trie-log-retention-threshold
  • Reuse --bonsai-historical-block-limit for limit trie logs feature #6445
• Rename --Xbonsai-trie-log-pruning-enabled to --Xbonsai-limit-trie-logs-enabled - YES
• Done as part of Import export trie log #6363
• Optimisation 1: Move pruning to async thread
  • Trie log prune using TrieLogEvent #6394
• Add trielog pruning to canary nodes - this was split out to a new internal issue

---

Nice to have:

• Reorg acceptance test
  • Convert this script to AT: https://gist.github.com/siladu/bde571af9880733d4dfdee9f30ce0fc4
• Optimisation 3: Cache finalizedBlockHeader in Blockchain or use Observer pattern to pass it to the pruner
• Consider an option to only prune forks
• Prevent block building from saving orphaned trie logs ?
• Optimisation 2: Prune in batches instead of every block?

[siladu]

Might be useful for testing https://github.com/ahamlat/RocksdDB-Column-Families-Size

[siladu]

• Are we also saving trie logs when block building?
  Yes...see code path starting here

  besu/consensus/merge/src/main/java/org/hyperledger/besu/consensus/merge/blockcreation/MergeCoordinator.java

  Line 546 in 1c261db

  private BlockProcessingResult validateProposedBlock(final Block block) {