Add experimental x-trie-log subcommand for one-off backlog prune #6188
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java
Fixed
Show fixed
Hide fixed
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
21 tasks
siladu
changed the title
Closed
|
|
My last run on this with a medium-running node (3.5 months) gave this results: Running total number pruned = real 195m11.516s DB was reduced by 64GB After: |
garyschulte
reviewed
Dec 12, 2023
Comment on lines
+145
to
+149
| rootWorldStateStorage | ||
| .streamTrieLogKeys(limit) | ||
| .map(Bytes32::wrap) | ||
| .map(Hash::wrap) | ||
| .forEach( |
There was a problem hiding this comment.
I am guessing that most users of this subcommand are going to want to prune pretty close to head. Meaning there will be far more trielogs deleted than retained. This implementation is safe, but probably going to be pretty slow in most cases.
IMO we could/should add an alternate implementation that checks that the number of retained trielogs is below a certain reasonable threshold. We copy those trielogs to an alternate column family, truncate the trielog family, and move the trielogs back into the newly truncated CF. That should make this operation markedly faster for what I suspect will be the typical use case (offline pruning a huge number of trielogs)
There was a problem hiding this comment.
to be clear, I would start from the blockchain storage and get the latest 'x' hashes that we want to retain rather than streaming all of the keys in the trielog CF
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR description
Part of #5390
TODO
To install on a node
Usage and examples