Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add block header pruning. #3033

Conversation

shamil-gadelshin
Copy link
Contributor

This PR introduces block header pruning for blockchain-db. Original issue: #1570
This is a draft PR to show a possible solution while having several open questions.

Pruning block headers include:

  • removing data from DB using columns::KEY_LOOKUP, columns::HEADER, and block hash.
  • removing data from block header in-memory cache
  • removing data from header metadata in-memory cache (the last two using remove_header_metadata)

The PR consists of two parts:

  • change stale fork calculation: we needed this change because the previous algorithm relied on the block header persistence
  • block header pruning: we enable it using the new CLI argument ---prune-block-headers which defaults to false

Open questions:

  • What is the correct command to test the whole solution locally?
  • Are there other places to clean data from during the block header pruning?
  • The test prune_blocks_on_finalize_and_reorg marks several blocks in a row as finalized and commits the transaction later in contrast with prune_blocks_on_finalize which finalized every block in a separate transaction. If we modify prune_blocks_on_finalize_and_reorg test to query a header from the pruned fork we will get one, however, the database won't contain it. It seems that the cache gets polluted by dirty reads originating in meta updates because meta updates are saved separately and after the transaction. Is there a way to prevent it? I appreciate an advice here.

cc @nazar-pc @bkchr

- Prune all possible forks after block finalizing without height limit.
@cla-bot-2021
Copy link

cla-bot-2021 bot commented Jan 23, 2024

User @shamil-gadelshin, please sign the CLA here.

@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5008692

@shamil-gadelshin
Copy link
Contributor Author

Closed in favor separate PRs: #3962

@Polkadot-Forum
Copy link

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/block-header-pruning/7198/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants