Backfill blob storage fix #5119
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
realbigsean
added
bug
pawanjay176
approved these changes
Jan 23, 2024
There was a problem hiding this comment.
LGTM.
Tested the migration on goerli and it works as expected.
Jan 23 17:47:10.842 DEBG Attempting schema migration to_version: 19, from_version: 18, service: freezer_db
Jan 23 17:47:10.842 DEBG Migrating from v18 to v19
Jan 23 17:47:11.884 DEBG Collected 5065 blob lists to migrate
Jan 23 17:47:34.939 DEBG Wrote 5065 blobs to the blobs db
Jan 23 17:47:34.939 INFO Upgrading to v19 schema info: ready for Deneb once it is scheduled
lighthouse db inspect --network goerli --column blb
also shows that all blob data has been deleted from the hot db
Jan 23 17:59:24.158 INFO Running database manager for goerli network
Jan 23 17:59:24.235 INFO Blob DB initialized oldest_blob_slot: Some(Slot(7413760)), path: "/Users/pawan/.lighthouse/goerli/beacon/blobs_db"
Num keys: 0
Total: 0 bytes
| let ops = migration_schema_v19::upgrade_to_v19::<T>(db.clone(), log)?; | ||
| db.store_schema_version_atomically(to, ops) | ||
| } | ||
| (SchemaVersion(19), SchemaVersion(18)) => { |
There was a problem hiding this comment.
Why do we need the downgrade if it's no-op?
There was a problem hiding this comment.
I was thinking so we could support 19 -> 17 for example
beacon_node/beacon_chain/src/schema_change/migration_schema_v19.rs
Outdated
Show resolved
Hide resolved
Merged
|
The migration on goerli took about 69s without batch and 55s with batching: Initial database No batching during migration Batching 500 writes |
michaelsproul
added
ready-for-merge
danielrachi1
pushed a commit
to danielrachi1/lighthouse
that referenced
this pull request
Feb 14, 2024
* store blobs in the correct db in backfill * add database migration * add migration file * remove log info suggesting deneb isn't schedule * add batching in blob migration
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue Addressed
Resolves #5114
Proposed Changes
import_historical_block_batch.Additional Info
I'm testing out the migration on goerli without batching any of the blob migration writes. I'm going to experiment with writing batches of 100 blob lists (max size ~60MB).