[Merged by Bors] - Add `db inspect --output values` option to support dumping raw db values #4324

jimmygchen · 2023-05-23T07:46:08Z

Issue Addressed

Add a new --output values option to db inspect for dumping raw database values to SSZ files.

This could be useful for inspecting the database when we're unable to start the beacon node.

Example usage:

# Output the `ForkChoice` column to an SSZ file
lighthouse db inspect --column frk --output values

By default, it stores the output files in the current directory, and can be overriden with --ouput-dir.

List of columns can be found here:

lighthouse/beacon_node/store/src/lib.rs

Lines 169 to 216 in c547a11

    
           pub enum DBColumn { 
        
               /// For data related to the database itself. 
        
               #[strum(serialize = "bma")] 
        
               BeaconMeta, 
        
               #[strum(serialize = "blk")] 
        
               BeaconBlock, 
        
               /// For full `BeaconState`s in the hot database (finalized or fork-boundary states). 
        
               #[strum(serialize = "ste")] 
        
               BeaconState, 
        
               /// For the mapping from state roots to their slots or summaries. 
        
               #[strum(serialize = "bss")] 
        
               BeaconStateSummary, 
        
               /// For the list of temporary states stored during block import, 
        
               /// and then made non-temporary by the deletion of their state root from this column. 
        
               #[strum(serialize = "bst")] 
        
               BeaconStateTemporary, 
        
               /// Execution payloads for blocks more recent than the finalized checkpoint. 
        
               #[strum(serialize = "exp")] 
        
               ExecPayload, 
        
               /// For persisting in-memory state to the database. 
        
               #[strum(serialize = "bch")] 
        
               BeaconChain, 
        
               #[strum(serialize = "opo")] 
        
               OpPool, 
        
               #[strum(serialize = "etc")] 
        
               Eth1Cache, 
        
               #[strum(serialize = "frk")] 
        
               ForkChoice, 
        
               #[strum(serialize = "pkc")] 
        
               PubkeyCache, 
        
               /// For the table mapping restore point numbers to state roots. 
        
               #[strum(serialize = "brp")] 
        
               BeaconRestorePoint, 
        
               #[strum(serialize = "bbr")] 
        
               BeaconBlockRoots, 
        
               #[strum(serialize = "bsr")] 
        
               BeaconStateRoots, 
        
               #[strum(serialize = "bhr")] 
        
               BeaconHistoricalRoots, 
        
               #[strum(serialize = "brm")] 
        
               BeaconRandaoMixes, 
        
               #[strum(serialize = "dht")] 
        
               DhtEnrs, 
        
               /// For Optimistically Imported Merge Transition Blocks 
        
               #[strum(serialize = "otb")] 
        
               OptimisticTransitionBlock, 
        
               #[strum(serialize = "bhs")] 
        
               BeaconHistoricalSummaries,

database_manager/src/lib.rs

…file.

michaelsproul

Great improvement, thanks!

michaelsproul · 2023-05-30T01:03:29Z

bors r+

…ues (#4324) ## Issue Addressed Add a new `--output values` option to `db inspect` for dumping raw database values to SSZ files. This could be useful for inspecting the database when we're unable to start the beacon node. Example usage: ``` # Output the `ForkChoice` column to an SSZ file lighthouse db inspect --column frk --output values ``` By default, it stores the output files in the current directory, and can be overriden with `--ouput-dir`. List of columns can be found here: https://github.com/sigp/lighthouse/blob/c547a11b0da48db6fdd03bca2c6ce2448bbcc3a9/beacon_node/store/src/lib.rs#L169-L216

bors · 2023-05-30T01:21:38Z

Build failed (retrying...):

compile-with-beta-compiler

michaelsproul · 2023-05-30T01:32:14Z

bors r-

bors · 2023-05-30T01:32:16Z

Canceled.

michaelsproul · 2023-05-30T01:38:27Z

bors r+

…ues (#4324) ## Issue Addressed Add a new `--output values` option to `db inspect` for dumping raw database values to SSZ files. This could be useful for inspecting the database when we're unable to start the beacon node. Example usage: ``` # Output the `ForkChoice` column to an SSZ file lighthouse db inspect --column frk --output values ``` By default, it stores the output files in the current directory, and can be overriden with `--ouput-dir`. List of columns can be found here: https://github.com/sigp/lighthouse/blob/c547a11b0da48db6fdd03bca2c6ce2448bbcc3a9/beacon_node/store/src/lib.rs#L169-L216

bors · 2023-05-30T04:13:44Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

…ues (sigp#4324) ## Issue Addressed Add a new `--output values` option to `db inspect` for dumping raw database values to SSZ files. This could be useful for inspecting the database when we're unable to start the beacon node. Example usage: ``` # Output the `ForkChoice` column to an SSZ file lighthouse db inspect --column frk --output values ``` By default, it stores the output files in the current directory, and can be overriden with `--ouput-dir`. List of columns can be found here: https://github.com/sigp/lighthouse/blob/c547a11b0da48db6fdd03bca2c6ce2448bbcc3a9/beacon_node/store/src/lib.rs#L169-L216

Add db inspect option to dump raw values

8bab02f

jimmygchen self-assigned this May 23, 2023

jimmygchen added 2 commits May 23, 2023 17:48

Fix clippy issues

2c0aa63

Remove unused function

6360a58

michaelsproul added the database label May 23, 2023

jimmygchen added the ready-for-review The code is ready for review label May 23, 2023

jimmygchen commented May 24, 2023

View reviewed changes

database_manager/src/lib.rs Show resolved Hide resolved

michaelsproul added work-in-progress PR is a work-in-progress and removed ready-for-review The code is ready for review labels May 25, 2023

michaelsproul reviewed May 25, 2023

View reviewed changes

database_manager/src/lib.rs Outdated Show resolved Hide resolved

Create output directory if not already exist, and overwrite existing …

e63d192

…file.

jimmygchen added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels May 26, 2023

Empty commit.

b0b4db0

michaelsproul approved these changes May 30, 2023

View reviewed changes

michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels May 30, 2023

bors bot changed the title ~~Add db inspect --output values option to support dumping raw db values~~ [Merged by Bors] - Add db inspect --output values option to support dumping raw db values May 30, 2023

bors bot closed this May 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Add `db inspect --output values` option to support dumping raw db values #4324

[Merged by Bors] - Add `db inspect --output values` option to support dumping raw db values #4324

jimmygchen commented May 23, 2023

michaelsproul left a comment

michaelsproul commented May 30, 2023

bors bot commented May 30, 2023

michaelsproul commented May 30, 2023

bors bot commented May 30, 2023

michaelsproul commented May 30, 2023

bors bot commented May 30, 2023

	pub enum DBColumn {
	/// For data related to the database itself.
	#[strum(serialize = "bma")]
	BeaconMeta,
	#[strum(serialize = "blk")]
	BeaconBlock,
	/// For full `BeaconState`s in the hot database (finalized or fork-boundary states).
	#[strum(serialize = "ste")]
	BeaconState,
	/// For the mapping from state roots to their slots or summaries.
	#[strum(serialize = "bss")]
	BeaconStateSummary,
	/// For the list of temporary states stored during block import,
	/// and then made non-temporary by the deletion of their state root from this column.
	#[strum(serialize = "bst")]
	BeaconStateTemporary,
	/// Execution payloads for blocks more recent than the finalized checkpoint.
	#[strum(serialize = "exp")]
	ExecPayload,
	/// For persisting in-memory state to the database.
	#[strum(serialize = "bch")]
	BeaconChain,
	#[strum(serialize = "opo")]
	OpPool,
	#[strum(serialize = "etc")]
	Eth1Cache,
	#[strum(serialize = "frk")]
	ForkChoice,
	#[strum(serialize = "pkc")]
	PubkeyCache,
	/// For the table mapping restore point numbers to state roots.
	#[strum(serialize = "brp")]
	BeaconRestorePoint,
	#[strum(serialize = "bbr")]
	BeaconBlockRoots,
	#[strum(serialize = "bsr")]
	BeaconStateRoots,
	#[strum(serialize = "bhr")]
	BeaconHistoricalRoots,
	#[strum(serialize = "brm")]
	BeaconRandaoMixes,
	#[strum(serialize = "dht")]
	DhtEnrs,
	/// For Optimistically Imported Merge Transition Blocks
	#[strum(serialize = "otb")]
	OptimisticTransitionBlock,
	#[strum(serialize = "bhs")]
	BeaconHistoricalSummaries,

[Merged by Bors] - Add db inspect --output values option to support dumping raw db values #4324

[Merged by Bors] - Add db inspect --output values option to support dumping raw db values #4324

Conversation

jimmygchen commented May 23, 2023

Issue Addressed

michaelsproul left a comment

Choose a reason for hiding this comment

michaelsproul commented May 30, 2023

bors bot commented May 30, 2023

michaelsproul commented May 30, 2023

bors bot commented May 30, 2023

michaelsproul commented May 30, 2023

bors bot commented May 30, 2023

[Merged by Bors] - Add `db inspect --output values` option to support dumping raw db values #4324

[Merged by Bors] - Add `db inspect --output values` option to support dumping raw db values #4324