Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[R4R]offline block prune #543

Merged
merged 26 commits into from
Jan 19, 2022
Merged

[R4R]offline block prune #543

merged 26 commits into from
Jan 19, 2022

Conversation

Mercybudda
Copy link
Contributor

@Mercybudda Mercybudda commented Nov 14, 2021

Description

This is for the off-line block prune, in bsc storage, there's db wrapper of ancientDb(freezer) and kvDb(levelDB), the space was mainly occupied by ancientDb, the purpose of this is to prune and remain the specific number of old blocks in ancientDb by specifying the amount of blocks you'd like to remain after pruning in the CLI flag. Then the pruner will prune and only remain this specified amount backward, this first block is not necessarily the genesis block because we may prune many rounds. Actually we can prune and remain any amount of blocks as long as the amount specified did exceed the number of blocks inside ancientDb
###Example:
Assume all blocks in ancientDb are [0,1,... 1000], if pruning amount of 500 blocks, then [0,...499] will be pruned, the data in ancientDb will be [500, 1000]. The next round for pruning will start with BlockNumber 500.

Brief Coding workflow is as follows

  1. Create a new ancientDb without a kvdb wrapper, it will create a new folder ./node/geth/chaindata/ancient_backup for initializing new ancientDb.
  2. Back-up all the blocks at and after oldOffSet + frozen - BlockPruneAmountLeft from old ancientDb into new ancientBackup. In this process, pruner will read one by one for block, receipts, difficulty from old ancientDb and write to new backup ancientDb. To be noted that there’ll be a new offset of new ancientDb, this offset will be recorded into kvdb.
  3. Delete directly for the old folder./node/geth /chaindata/ancient/, i.e. the old ancientDb.
  4. Rename the new folder of ./node/geth/chaindata/ancient_backup into ./node/geth/chaindata/ancient, then assemble with the same kvdb of the original DB wrapper in which old ancientDb resides.
  5. Finally the ancientDb in DB wrapper is replaced successfully, all the block data we want to reserve were in this new ancientDb.

Self Test

  1. Unit test: Initialized a blockchain of 500 blocks, testing the pruning of 328 amount of blocks remaining. passed
  2. Locally geth test, start a full node: ./geth --config ./config.toml --datadir ./node --cache 12000 --rpc.allow-unprotected-txs --txlookuplimit 0 --syncmode full --usb, after a while until there's enough old block for pruning in ancientDb.
  3. 1st round of prune test: Start the prune process: ./geth snapshot prune-block --datadir ./node --datadir.ancient ./chaindata/ancient --block-amount-reserved 10000. This will prune and reserve 10000 blocks.
    Passed and get a new ancientDb with amount of 10000 blocks
  4. 2nd round of prune test, start the geth again, repeat the step 2 & 3.
  5. Introduced the flag inspect-reserved-oldest-blocks, can be used for checking information of ancientStore like offset and reserved items amount after pruning done.
    ./geth db inspect-reserved-oldest-blocks --datadir ./node
    +--------------------------------+--------------------------------+--------+
    | DATABASE | CATEGORY | ITEMS |
    +--------------------------------+--------------------------------+--------+
    | Offset/StartBlockNumber | Offset/StartBlockNumber after | 302672 |
    | | BlockPrune | |
    | Amount of remained items in | remaining items after | 10000 |
    | AncientStore | BlockPrune | |
    +--------------------------------+--------------------------------+--------+
    | ANCIENTSTORE INFORMATION AFTER |
    | OFFLINE BLOCKPRUNE |
    +--------------------------------+--------------------------------+--------+

Running command

e.g.
./geth snapshot prune-block --datadir ./node --datadir.ancient ./chaindata/ancient --block-amount-reserved 9000 --triesInMemory 32 --check-snapshot-with-mpt
--block-amount-reserved: Amount of blocks want to reserve.
--triesInMemory(optional): The layer of tries trees that keep in memory.
--check-snapshot-with-mpt(optinal): Indicate if enable/disable checking for the match between MPT and snapshot.

Check the info of ancientDB
./geth db inspect-reserved-oldest-blocks --datadir ./node

Result

  1. The offset of ancientDb changed as expected.
  2. Geth client restarted successfully.
  3. new block data freeze() moving from kvDb to ancientDb as expected.

Preflight checks

  • build passed (make build)
  • tests passed (make test)
  • manual transaction test passed

@Mercybudda Mercybudda force-pushed the blockprune branch 3 times, most recently from 4fe76ff to cad935c Compare December 17, 2021 08:00
@unclezoro unclezoro changed the base branch from master to develop December 20, 2021 02:21
@unclezoro unclezoro changed the base branch from develop to master December 20, 2021 04:02
cmd/geth/snapshot.go Outdated Show resolved Hide resolved
},
Description: `
Offline prune for block data.
For AncientFlag, please specify the absolute path of node/geth/chaindata.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why it have to be absolute path?

return nil
}

func getAncientPath(ctx *cli.Context) string {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let us follow the same code style. Please remove this func, but use ctx.GlobalString(utils.AncientFlag.Name) directly.

defer stack.Close()

chaindb := utils.MakeChainDatabaseForBlockPrune(ctx, stack, false)
chaindb.Close()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you meaning defer chaindb.Close()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is for closing it on purpose, because in the later code, there'll be many times of open/close the same db, crash will occur if did not close it.

oldAncientPath = path + "/ancient"
newAncientPath = path + "/ancient_back"
} else {
utils.Fatalf("Prune failed, did not specify the AncientPath %v")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better have a default ancient path.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have %v here , but no arg.

return errors.New("Prune failed, did not specify the AncientPath")
}

//TODO for filelock
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will we have file lock?

// if err != nil {
// utils.Fatalf("Failed to read genesis file: %v", err)
// }
for _, name := range []string{"chaindata"} {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why range a slice that only have one const item?

root := stack.ResolvePath(name)
switch {
case oldAncientPath == "":
oldAncientPath = filepath.Join(root, "ancient")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not possible to happen, since you forbidden it to be empty in previous logic.

}
pruner, err := pruner.NewBlockPruner(chaindb, stack, oldAncientPath)
if err != nil {
utils.Fatalf("Failed to create block pruner", err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get err arg here, but no %v

log.Info("geth block offline pruning backup successfully")

//After backing up successfully, rename the new ancientdb name to the original one, and delete the old ancientdb
if err := pruner.BlockPrune(oldAncientPath, newAncientPath); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is quite weried you already pass oldAncientPath when NewBlockPruner, but pass it again when doing BlockPruneBackUp, it is duplicated.

err error
chainDb ethdb.Database
)
if ctx.GlobalString(SyncModeFlag.Name) == "light" {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if sync mode is light, there is no need to do block prune

node/node.go Outdated
// OpenDatabaseWithFreezerForPruneBlock opens an existing database with the given name (or
// creates one if no previous can be found) from within the node's data directory,
// also attaching a chain freezer to it. If the node is an ephemeral one, a memory database is returned.
func (n *Node) OpenDatabaseWithFreezerForPruneBlock(name string, cache, handles int, freezer, namespace string, readonly bool) (ethdb.Database, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please try not to invent new function, too many duplicated code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These new functions were the function without freeze go-routine running, to avoid the case that freeze go-routine did not stop while db closing.

node/node.go Outdated
// creates one if no previous can be found) from within the node's data directory,
// also attaching a chain freezer to it. If the node is an ephemeral one, a
// memory database is returned.
func (n *Node) OpenDatabaseWithFreezerBackup(offset uint64, name string, cache, handles int, freezer, namespace string, readonly bool) (ethdb.Database, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

return errors.New("can't access the freezer or it's empty, abort")
}
// Get the actual start block number.
startBlockNumber := frozen - 128 + oldOffSet
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

128 is a magic number here, please use constant value.

@unclezoro unclezoro merged commit 476d520 into develop Jan 19, 2022
This was referenced Jan 26, 2022
@Mercybudda Mercybudda deleted the blockprune branch February 8, 2022 03:04
unclezoro pushed a commit that referenced this pull request Mar 1, 2022
* offline block prune

* update

* update

* update and add unit test

* addressed comments from walt

* Addressed comments from walt and Igor

* ensure MPT and snapshot matched

* add one more parameter to indicate blockprune

* update the logic of creating freezerDb

* update flag command description

* expose the function for db inspect the offset/startBlockNumber

* add flags to inspect prune info

* rename flag of reserved-recent-blocks to block-amount-reserved

* addressed comments from walt

* handle the case of command interruption

* refined goimports

* addressed comments from walt

* change the logic as restarting prune after interruption

* addressed comments

* reclaimed freezer logic

* introduce flag to enable/disable check between MPT and snapshot

* update the logic of frozen field in freezerDB

* update the code in all places related to freezer change

* addressed comments from dylan

* update the logic for backup block difficulty

* addressed comments from dylan
keefel pushed a commit to keefel/bsc that referenced this pull request Jun 6, 2022
* offline block prune

* update

* update

* update and add unit test

* addressed comments from walt

* Addressed comments from walt and Igor

* ensure MPT and snapshot matched

* add one more parameter to indicate blockprune

* update the logic of creating freezerDb

* update flag command description

* expose the function for db inspect the offset/startBlockNumber

* add flags to inspect prune info

* rename flag of reserved-recent-blocks to block-amount-reserved

* addressed comments from walt

* handle the case of command interruption

* refined goimports

* addressed comments from walt

* change the logic as restarting prune after interruption

* addressed comments

* reclaimed freezer logic

* introduce flag to enable/disable check between MPT and snapshot

* update the logic of frozen field in freezerDB

* update the code in all places related to freezer change

* addressed comments from dylan

* update the logic for backup block difficulty

* addressed comments from dylan
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023
Signed-off-by: Delweng <delweng@gmail.com>
jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023
Signed-off-by: Delweng <delweng@gmail.com>
manav2401 pushed a commit to maticnetwork/bor that referenced this pull request Apr 9, 2024
Signed-off-by: jsvisa <delweng@gmail.com>
manav2401 pushed a commit to maticnetwork/bor that referenced this pull request Apr 9, 2024
Signed-off-by: Delweng <delweng@gmail.com>
manav2401 added a commit to maticnetwork/bor that referenced this pull request May 9, 2024
* core/state: typo

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <delweng@gmail.com>

* eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <delweng@gmail.com>

* eth,core: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <delweng@gmail.com>

* cmd: open db with freeze disabled

Signed-off-by: Delweng <delweng@gmail.com>

* cli: snapshot prune-block

Signed-off-by: Delweng <delweng@gmail.com>

* fix typo

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: fix the issue of dup open db error

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: resolve datadir and ancient before backup

Signed-off-by: Delweng <delweng@gmail.com>

* core: more prune-block log

Signed-off-by: Delweng <delweng@gmail.com>

* core: truncatetail missing f.offset

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: indextx adjust offset of pruned block

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: freezer batch should implement the offset commit, ref bnb-chain/bsc#1005

Signed-off-by: Delweng <delweng@gmail.com>

* core: check of ancientdb, backport bnb-chain/bsc#817

Signed-off-by: Delweng <delweng@gmail.com>

* core/state: read raw borReceipt to backup

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: bor receipt maybe in []Receipt or Receipt RLP format

Signed-off-by: Delweng <delweng@gmail.com>

* core/state: typo and error msg

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: offSet -> offset

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: comment

Signed-off-by: Delweng <delweng@gmail.com>

* cli/snapshot: add prune-block doc

Signed-off-by: Delweng <delweng@gmail.com>

* docs: add prune-block document

Signed-off-by: Delweng <delweng@gmail.com>

* core/rawdb: print wrong bor-receipt length

Signed-off-by: Delweng <delweng@gmail.com>

* internal/cli: add snapshot prune block tests (referenced from bsc's PR)

* improve errors

* cmd, core, eth, internal: fix lint

* internal/cli: refactor snapshot prune block test

* fix linters in tests

* internal/cli: add inspect-ancient-db command, update docs

* pruner: use a generic function for simplification

* internal/cli: fixes for inspect-db command

* internal/cli: improve pruning tests

* core/rawdb: update end block calculation logic in inspect command

* core/rawdb: improve checks db initialisation

* core/rawdb: remove offset check

* update mocks for span, ethdb and add command in makefile

* docs/cli: update docs with inspect command

* go mod tidy

* refactor and resolve conflicts

* resolve more conflicts

* refactor

* explicitly read node for hash scheme

* add check for hash scheme, fix tests

* fix typo

* update docs and add warning

* raise error if pbss is enabled

* revert read raw bor receipt change

* consensus/bor: handle nil header case in get root hash

* address comments

* core/rawdb: check chain continuity by matching parent hash

* core/rawdb: account for pruned ancient blocks

* go mod tidy

* fix tests

* fix tests

---------

Signed-off-by: Delweng <delweng@gmail.com>
Co-authored-by: Delweng <delweng@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants