Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

verkle migration: faster stage2 conversion #196

Conversation

jsign
Copy link
Collaborator

@jsign jsign commented Apr 6, 2023

This PR optimizes stage2 Verkle Tree state migration both in CPU and memory.

The complete changeset involves not only this PR, but also assistance from new methods and other changes in go-verkle (see ethereum/go-verkle#338).

As a quick comparison of performance (my machine, 16 cores and 16GiB of RAM):

  • Previous version: a single file took ~4min using >16GiB of RAM, and it only did tree building.
  • This version: a single file takes ~20s using up to <5GiB of RAM, and it does the full conversion (tree building, serialization, and saving to DB). This is in a context where LevelDB compactions aren't running.

I went somewhat heavy in commenting on the code, but here are some highlights:

  • It runs one file at a time in the most efficient way possible.
  • It uses all available cores for all the needed CPU work. (We could easily allow adding a flag to put a cap on the number of cores allowed to be used for this if needed)
  • The overall code architecture is a pipeline of processing the raw file, building the tree, serializing nodes, and saving them to the database simultaneously. This is carefully crafted to avoid blocking and liberate heap memory as fast as possible.
  • The mentioned go-verkle PR has some new specially created methods for cryptographic optimizations regarding batching LeafNode creation and serialization work.
  • All key/values saved to the database are presorted by keys to help for db compactions.

Considering all this, in my machine, using <5GiB of RAM and 16 cores (and probably even less) we can process data much faster than the database can keep up with the insertions. The mentioned 20s-per-file is in a context where no database compaction runs simultaneously.

I'm doing a full run in the 256 files now, and definitely, LevelDB compactions are dominating overhead. We're running things here with the default flags, but I'm interested in hearing if a different set of options could be better.

I can share some logs about this run since I also included some time elapsed per file and estimated time, which allows us to understand how compactions affect the run. I think the conclusion is that we probably achieved more than enough efficiency now at this stage2 conversion.

jsign added 21 commits April 13, 2023 16:04
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
@jsign jsign force-pushed the jsign/newsteamordered branch from 51c2386 to e5ee037 Compare April 13, 2023 19:08
Copy link
Owner

@gballet gballet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just left a minor comment.

cmd/geth/verkle.go Outdated Show resolved Hide resolved
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
@jsign
Copy link
Collaborator Author

jsign commented Sep 25, 2023

Closing this since we've moved to the overlay tree strategy.
There's probably useful code/logic here, but we can always recover it from the branch if needed.

@jsign jsign closed this Sep 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants