forked from ethereum/go-ethereum
-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
verkle migration: faster stage2 conversion #196
Closed
jsign
wants to merge
22
commits into
gballet:verkle/convert-through-files-rewrite
from
jsign:jsign/newsteamordered
Closed
verkle migration: faster stage2 conversion #196
jsign
wants to merge
22
commits into
gballet:verkle/convert-through-files-rewrite
from
jsign:jsign/newsteamordered
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
jsign
force-pushed
the
jsign/newsteamordered
branch
from
April 13, 2023 19:08
51c2386
to
e5ee037
Compare
gballet
approved these changes
Apr 17, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just left a minor comment.
Co-authored-by: Guillaume Ballet <3272758+gballet@users.noreply.github.com>
Closing this since we've moved to the overlay tree strategy. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR optimizes stage2 Verkle Tree state migration both in CPU and memory.
The complete changeset involves not only this PR, but also assistance from new methods and other changes in
go-verkle
(see ethereum/go-verkle#338).As a quick comparison of performance (my machine, 16 cores and 16GiB of RAM):
I went somewhat heavy in commenting on the code, but here are some highlights:
Considering all this, in my machine, using <5GiB of RAM and 16 cores (and probably even less) we can process data much faster than the database can keep up with the insertions. The mentioned 20s-per-file is in a context where no database compaction runs simultaneously.
I'm doing a full run in the 256 files now, and definitely, LevelDB compactions are dominating overhead. We're running things here with the default flags, but I'm interested in hearing if a different set of options could be better.
I can share some logs about this run since I also included some time elapsed per file and estimated time, which allows us to understand how compactions affect the run. I think the conclusion is that we probably achieved more than enough efficiency now at this stage2 conversion.