-
Notifications
You must be signed in to change notification settings - Fork 839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Keep Worldstate Storage open for Bonsai archive latest layer #5039
Keep Worldstate Storage open for Bonsai archive latest layer #5039
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes sense to me. changelog entry?
I tested eth_getBalance several times with this PR and the result matches what I can see in Etherscan or by executing eth_getBalance on 22.10.3 (which does not have the regression). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just asked some questions to understand
localUpdater, | ||
newWorldStateRootHash, | ||
blockHeader, | ||
(BonsaiPersistedWorldState) this.copy()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this copy can impact the performance ? I see some synchronized method , call to DB . if @ahamlat can confirm that it is safe
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, an alternate implementation is to directly subscribe and unsubscribe to the underlying worldstate storage. This is cleaner code-wise, but if there is a performance impact we can directly sub/unsub to the storage itself.
@@ -75,9 +75,29 @@ public Optional<BonsaiWorldView> getNextWorldView() { | |||
} | |||
|
|||
public void setNextWorldView(final Optional<BonsaiWorldView> nextWorldView) { | |||
maybeUnSubscribe(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we close the nextview, how will the layeredworldstate be able to ask it to retrieve the different values? Is it going to recreate it when needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is to close/unsubscribe from the worldstate this layer was initially created with, so we do not leak any snapshots. since this is a mutator, this is the last opportunity we will have to unsubscribe, typically when the BonsaiWorldStateArchive is responding to a BlockAddedEvent, and changes the nextWorldView.
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiLayeredWorldState.java
Show resolved
Hide resolved
c4274fb
to
2f1faa2
Compare
Signed-off-by: garyschulte <garyschulte@gmail.com>
* unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
eaba4b9
to
3e92f02
Compare
…dger#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements Signed-off-by: garyschulte <garyschulte@gmail.com>
…dger#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Justin Florentine <justin+github@florentine.us>
…dger#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements Signed-off-by: garyschulte <garyschulte@gmail.com>
* Fix transaction pool issue (#4964) * Cache empty slots (#4874) * clear after each block and copy during clone * fix transaction pool issue * add block replay * support in-memory snapshots * Keep Worldstate Storage open for Bonsai archive latest layer (#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements * Avoid triggering a calculate root hash when empty slot cache is not empty. * use the updater cache to get an account during block processing (#4897) * Worldstate-only resync behavior (#4875) * use debug rpc endpoint to resync worldstate * Reset transaction pool state every time the initial sync is done * init heal code --------- Signed-off-by: Karim TAAM <karim.t2am@gmail.com> Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Justin Florentine <justin+github@florentine.us> Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> Signed-off-by: ahamlat <ameziane.hamlat@consensys.net> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: matkt <karim.t2am@gmail.com> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: ahamlat <ameziane.hamlat@consensys.net> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
…yperledger#5039)" This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
…yperledger#5039)" (hyperledger#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…yperledger#5039)" (hyperledger#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…yperledger#5039)" (hyperledger#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: garyschulte <garyschulte@gmail.com>
…yperledger#5039)" (hyperledger#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: garyschulte <garyschulte@gmail.com>
* Prepare for version 23.1.1-SNAPSHOT (#5067) Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Add getPayloadBodiesByRangeV1 and getPayloadBodiesByHash engine methods (#4980) * Add engine get payload body methods and test Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add header Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com> * Update result struct & add test Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com> * Change constant to use upper case Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add PayloadBody class and withdrawals to response of methods Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add unit tests Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add changelog Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * spotless Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add check to prevent returning trailing null results past the head Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add test to check trailing null post head scenario Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Split tests into pre and post shanghai Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * spotless Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Rename methods Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Use getName() to log method name Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * spotless Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Rename variable Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Call constructor directly Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Fix ByHash json parsing Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Fix json parsing again Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Add check to prevent unnecessary queries Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Refactor method Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add new error code for EngineGetPayloadBodies methods Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add return error for request above the API limit Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add constructor for empty response Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * add check for number of blocks requested and for requests of post head Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add test to check error code when request exceeds API limits Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * add constant for max blocks allowed per request Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * spotless Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Fix some nits Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add invalid params check Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add tests for invalid params check Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Fix test and spotless Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Revert "Fix json parsing again" This reverts commit 558d325. Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Revert "Fix ByHash json parsing Signed-off-by: Simon Dudley <simon.dudley@consensys.net>" This reverts commit 663e11e Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Use UnsignedLongParameter to cast params of the request Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add optional withdrawals to the NewPayload log (#5021) Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * kubernetes and errorprone - update versions (#5013) * update errorprone and kubernetes versions * fixed errorprone issues in prod cod * fixed errorprone issues in test code --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Rename JsonRpcService to EngineJsonRpcService (#5036) * rename JsonRpcService to EngineJsonRpcService Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> * Params should be single item of array type, not outer array of strings (#5037) Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Add EIP-2537 (BLS12-381 curve precompiles) to Cancun (#5017) Add the BLS curve precompiles into the registry for cancun. All of the curve precompiles have been here since berlin, so this is just wiring them in. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Use UnsignedLongParameter to cast params of the request Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add optional withdrawals to the NewPayload log (#5021) Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * kubernetes and errorprone - update versions (#5013) * update errorprone and kubernetes versions * fixed errorprone issues in prod cod * fixed errorprone issues in test code --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Rename JsonRpcService to EngineJsonRpcService (#5036) * rename JsonRpcService to EngineJsonRpcService Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> * Params should be single item of array type, not outer array of strings (#5037) Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Add EIP-2537 (BLS12-381 curve precompiles) to Cancun (#5017) Add the BLS curve precompiles into the registry for cancun. All of the curve precompiles have been here since berlin, so this is just wiring them in. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Convert start and count from hex to match JSON-RPC Spec standard Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> --------- Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Co-authored-by: Zhenyang Shi <wcgcyx@gmail.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Daniel Lehrner <daniel.lehrner@consensys.net> Co-authored-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * Revert "Keep Worldstate Storage open for Bonsai archive latest layer (#5039)" (#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * Support post merge forks at genesis for hive tests (#5019) Signed-off-by: Jason Frame <jason.frame@consensys.net> * Fix manifest docker not skipping interim builds for RCs (#5068) Signed-off-by: Jason Frame <jason.frame@consensys.net> * Fix PoS checkpoint validation (#5081) * change validation to lessThan Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Change exception message to PoS instead of Near head Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add unit test and fix message of existing unit test Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> --------- Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * moves check for init code length before balance check (#5077) Signed-off-by: Justin Florentine <justin+github@florentine.us> * bump revision for 23.1.x release branch Signed-off-by: garyschulte <garyschulte@gmail.com> * Burn in build of 23.1.0 (#5093) * rebase off of burn-in release, remove unreleased 23.1.0-RC2 from CHANGELOG Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * re-default global max rpc batch size to 1k (#5104) (#5105) * default global max rpc batch size to 1000 for now Signed-off-by: garyschulte <garyschulte@gmail.com> * Fix Layered World State issue (#5076) * Use the copy during prepareTrieLog instead of saveTrieLog * add final flag on BonsaiWorldStateUpdater * Use a copy of BonsaiInMemoryWorldState inside prepareTrieLog * add link to persisted worldstate storage * fix tests * Make a copy of the worldstate after committing changes to the trielog * spotless + remove maybeUnSubscribe in setNextWorldView * subscribe storage on layered worldstate * fix null issue * not close layered worldstate during getAccount * clean code * Add changelog entry --------- Signed-off-by: ahamlat <ameziane.hamlat@consensys.net> Signed-off-by: Karim TAAM <karim.t2am@gmail.com> Co-authored-by: Karim TAAM <karim.t2am@gmail.com> * Add shanghaiTime to sepolia (#5088) Update and fix forkId tests Move timestamp forks from getForkBlockNumbers to getForkBlockTimestamps in JsonGenesisConfigOptions - this ultimately gets used to popoulate the ForkIdManager which handles lists of blocks and timestamps the same way so this hasn't changed any actual behaviour, but rather supports the test fixes. Implement TransitionProtocolSchedule.streamMilestoneBlocks as a concatenation of blockNumbers++blockTimestamps. This may have been a latent bug since it's used to update the node record when a fork transition occurs. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * If a PoS block creation repetition takes less than a configurable dur… (#5048) * If a PoS block creation repetition takes less than a configurable duration, then waits before next repetition Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Update CHANGELOG Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Add unit test Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Update besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Update besu/src/main/java/org/hyperledger/besu/cli/options/unstable/MiningOptions.java Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> --------- Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Allow dashes in ethstats password (#5090) Signed-off-by: Simon Dudley <simon.dudley@consensys.net> * reintroduce checking of block height for certain tasks when we are not PoS (Revert PR#3911) (#5083) * reintroduce checking of block height for certain tasks when we are not PoS (Revert PR#3911) Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> * Allow other users to read the /opt/besu dir when using docker (#5092) Signed-off-by: Rafael Matias <rafael@skyle.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Only use MAINNET version of KZG (#5095) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * add more context to exception messages and debug logging (#5066) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Fix block value calculation (#5100) * Add gasUsed calculation Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Fix unit test Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Fix typo Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> --------- Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add 23.1.1-RC1 changelog Signed-off-by: Jason Frame <jason.frame@consensys.net> * Change version to 23.1.1-RC1 Signed-off-by: Jason Frame <jason.frame@consensys.net> --------- Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Zhenyang Shi <wcgcyx@gmail.com> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Signed-off-by: Jason Frame <jason.frame@consensys.net> Signed-off-by: Justin Florentine <justin+github@florentine.us> Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> Signed-off-by: Karim TAAM <karim.t2am@gmail.com> Signed-off-by: ahamlat <ameziane.hamlat@consensys.net> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Signed-off-by: Rafael Matias <rafael@skyle.net> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com> Co-authored-by: Zhenyang Shi <wcgcyx@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Daniel Lehrner <daniel.lehrner@consensys.net> Co-authored-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: ahamlat <ameziane.hamlat@consensys.net> Co-authored-by: Karim TAAM <karim.t2am@gmail.com> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Rafael Matias <rafael@skyle.net>
…dger#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements Signed-off-by: garyschulte <garyschulte@gmail.com>
…yperledger#5039)" (hyperledger#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…dger#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements Signed-off-by: garyschulte <garyschulte@gmail.com>
…yperledger#5039)" (hyperledger#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
…dger#5039) * bonsai layered worldstate subscription * unsubscribe from worldstatestorage on close of BonsaiLayeredWorldState * minor txpool logging improvements Signed-off-by: garyschulte <garyschulte@gmail.com>
…yperledger#5039)" (hyperledger#5073) This reverts commit e715010. Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
PR description
Fixes an issue where the latest block in BonsaiWorldStateArchive had a closed worldstate. Persisting BonsaiInMemoryWorldState instances were creating entries in the archive with a snapshot worldstate that would subsequently be closed. The net result was that queries using the worldstate for the
latest
block would frequently return Optional.empty() when accessing the closed worldstate.this PR:
Fixed Issue(s)
fixes #5038
Documentation
doc-change-required
label to this PR ifupdates are required.
Changelog