Create a specific calculateRootHash method for BonsaiInMemoryWorldState to improve performance #4568

ahamlat · 2022-10-27T14:22:12Z

PR description

Create a specific calculateRootHash for BonsaiInMemoryWorldState, avoid committing and rollbacking as it is not necessary. This PR parallelize also the accounts update storage in calculateRootHash method.

The yellow line below is the version with the modifications in this PR, the other node is used as a reference node (a baseline) regarding block size and the number of transactions per block.

We have below engine_newPayloadV1 RPC call (Block processing) mean time comparaison between this PR, version 22.7.6 and version 22.7.7

We can notice also that calculateRootHash method is consuming less (real) time on CPU because of the parallelization and the fact we removed commit and rollback method.

Before this PR

After this PR

Fixed Issue(s)

Mitigate #4549

Documentation

I thought about documentation and added the doc-change-required label to this PR if
updates are required.

Changelog

I thought about the changelog and included a changelog update if required.

macfarla · 2022-11-02T02:50:52Z

These files required manual handling in commit da8904d:
both modified: ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/query/BlockchainQueries.java
both modified: ethereum/blockcreation/src/main/java/org/hyperledger/besu/ethereum/blockcreation/AbstractBlockCreator.java
both modified: ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiPersistedWorldState.java
both modified: ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiSnapshotWorldState.java
both modified: ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiWorldStateArchive.java
both modified: ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiWorldStateUpdater.java
both modified: ethereum/core/src/test/java/org/hyperledger/besu/ethereum/MainnetBlockValidatorTest.java
both added: ethereum/core/src/test/java/org/hyperledger/besu/ethereum/bonsai/BonsaiSnapshotWorldStateArchiveTest.java
both modified: plugin-api/build.gradle

Signed-off-by: garyschulte <garyschulte@gmail.com>

… method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

garyschulte · 2022-11-02T03:44:28Z

@ahamlat - I took the liberty of rebasing this off of main. the merges of main and rc3 branch were making the commit history too hard to follow.

Signed-off-by: garyschulte <garyschulte@gmail.com>

macfarla

LGTM

Signed-off-by: garyschulte <garyschulte@gmail.com>

matkt · 2022-11-02T15:35:16Z

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiInMemoryWorldState.java

-    } finally {
-      updater.rollback();
+    final Hash calculatedRootHash = calculateRootHash(localUpdater);
+    return Hash.wrap(calculatedRootHash);


it can be one line

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiPersistedWorldState.java

commit 0d24b22 Author: Fabio Di Fabio <fabio.difabio@consensys.net> Date: Wed Nov 2 21:34:35 2022 +0100 Preparing for next release (hyperledger#4595) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> commit b50f866 Author: ahamlat <ameziane.hamlat@consensys.net> Date: Wed Nov 2 18:05:15 2022 +0100 Create a specific calculateRootHash method for BonsaiInMemoryWorldState to improve performance (hyperledger#4568) * Parallelize some steps in BonsaiPersistedWorldState.calculateRootHash method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Add synchronized on storage flat database remove and update method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Add synchronized on storage flat database remove and update method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Fix this error org.rocksdb.RocksDBException: unknown WriteBatch tag Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * create a specific calculateRootHash for BonsaiInMemoryWorldState class Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * create a specific calculateRootHash for BonsaiInMemoryWorldState class Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Fix nullPointerException on Collections.synchronizedSet Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Use parallelStreams instead of CompletableFuture API Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Modify CHANGELOG.md Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * spotless and synchronizedSet initializaton Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> Signed-off-by: garyschulte <garyschulte@gmail.com> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>

…te to improve performance (hyperledger#4568) * Parallelize some steps in BonsaiPersistedWorldState.calculateRootHash method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Add synchronized on storage flat database remove and update method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Add synchronized on storage flat database remove and update method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Fix this error org.rocksdb.RocksDBException: unknown WriteBatch tag Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * create a specific calculateRootHash for BonsaiInMemoryWorldState class Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * create a specific calculateRootHash for BonsaiInMemoryWorldState class Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Fix nullPointerException on Collections.synchronizedSet Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Use parallelStreams instead of CompletableFuture API Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Modify CHANGELOG.md Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * spotless and synchronizedSet initializaton Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> Signed-off-by: garyschulte <garyschulte@gmail.com> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

…te to improve performance (hyperledger#4568) * Parallelize some steps in BonsaiPersistedWorldState.calculateRootHash method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Add synchronized on storage flat database remove and update method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Add synchronized on storage flat database remove and update method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Fix this error org.rocksdb.RocksDBException: unknown WriteBatch tag Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * create a specific calculateRootHash for BonsaiInMemoryWorldState class Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * create a specific calculateRootHash for BonsaiInMemoryWorldState class Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Fix nullPointerException on Collections.synchronizedSet Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Use parallelStreams instead of CompletableFuture API Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * Modify CHANGELOG.md Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> * spotless and synchronizedSet initializaton Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> Signed-off-by: garyschulte <garyschulte@gmail.com> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>

ahamlat mentioned this pull request Oct 28, 2022

Besu block import an order of magnitude slower than Geth (side by side comparison) #4549

Closed

ahamlat marked this pull request as ready for review October 31, 2022 12:37

garyschulte mentioned this pull request Nov 2, 2022

merge Parallelize calculate root hash into RC3 #4590

Merged

2 tasks

garyschulte added a commit to garyschulte/besu that referenced this pull request Nov 2, 2022

squash commit of diff of main and hyperledger#4568

e890f89

Signed-off-by: garyschulte <garyschulte@gmail.com>

ahamlat added 9 commits November 1, 2022 20:18

Parallelize some steps in BonsaiPersistedWorldState.calculateRootHash…

8cf0df3

… method Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Add synchronized on storage flat database remove and update method

27b7e22

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Add synchronized on storage flat database remove and update method

03f9899

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Fix this error org.rocksdb.RocksDBException: unknown WriteBatch tag

4511b70

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

create a specific calculateRootHash for BonsaiInMemoryWorldState class

c2b097f

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

create a specific calculateRootHash for BonsaiInMemoryWorldState class

8178de2

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Fix nullPointerException on Collections.synchronizedSet

63b0c07

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Use parallelStreams instead of CompletableFuture API

19c9586

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

Modify CHANGELOG.md

55a510c

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

garyschulte force-pushed the parallelize-calculateRootHash branch from da8904d to e650a67 Compare November 2, 2022 03:41

spotless and synchronizedSet initializaton

646482f

Signed-off-by: garyschulte <garyschulte@gmail.com>

garyschulte force-pushed the parallelize-calculateRootHash branch from e650a67 to 646482f Compare November 2, 2022 04:14

garyschulte added a commit to garyschulte/besu that referenced this pull request Nov 2, 2022

re-diff of hyperledger#4568

1f34dc2

Signed-off-by: garyschulte <garyschulte@gmail.com>

garyschulte added a commit to garyschulte/besu that referenced this pull request Nov 2, 2022

re-diff of hyperledger#4568

20067d1

Signed-off-by: garyschulte <garyschulte@gmail.com>

macfarla approved these changes Nov 2, 2022

View reviewed changes

garyschulte added a commit that referenced this pull request Nov 2, 2022

re-diff of #4568 (#4590)

a13ef11

Signed-off-by: garyschulte <garyschulte@gmail.com>

Merge branch 'main' into parallelize-calculateRootHash

1baf5cf

matkt reviewed Nov 2, 2022

View reviewed changes

Merge branch 'main' into parallelize-calculateRootHash

2717ff5

ahamlat merged commit b50f866 into hyperledger:main Nov 2, 2022

macfarla mentioned this pull request Nov 3, 2022

Refactoring of calculate root hash #4551

Closed

non-fungible-nelson mentioned this pull request Jan 26, 2023

Calculate rootHash optimisation for Bonsai #4593

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a specific calculateRootHash method for BonsaiInMemoryWorldState to improve performance #4568

Create a specific calculateRootHash method for BonsaiInMemoryWorldState to improve performance #4568

ahamlat commented Oct 27, 2022 •

edited

Loading

macfarla commented Nov 2, 2022

garyschulte commented Nov 2, 2022

macfarla left a comment

matkt Nov 2, 2022

Create a specific calculateRootHash method for BonsaiInMemoryWorldState to improve performance #4568

Create a specific calculateRootHash method for BonsaiInMemoryWorldState to improve performance #4568

Conversation

ahamlat commented Oct 27, 2022 • edited Loading

PR description

Fixed Issue(s)

Documentation

Changelog

macfarla commented Nov 2, 2022

garyschulte commented Nov 2, 2022

macfarla left a comment

Choose a reason for hiding this comment

matkt Nov 2, 2022

Choose a reason for hiding this comment

ahamlat commented Oct 27, 2022 •

edited

Loading