Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix safe optimistic import #4340

Merged
merged 5 commits into from
Jul 26, 2022
Merged

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Jul 23, 2022

Motivation
The safe optimistic import got broken with the PR to optimize the sync speed:
because in safe optimistic import, we used to check if are we syncing near head and the block's parent in forkchoice is not post merge then its not safe to import block with optimistic status.

But this was giving error (EL still syncing and spinned up the lodestar on kiln with weak subjectivity sync, so we are syncing near head), as the parent block would not be found in the forkchoice as we no longer verify then import block by block, now we verify a segment and then import

So updated verification to where the segment's safeness is evaluated and updated and based on that the block import is accepted if its safe.

Also removes the check for justified block being post-merge for safe import as it was a redundant check, as removed in this PR: ethereum/consensus-specs#2881

@g11tech g11tech requested a review from a team as a code owner July 23, 2022 11:28
@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 345fb49 Previous: 3320c20 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.8821 ms/op 2.0988 ms/op 1.37
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 93.870 us/op 72.500 us/op 1.29
BLS verify - blst-native 2.3538 ms/op 1.8568 ms/op 1.27
BLS verifyMultipleSignatures 3 - blst-native 4.7907 ms/op 3.8053 ms/op 1.26
BLS verifyMultipleSignatures 8 - blst-native 10.032 ms/op 8.1928 ms/op 1.22
BLS verifyMultipleSignatures 32 - blst-native 36.753 ms/op 29.702 ms/op 1.24
BLS aggregatePubkeys 32 - blst-native 49.328 us/op 39.264 us/op 1.26
BLS aggregatePubkeys 128 - blst-native 197.31 us/op 152.83 us/op 1.29
getAttestationsForBlock 58.569 ms/op 47.782 ms/op 1.23
isKnown best case - 1 super set check 517.00 ns/op 440.00 ns/op 1.18
isKnown normal case - 2 super set checks 493.00 ns/op 441.00 ns/op 1.12
isKnown worse case - 16 super set checks 501.00 ns/op 448.00 ns/op 1.12
CheckpointStateCache - add get delete 11.348 us/op 8.7630 us/op 1.29
validate gossip signedAggregateAndProof - struct 5.3969 ms/op 4.3798 ms/op 1.23
validate gossip attestation - struct 2.6049 ms/op 2.0889 ms/op 1.25
altair verifyImport mainnet_s3766816:31 10.492 s/op 8.9376 s/op 1.17
pickEth1Vote - no votes 2.8111 ms/op 2.0797 ms/op 1.35
pickEth1Vote - max votes 29.294 ms/op 22.261 ms/op 1.32
pickEth1Vote - Eth1Data hashTreeRoot value x2048 14.806 ms/op 11.216 ms/op 1.32
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 27.359 ms/op 21.032 ms/op 1.30
pickEth1Vote - Eth1Data fastSerialize value x2048 2.0164 ms/op 1.6299 ms/op 1.24
pickEth1Vote - Eth1Data fastSerialize tree x2048 20.417 ms/op 16.645 ms/op 1.23
bytes32 toHexString 1.5580 us/op 1.1280 us/op 1.38
bytes32 Buffer.toString(hex) 873.00 ns/op 685.00 ns/op 1.27
bytes32 Buffer.toString(hex) from Uint8Array 1.2630 us/op 987.00 ns/op 1.28
bytes32 Buffer.toString(hex) + 0x 865.00 ns/op 726.00 ns/op 1.19
Object access 1 prop 0.48700 ns/op 0.36500 ns/op 1.33
Map access 1 prop 0.33900 ns/op 0.29400 ns/op 1.15
Object get x1000 15.871 ns/op 18.025 ns/op 0.88
Map get x1000 0.95400 ns/op 0.99800 ns/op 0.96
Object set x1000 139.58 ns/op 118.31 ns/op 1.18
Map set x1000 92.906 ns/op 71.432 ns/op 1.30
Return object 10000 times 0.43320 ns/op 0.36700 ns/op 1.18
Throw Error 10000 times 6.7019 us/op 5.8871 us/op 1.14
enrSubnets - fastDeserialize 64 bits 3.7860 us/op 2.9370 us/op 1.29
enrSubnets - ssz BitVector 64 bits 921.00 ns/op 752.00 ns/op 1.22
enrSubnets - fastDeserialize 4 bits 501.00 ns/op 415.00 ns/op 1.21
enrSubnets - ssz BitVector 4 bits 951.00 ns/op 739.00 ns/op 1.29
prioritizePeers score -10:0 att 32-0.1 sync 2-0 124.88 us/op 93.048 us/op 1.34
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 150.19 us/op 135.75 us/op 1.11
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 312.03 us/op 227.48 us/op 1.37
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 651.88 us/op 393.01 us/op 1.66
prioritizePeers score 0:0 att 64-1 sync 4-1 578.32 us/op 466.15 us/op 1.24
RateTracker 1000000 limit, 1 obj count per request 227.55 ns/op 188.92 ns/op 1.20
RateTracker 1000000 limit, 2 obj count per request 168.35 ns/op 142.90 ns/op 1.18
RateTracker 1000000 limit, 4 obj count per request 151.38 ns/op 116.47 ns/op 1.30
RateTracker 1000000 limit, 8 obj count per request 130.50 ns/op 105.51 ns/op 1.24
RateTracker with prune 7.0070 us/op 4.4910 us/op 1.56
array of 16000 items push then shift 5.3930 us/op 3.2036 us/op 1.68
LinkedList of 16000 items push then shift 20.907 ns/op 29.065 ns/op 0.72
array of 16000 items push then pop 299.17 ns/op 274.49 ns/op 1.09
LinkedList of 16000 items push then pop 18.072 ns/op 23.875 ns/op 0.76
array of 24000 items push then shift 7.9343 us/op 4.5810 us/op 1.73
LinkedList of 24000 items push then shift 21.476 ns/op 30.031 ns/op 0.72
array of 24000 items push then pop 251.20 ns/op 220.40 ns/op 1.14
LinkedList of 24000 items push then pop 17.869 ns/op 23.656 ns/op 0.76
intersect bitArray bitLen 8 12.342 ns/op 11.762 ns/op 1.05
intersect array and set length 8 242.64 ns/op 167.58 ns/op 1.45
intersect bitArray bitLen 128 70.312 ns/op 62.045 ns/op 1.13
intersect array and set length 128 2.7083 us/op 2.2282 us/op 1.22
pass gossip attestations to forkchoice per slot 3.8880 ms/op 3.2418 ms/op 1.20
computeDeltas 3.4539 ms/op 3.6937 ms/op 0.94
computeProposerBoostScoreFromBalances 888.04 us/op 908.49 us/op 0.98
altair processAttestation - 250000 vs - 7PWei normalcase 5.9644 ms/op 3.9213 ms/op 1.52
altair processAttestation - 250000 vs - 7PWei worstcase 8.6435 ms/op 6.0644 ms/op 1.43
altair processAttestation - setStatus - 1/6 committees join 274.44 us/op 208.26 us/op 1.32
altair processAttestation - setStatus - 1/3 committees join 503.17 us/op 396.00 us/op 1.27
altair processAttestation - setStatus - 1/2 committees join 727.34 us/op 561.41 us/op 1.30
altair processAttestation - setStatus - 2/3 committees join 932.08 us/op 706.09 us/op 1.32
altair processAttestation - setStatus - 4/5 committees join 1.2598 ms/op 997.91 us/op 1.26
altair processAttestation - setStatus - 100% committees join 1.5329 ms/op 1.1830 ms/op 1.30
altair processBlock - 250000 vs - 7PWei normalcase 34.200 ms/op 30.000 ms/op 1.14
altair processBlock - 250000 vs - 7PWei normalcase hashState 44.021 ms/op 41.701 ms/op 1.06
altair processBlock - 250000 vs - 7PWei worstcase 98.267 ms/op 85.829 ms/op 1.14
altair processBlock - 250000 vs - 7PWei worstcase hashState 124.78 ms/op 98.974 ms/op 1.26
phase0 processBlock - 250000 vs - 7PWei normalcase 6.3621 ms/op 4.3909 ms/op 1.45
phase0 processBlock - 250000 vs - 7PWei worstcase 61.870 ms/op 47.163 ms/op 1.31
altair processEth1Data - 250000 vs - 7PWei normalcase 1.1932 ms/op 862.09 us/op 1.38
Tree 40 250000 create 1.1735 s/op 733.00 ms/op 1.60
Tree 40 250000 get(125000) 343.49 ns/op 289.16 ns/op 1.19
Tree 40 250000 set(125000) 3.4890 us/op 2.3984 us/op 1.45
Tree 40 250000 toArray() 37.473 ms/op 34.721 ms/op 1.08
Tree 40 250000 iterate all - toArray() + loop 37.986 ms/op 34.743 ms/op 1.09
Tree 40 250000 iterate all - get(i) 136.95 ms/op 113.93 ms/op 1.20
MutableVector 250000 create 18.164 ms/op 18.435 ms/op 0.99
MutableVector 250000 get(125000) 15.605 ns/op 13.094 ns/op 1.19
MutableVector 250000 set(125000) 1.0505 us/op 642.28 ns/op 1.64
MutableVector 250000 toArray() 8.3071 ms/op 7.8485 ms/op 1.06
MutableVector 250000 iterate all - toArray() + loop 8.5725 ms/op 8.0488 ms/op 1.07
MutableVector 250000 iterate all - get(i) 3.8349 ms/op 3.2846 ms/op 1.17
Array 250000 create 8.1940 ms/op 7.2277 ms/op 1.13
Array 250000 clone - spread 5.1652 ms/op 4.0619 ms/op 1.27
Array 250000 get(125000) 2.1740 ns/op 1.6150 ns/op 1.35
Array 250000 set(125000) 2.1440 ns/op 1.6190 ns/op 1.32
Array 250000 iterate all - loop 149.02 us/op 169.72 us/op 0.88
effectiveBalanceIncrements clone Uint8Array 300000 131.87 us/op 86.406 us/op 1.53
effectiveBalanceIncrements clone MutableVector 300000 1.6900 us/op 1.2380 us/op 1.37
effectiveBalanceIncrements rw all Uint8Array 300000 294.60 us/op 252.61 us/op 1.17
effectiveBalanceIncrements rw all MutableVector 300000 331.09 ms/op 219.24 ms/op 1.51
phase0 afterProcessEpoch - 250000 vs - 7PWei 201.91 ms/op 185.00 ms/op 1.09
phase0 beforeProcessEpoch - 250000 vs - 7PWei 72.663 ms/op 79.922 ms/op 0.91
altair processEpoch - mainnet_e81889 672.75 ms/op 597.39 ms/op 1.13
mainnet_e81889 - altair beforeProcessEpoch 186.12 ms/op 167.85 ms/op 1.11
mainnet_e81889 - altair processJustificationAndFinalization 65.401 us/op 23.089 us/op 2.83
mainnet_e81889 - altair processInactivityUpdates 11.500 ms/op 11.844 ms/op 0.97
mainnet_e81889 - altair processRewardsAndPenalties 108.86 ms/op 101.70 ms/op 1.07
mainnet_e81889 - altair processRegistryUpdates 14.085 us/op 4.1380 us/op 3.40
mainnet_e81889 - altair processSlashings 3.7240 us/op 976.00 ns/op 3.82
mainnet_e81889 - altair processEth1DataReset 3.6980 us/op 1.2140 us/op 3.05
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.4551 ms/op 2.4334 ms/op 1.01
mainnet_e81889 - altair processSlashingsReset 21.076 us/op 10.137 us/op 2.08
mainnet_e81889 - altair processRandaoMixesReset 24.530 us/op 9.3420 us/op 2.63
mainnet_e81889 - altair processHistoricalRootsUpdate 3.9920 us/op 1.2780 us/op 3.12
mainnet_e81889 - altair processParticipationFlagUpdates 15.264 us/op 3.0650 us/op 4.98
mainnet_e81889 - altair processSyncCommitteeUpdates 3.0360 us/op 979.00 ns/op 3.10
mainnet_e81889 - altair afterProcessEpoch 205.99 ms/op 194.01 ms/op 1.06
phase0 processEpoch - mainnet_e58758 624.78 ms/op 560.44 ms/op 1.11
mainnet_e58758 - phase0 beforeProcessEpoch 274.37 ms/op 267.70 ms/op 1.02
mainnet_e58758 - phase0 processJustificationAndFinalization 63.191 us/op 25.608 us/op 2.47
mainnet_e58758 - phase0 processRewardsAndPenalties 150.21 ms/op 159.34 ms/op 0.94
mainnet_e58758 - phase0 processRegistryUpdates 33.767 us/op 11.877 us/op 2.84
mainnet_e58758 - phase0 processSlashings 3.2890 us/op 978.00 ns/op 3.36
mainnet_e58758 - phase0 processEth1DataReset 3.5400 us/op 1.0280 us/op 3.44
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 3.1097 ms/op 2.1547 ms/op 1.44
mainnet_e58758 - phase0 processSlashingsReset 20.627 us/op 5.8130 us/op 3.55
mainnet_e58758 - phase0 processRandaoMixesReset 24.836 us/op 6.9770 us/op 3.56
mainnet_e58758 - phase0 processHistoricalRootsUpdate 4.0210 us/op 1.2190 us/op 3.30
mainnet_e58758 - phase0 processParticipationRecordUpdates 23.776 us/op 5.7580 us/op 4.13
mainnet_e58758 - phase0 afterProcessEpoch 168.06 ms/op 158.81 ms/op 1.06
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.5685 ms/op 2.5653 ms/op 1.00
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.9383 ms/op 3.1037 ms/op 0.95
altair processInactivityUpdates - 250000 normalcase 51.292 ms/op 46.760 ms/op 1.10
altair processInactivityUpdates - 250000 worstcase 60.053 ms/op 56.383 ms/op 1.07
phase0 processRegistryUpdates - 250000 normalcase 27.771 us/op 11.172 us/op 2.49
phase0 processRegistryUpdates - 250000 badcase_full_deposits 495.77 us/op 409.90 us/op 1.21
phase0 processRegistryUpdates - 250000 worstcase 0.5 254.67 ms/op 233.63 ms/op 1.09
altair processRewardsAndPenalties - 250000 normalcase 102.27 ms/op 142.12 ms/op 0.72
altair processRewardsAndPenalties - 250000 worstcase 151.29 ms/op 130.08 ms/op 1.16
phase0 getAttestationDeltas - 250000 normalcase 14.552 ms/op 12.863 ms/op 1.13
phase0 getAttestationDeltas - 250000 worstcase 15.821 ms/op 12.911 ms/op 1.23
phase0 processSlashings - 250000 worstcase 7.0119 ms/op 5.5865 ms/op 1.26
altair processSyncCommitteeUpdates - 250000 354.44 ms/op 277.68 ms/op 1.28
BeaconState.hashTreeRoot - No change 642.00 ns/op 475.00 ns/op 1.35
BeaconState.hashTreeRoot - 1 full validator 71.952 us/op 62.393 us/op 1.15
BeaconState.hashTreeRoot - 32 full validator 743.91 us/op 650.56 us/op 1.14
BeaconState.hashTreeRoot - 512 full validator 8.0183 ms/op 6.7646 ms/op 1.19
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 100.47 us/op 81.436 us/op 1.23
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3850 ms/op 1.2239 ms/op 1.13
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 18.568 ms/op 16.390 ms/op 1.13
BeaconState.hashTreeRoot - 1 balances 70.901 us/op 64.280 us/op 1.10
BeaconState.hashTreeRoot - 32 balances 683.81 us/op 646.80 us/op 1.06
BeaconState.hashTreeRoot - 512 balances 6.1647 ms/op 5.8644 ms/op 1.05
BeaconState.hashTreeRoot - 250000 balances 93.453 ms/op 94.808 ms/op 0.99
aggregationBits - 2048 els - zipIndexesInBitList 33.530 us/op 42.427 us/op 0.79
regular array get 100000 times 54.312 us/op 67.458 us/op 0.81
wrappedArray get 100000 times 53.046 us/op 67.471 us/op 0.79
arrayWithProxy get 100000 times 33.276 ms/op 29.058 ms/op 1.15
ssz.Root.equals 528.00 ns/op 564.00 ns/op 0.94
byteArrayEquals 554.00 ns/op 556.00 ns/op 1.00
shuffle list - 16384 els 11.012 ms/op 11.197 ms/op 0.98
shuffle list - 250000 els 154.46 ms/op 164.53 ms/op 0.94
processSlot - 1 slots 15.853 us/op 14.050 us/op 1.13
processSlot - 32 slots 2.2721 ms/op 1.9349 ms/op 1.17
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 546.07 us/op 437.83 us/op 1.25
getCommitteeAssignments - req 1 vs - 250000 vc 5.7025 ms/op 5.3011 ms/op 1.08
getCommitteeAssignments - req 100 vs - 250000 vc 7.8670 ms/op 7.3553 ms/op 1.07
getCommitteeAssignments - req 1000 vs - 250000 vc 8.7249 ms/op 7.8562 ms/op 1.11
computeProposers - vc 250000 19.782 ms/op 18.772 ms/op 1.05
computeEpochShuffling - vc 250000 170.70 ms/op 166.11 ms/op 1.03
getNextSyncCommittee - vc 250000 340.10 ms/op 275.77 ms/op 1.23

by benchmarkbot/action

// It becomes optimistically safefor following blocks if a post-merge block is deemed fit
// for import. If it would not have been safe verifyBlockExecutionPayload would throw error
// and we would not be here.
isOptimisticallySafe ||= executionStatus !== ExecutionStatus.PreMerge;
Copy link
Contributor

@dapplion dapplion Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While the ||= operand is pretty cool, can you use a regular if statement since it's more readable?

// - we have transitioned to post-merge world or
// - all the blocks are way behind the current slot
// - or we have already imported a post-merge parent of first block of this chain in forkchoice
const lastBlock = blocks[blocks.length - 1];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this function ensure that there are more than 0 blocks?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be somewhere from where its called? because it didn't give any type issues, but will return early to be safe

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a throw just to be safe

if (!isOptimisticallySafe) {
const firstBlock = blocks[0];
const parentRoot = toHexString(firstBlock.message.parentRoot);
const parentBlock = chain.forkChoice.getBlockHex(parentRoot);
Copy link
Contributor

@dapplion dapplion Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could return the parent block of the segment in sanity checks

and re-use here

g11tech and others added 3 commits July 26, 2022 12:49
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
@g11tech g11tech enabled auto-merge (squash) July 26, 2022 10:06
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Beautifully documented too thank you!!! ❤️

@g11tech g11tech merged commit 384e393 into unstable Jul 26, 2022
@g11tech g11tech deleted the g11tech/fix-safe-optimistic-import branch July 26, 2022 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants