New metric filtering missed blocks #3927
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
twoeths
changed the title
Codecov Report
@@ Coverage Diff @@
## master #3927 +/- ##
==========================================
+ Coverage 36.32% 37.22% +0.89%
==========================================
Files 324 325 +1
Lines 9059 9924 +865
Branches 1453 1696 +243
==========================================
+ Hits 3291 3694 +403
- Misses 5597 6007 +410
- Partials 171 223 +52 |
Performance Report✔️ no performance regression detected Full benchmark results
|
|
New metric looks like
|
dadepo
reviewed
Apr 15, 2022
| }); | ||
| this.syncContributionAndProofPool.prune(head.slot); | ||
| this.seenContributionAndProof.prune(head.slot); | ||
| this.metrics?.headSlot.set(head.slot); | ||
| this.metrics?.gossipBlock.elappsedTimeTillBecomeHead.observe(delaySec); |
dadepo
reviewed
Apr 15, 2022
| @@ -181,9 +184,11 @@ export async function onBlock( | |||
|
|
|||
| this.reprocessController.onBlockImported({slot: block.message.slot, root: blockRoot}, advancedSlot); | |||
|
|
|||
| const delaySec = Date.now() / 1000 - (this.genesisTime + block.message.slot * this.config.SECONDS_PER_SLOT); | |||
There was a problem hiding this comment.
Looks like this logic to calculate delaySec is already provided in clock.secFromSlot(slot: Slot)
There was a problem hiding this comment.
@tuyennhv Are you sure this metric won't be triggered during sync, and unknown block root fetches?
There was a problem hiding this comment.
I added BlockSource enum and track that in the new metric, please have a look
dapplion
reviewed
Apr 16, 2022
| elappsedTimeTillBecomeHead: register.histogram({ | ||
| name: "lodestar_gossip_block_elappsed_time_till_become_head", | ||
| help: "Time elappsed between block slot time and the time block becomes head", | ||
| buckets: [0.1, 1, 10], |
There was a problem hiding this comment.
It's very unlikely for a block to come exactly between 0 and 0.1 seconds, better bucket values would be:
[0.5, 1, 2, 4, 6, 12]
This reverts commit 5fe6220.
dapplion
approved these changes
May 9, 2022
There was a problem hiding this comment.
@tuyennhv I reverted your commit adding the block source since I think it's valuable to track blocks that become head late due to unknown block sync too. For example, a block through gossip can trigger unknown block sync and its parent won't be registered like it came from gossip. The PR adds less complexity too. If the metrics are not clear we can add the source thing
dapplion
added a commit
that referenced
this pull request
May 30, 2022
* New metric filtering missed blocks (#3927) * Log block delay second * Add elappsedTimeTillBecomeHead metric * Add 'till become head' metric to dashboard * chore: correct the metric name to elapsedTimeTillBecomeHead * Add and use secFromSlot to clock * Track block source * Revert "Track block source" This reverts commit 5fe6220. * Update bucket values * Limit how old blocks are tracked in elapsedTimeTillBecomeHead * Simplify secFromSlot Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com> * Fix the terminal validations of the merge block (#3984) * Fix the terminal validations of the merge block * activate merge transition block spec tests * some comments to explain the merge block validations movement * Extend error messages when voluntary exit errors because of present of lockfile (#3935) * Extend error and Clean up * Only showing the message to use --force to override in case of voluntary exit * Simplify gitData and version guessing (#3992) Don't print double slash in version string Dont add git-data.json to NPM releases Write git-data.json only in from source docker build Remove numCommits Test git-data.json generation from within the test Move comment Revert "Dont add git-data.json to NPM releases" This reverts commit 5fe2d38. Simplify gitData and version guessing Run cmd * Activate ex-ante fork-choice spec tests (#4003) * Prepare custom version on next release (#3990) * Prepare custom version on next release * Test in branch * Don't set version in advance * Remove --canary flag * Change and commit version * Setup git config * Revert temp changes * Lightclient e2e: increase validator client (#4006) * Bump to v0.37.0 nightly builds (#4013) * Guarantee full spec tests coverage (#4012) * Ensure all spec tests are run * Fix general bls tests * Improve docs of specTestIterator * Fix fork_choice tests * Remove Check spec tests step * Add merge transition/finalization banners (#3963) * Add merge transition/finalization banners * fix signatures * Benchmark initial sync (#3995) * Basic range sync perf test * Benchmark initial sync * Add INFURA_ETH2_CREDENTIALS to benchmark GA * Download test cache file from alternative source * Re-org beforeValue and testCase helpers * Break light-client - state-transition test dependency * Revert adding downloadTestCacheFile * Download files from a Github release * Clarify #3977 with unbounded uint issue (#4018) * Update mainnet-shadow-5 configs (#4021) * Bump moment from 2.29.1 to 2.29.2 (#3901) Bumps [moment](https://github.com/moment/moment) from 2.29.1 to 2.29.2. - [Release notes](https://github.com/moment/moment/releases) - [Changelog](https://github.com/moment/moment/blob/develop/CHANGELOG.md) - [Commits](moment/moment@2.29.1...2.29.2) --- updated-dependencies: - dependency-name: moment dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Implement support for validator next-epoch proposer duties (#3782) * Implementation to be able to get block proposer an epoch ahead - still need optimization * revert changes made to waitForSlot * caching the results of computing future proposers. Also extended test * using effectiveBalanceIncrements from state instead of recomputing it * fix lint errors * revert check not needed in getBeaconProposer * Update tests to include assertion messages * Move caching of next proposer duties to BeaconChain class * Delete the block proposer previously cached when next proposer was requested at current epoch * moved next epoch proposers from the chain to the state * Compute next proposer on demand and cache * Fix lint errors * update implementation to work with changes from master * caching epoch seed in context so that getNextEpochBeaconProposer can be independent of state * Revert "caching epoch seed in context so that getNextEpochBeaconProposer can be independent of state" This reverts commit 02a722a. * caching epoch seed in context so that getNextEpochBeaconProposer can be independent of state * removing the need to delete from nextEpochProposers in call to getBeaconProposer * no need to recompute currrentProposerSeed again * Revert "no need to recompute currrentProposerSeed again" This reverts commit b6b1b8c. * removed empty file left after fixing merge conflicts * remove some unnecessary variable from the epoch context. * add some comments * Fix lint * import from the right location * Review PR * Merge imports * Delete get proposers api impl test * Remove duplicated comment Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com> * Extend timeout for gitData unit test (#4026) * Fix readAndGetGitData (#4025) * Ensure light client update is in a single period (#4029) * Handle merge block fetch error (#4016) * Handle merge block fetch error * Log errors on fetch errors for terminal pow * docs: Update nodeJS minimum requirement (#4037) * Remove child_process call in gitData before step (#4033) * Oppool aggregates use BitArray only for set logic (#4034) * Use BitArrays for aggregate merging * Test intersectUint8Arrays * Review PR * Update tests * Remove un-used code * Modify gossipsub params following consensus spec v1.1.10 (#4011) * Modify gossipsub params following consensus spec v1.1.10 * Specify GOSSIPSUB_HEARTBEAT_INTERVAL as a constant * Throw a more informative error on invalid keystore (#4022) * Throw a more informative error on invalid keystore * Make error more descriptive * Use template string * Update keys.ts * Update keys.ts Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com> * Ignore gossip AggregateAndProof if aggregate is seen (#4019) * Ignore gossip AggregateAndProof if aggregate is seen * Check for non-strict superset of seen attestation data * Fix validateGossipAggregateAndProof benchmark test * Fix import * Ultilize intersectUint8Arrays() * Implement SeenContributionAndProof.participantsKnown * Add metrics to seen cache * Add perf tests * Change method name to isSuperSetOrEqual() * Refactor metric names * Specify lerna exact version for release-nightly workflow (#4049) * Add ropsten network (#4051) * Force all packages to be versioned for exact (#4052) * Update discv5 to v0.7.1 (#4044) * Add ability to update the fee recipient for execution via beacon and/or validator defaults (#3958) * Add and use a default fee recipient for a validator process * transfer the proposer cache to beacon chain * mock chain fixes * test and perf fixes * fee recipient validation change * track and use free recipient as string instead of ExecutionAddress * fix unit test * fix merge test * use dummy address * refac and add proposer cache pruning * tests for beacon proposer cache * merge interop fee recipient check * fix the optional * feeRecipient confirmation and small refac * add the missing map * add flag to enable strict fee recipient check * Small refactor to setup merge for ropsten using baked in configs (#4053) * Issue advance fcU for builing the EL block (#3965) rebaseing to the refactored prepare beacon proposer refac payload id cache as separate class and add pruning issue payload fcus if synced rename issueNext.. to maybeIssueNext... * Simplify release process (#4030) * Simplify release process * Remove old postrelease script * Add lerna version check * Tweak RELEASE.md * Add force-publish to lerna version command * Update the proposer boost percentage to 40% (#4055) * ESM Support (#3978) * ESM changes * Fix root lodestar script * Fix some linter errors * trying directly re-exporting under an alias from networks module * Fix types exports * Fix more linter errors * Fix spec test download * Update bls to 7.1.0 * Fix spec tests * temp reverting eslint parser option to 10 and disabling the check of .js file extenstion. Should fix lint errors * temp commented out file-extension-in-import * Disable readme checks * Fix check-build * Fix params e2e tests * Bump @chainsafe/threads * Bump bls to v7.1.1 * Add timeouts after node initialization but before sim test run * Tweak timeouts * Tweak timeout * Tweak sim merge timeout * Tweak sim merge timeout * Tweak sim merge timeout * Tweak sim merge timeout * Add more timeouts * Add another timeout * Fix linter errors * Fix some tests * Fix some linter errors and spec tests * Fix benchmarks * Fix linter errors * Update each bls dependency * Tweak timeouts * Add another timeout * More timeouts * Fix bls pool size * Set root package.json to ESM * Remove old linter comment * Revert "Set root package.json to ESM" This reverts commit 347b0fd. * Remove stray file (probably old) * Undo unnecessary diff * Add comment on __dirname replacement * Import type @chainsafe/bls/types * Use lodestar path imports * Revert multifork to lodestar package * Format .mocharc.yaml * Use same @chainsafe/as-sha256 version * Fix lodash path imports * Use src instead of lib * Load db metrics * Remove experimental-specifier-resolution * Remove lodestat/chain export * Add stray missing file extension * Revert ValidatorDir changes * Fix stray missing file extensions * Fix check-types Co-authored-by: Dadepo Aderemi <dadepo@gmail.com> Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com> * chore(release): v0.37.0-beta.0 * Bump to v0.37.0 Co-authored-by: tuyennhv <vutuyen2636@gmail.com> Co-authored-by: g11tech <76567250+g11tech@users.noreply.github.com> Co-authored-by: dadepo <dadepo@gmail.com> Co-authored-by: Cayman <caymannava@gmail.com> Co-authored-by: Phil Ngo <58080811+philknows@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: g11tech <gajinder@g11.in>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Some gossip blocks come very late and become "missed/orphaned", we don't want to always see it
Description
New metric to track slot time till a block becomes new head
Closes #3925