Compute recent lightclient updates

[dapplion]

Current lightclient implementation is reactive, it produces updates in response to external queries.

This PR allows the beacon node to eagerly produce lightclient updates immediately after processing each block. This approach reduces the overall cost to serve lightclient data since each update is computed at most once. Lightclient updates need access to full states to compute proofs, by computing the updates immediately after block production block replay is never necessary in normal network conditions.

In summary lightclient updates are produced in two steps:

1. cache state proofs from the attested block
2. use the next block's sync aggregate to produce the update

to not delay the import_block routine, the second step in done in a separate thread with access to the network senders to push gossip updates in a latter PR.

Since this PR only produces recent updates, it's not necessary to persist them to disk. Gossip validation rules are simplified significantly since now you just need an equality check with the most recently produced update.

Issue Addressed

• Produce and cache light client data in the BeaconChain to support fast gossip validation and RPC processing #4926

Closes #4926

Next steps

• Wire recent updates producer to gossip and SSE https://github.com/dapplion/lighthouse/compare/lc-prod-recent-updates...dapplion:lighthouse:lc-prod-push-recent-updates?expand=1

[jimmygchen]

Hey @dapplion thanks for this PR. I haven't got a chance to look at the gossip verification changes yet - will continue tomorrow, but overall looks great so far!

[jimmygchen]

Nice updates, thanks ☺️ I've left a few more questions & comments, but I think we're close!

I've got a few more things in mind, but happy to address these in a separate PRS:

• It would be useful to capture some metrics on computing light client data, this will help evaluate the readiness to make light client data serving the default behaviour.
• Nit: we've unified existing code to separate the "light client" into two words (i.e. light_client or LightClient), we should keep the consistency
• We don't have any tests rn, we should add a test on the beacon_chain to verify the light client data computation.

[dapplion]

Where should we place such test? Could be something like

Great updates! I think we can add the test to a new file dedicated for light clients under beacon_node/beacon_chain/tests. It should be pretty easy to achieve with BeaconChainHarness, we could use harness.extend_chain with some blocks and access the beacon chain using harness.chain. light_client_server_cache. This would be useful as well when we make the light client server fork-aware.

Adding a test in beacon_node/beacon_chain/tests won't be complete as the harness does not start the extra thread that calls compute_light_client_updates in the beacon_node builder.

lighthouse/beacon_node/client/src/builder.rs

Line 860 in 31acd0c

compute_light_client_updates(

The chain instance inside the harness has None as light_client_server_tx so no update is ever produced. Is there a way to recreate the more "complete" beacon node setup without repeating code in the harness?

[jimmygchen]

@dapplion and I got a passing test in the simulator, which shows that it's constantly producing optimistic updates and finality updates as the chain progresses! 🎉

I haven't looked at the metrics yet on how efficient the caching is, but I think we'll be able to start running some tests on a testnet once we have #4946 (Capella & Deneb light client support) merged as well.

Ready for a final review!

[jimmygchen]

I've re-reviewed the PR again - I think there's no changes to the regular code path when ligth-client-server is disabled, and we've got altair light client updates available according to the simulator tests, so I'm happy with the changes now 👍 thanks @dapplion!

[dapplion]

Can't approve my own PR, but the last string of commits from @jimmygchen look great https://github.com/sigp/lighthouse/pull/4969/files/b41ed0759964990bec486e223cc3a25ae2c93377..a63a4886d140d19506c73ee54c4f6971654db9ad

thanks for nailing the tests!

[jimmygchen]

@Mergifyio queue

[mergify]

queue

🛑 The pull request has been removed from the queue default

The queue conditions cannot be satisfied due to failing checks.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

[michaelsproul]

@Mergifyio requeue

[mergify]

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request

[mergify]

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at b035638