Implement Electra devnet-1 changes (sigp#5937)

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into p2p-electra

* Merge branch 'p2p-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* fix genesis from eth1 for electra

* Fix Electra Fork Choice Tests (sigp#5764)

* add new operation tests

* Fix Electra Fork Choice Tests (sigp#5764)

* Fix Electra Fork Choice Tests (sigp#5764)

* Fix Electra Fork Choice Tests (sigp#5764)

* add epoch processing new tests

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into p2p-electra

* Merge branch 'ef-tests-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* Merge branch 'p2p-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* fix ef test compilation

* disable and enable ef test for epoch processing appropriately

* add consolidations to merkle calc for inclusion proof

* fix should_override_fc tests

* Merge branch 'ef-tests-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* get ssz tests compiling

* ssz merkle validity for only deneb

* im dumb af

* Merge remote-tracking branch 'upstream/ef-tests-electra' into ef-tests-electra

* enable tests

* Fix Consolidation Sigs & Withdrawals

* Merge pull request sigp#5766 from ethDreamer/two_fixes

Fix Consolidation Sigs & Withdrawals

* Merge branches 'block-processing-electra' and 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into p2p-electra

* Merge branch 'p2p-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* comment out consolidation operation ef tests until they are added

* add signed consolidation ssz tests

* add as_iter to all List and Vector fields in the beacon state

* remove dbg

* Send unagg attestation based on fork

* Merge branch 'beacon-api-electra' into p2p-electra

* Merge branch 'p2p-electra' into ef-tests-electra

* Fix Epoch Processing Consolidations

* Fix Final Spec Tests!

* Fix ser/de

* Merge branch 'electra-engine-api' into beacon-api-electra

* Merge branch 'beacon-api-electra' into p2p-electra

* Merge branch 'p2p-electra' into ef-tests-electra

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* update electra readiness with new endpoints

* don't block mev boost till genesis finalization

* Publish all aggregates

* just one more check bro plz..

* Fix Bug In Block Processing with 0x02 Credentials

* fix slashing handling

* Fix Bug In Block Processing with 0x02 Credentials

* Merge remote-tracking branch 'upstream/unstable'

* Send unagg attestation based on fork

* Publish all aggregates

* just one more check bro plz..

* Merge pull request sigp#5832 from ethDreamer/electra_attestation_changes_merge_unstable

Merge `unstable` into `electra_attestation_changes`

* Merge pull request sigp#5835 from realbigsean/fix-validator-logic

Fix validator logic

* Merge pull request sigp#5816 from realbigsean/electra-attestation-slashing-handling

Electra slashing handling

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into p2p-electra

* Merge branch 'p2p-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* Rename deploy block electra (sigp#5853)

* Rename deploy_block.txt to deposit_contract_block.txt

* fmt

* Change for prater too

* fix: serde rename camle case for execution payload body (sigp#5846)

* Merge branch 'electra-engine-api' into beacon-api-electra

* Merge branch 'beacon-api-electra' into p2p-electra

* Merge branch 'p2p-electra' into ef-tests-electra

* Electra attestation changes rm decode impl (sigp#5856)

* Remove Crappy Decode impl for Attestation

* Remove Inefficient Attestation Decode impl

* Implement Schema Upgrade / Downgrade

* Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Fix failing attestation tests and misc electra attestation cleanup (sigp#5810)

* - get attestation related beacon chain tests to pass
- observed attestations are now keyed off of data + committee index
- rename op pool attestationref to compactattestationref
- remove unwraps in agg pool and use options instead
- cherry pick some changes from ef-tests-electra

* cargo fmt

* fix failing test

* Revert dockerfile changes

* make committee_index return option

* function args shouldnt be a ref to attestation ref

* fmt

* fix dup imports

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* fix some todos (sigp#5817)

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* add consolidations to merkle calc for inclusion proof

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into p2p-electra

* Merge branch 'p2p-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* Remove Duplicate KZG Commitment Merkle Proof Code (sigp#5874)

* Remove Duplicate KZG Commitment Merkle Proof Code

* s/tree_lists/fields/

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* fix compile

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* Fix slasher tests (sigp#5906)

* Fix electra tests

* Add electra attestations to double vote tests

* Update superstruct to 0.8

* Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes

* Small cleanup in slasher tests

* Clean up Electra observed aggregates (sigp#5929)

* Use consistent key in observed_attestations

* Remove unwraps from observed aggregates

* move committee bits to the last index in attestation (sigp#5924)

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* De-dup attestation constructor logic

* Remove unwraps in Attestation construction

* Dedup match_attestation_data

* Remove outdated TODO

* Use ForkName Ord in fork-choice tests

* Use ForkName Ord in BeaconBlockBody

* Make to_electra not fallible

* Remove TestRandom impl for IndexedAttestation

* Remove IndexedAttestation faulty Decode impl

* Drop TestRandom impl

* Add PendingAttestationInElectra

* Indexed att on disk (sigp#35)

* indexed att on disk

* fix lints

* Update slasher/src/migrate.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* add electra fork enabled fn to ForkName impl (sigp#36)

* add electra fork enabled fn to ForkName impl

* remove inadvertent file

* Update common/eth2/src/types.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* Dedup attestation constructor logic in attester cache

* Use if let Ok for committee_bits

* Dedup Attestation constructor code

* Diff reduction in tests

* Fix beacon_chain tests

* Diff reduction

* Use Ord for ForkName in pubsub

* Resolve into_attestation_and_indices todo

* Remove stale TODO

* Fix beacon_chain tests

* Test spec invariant

* Use electra_enabled in pubsub

* Remove get_indexed_attestation_from_signed_aggregate

* Use ok_or instead of if let else

* committees are sorted

* remove dup method `get_indexed_attestation_from_committees`

* Merge pull request sigp#5940 from dapplion/electra_attestation_changes_lionreview

Electra attestations sigp#5712 review

* update default persisted op pool deserialization

* ensure aggregate and proof uses serde untagged on ref

* Fork aware ssz static attestation tests

* Electra attestation changes from Lions review (sigp#5971)

* dedup/cleanup and remove unneeded hashset use

* remove irrelevant TODOs

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* Merge branch 'ef-tests-electra' of https://github.com/sigp/lighthouse into electra-devnet-1

* Fix Compilation Break

* Merge pull request sigp#5973 from ethDreamer/beacon-api-electra

Fix Compilation Break

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* Merge branch 'ef-tests-electra' of https://github.com/sigp/lighthouse into electra-devnet-1

* Rename Constants, Structures, and Fields

* Second Round

* Third Round

* Round Four

* Electra attestation changes sean review (sigp#5972)

* instantiate empty bitlist in unreachable code

* clean up error conversion

* fork enabled bool cleanup

* remove a couple todos

* return bools instead of options in `aggregate` and use the result

* delete commented out code

* use map macros in simple transformations

* remove signers_disjoint_from

* get ef tests compiling

* get ef tests compiling

* update intentionally excluded files

* Avoid changing slasher schema for Electra

* Delete slasher schema v4

* Fix clippy

* Fix compilation of beacon_chain tests

* Update database.rs

* Update per_block_processing.rs

* Add electra lightclient types

* Update slasher/src/database.rs

* fix imports

* Merge pull request sigp#5980 from dapplion/electra-lightclient

Add electra lightclient types

* Merge pull request sigp#5975 from michaelsproul/electra-slasher-no-migration

Avoid changing slasher schema for Electra

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Merge pull request sigp#5974 from ethDreamer/electra-devnet-1

Electra: `devnet-1` Rename and Move Stuff

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Merge branch 'electra_attestation_changes' of https://github.com/realbigsean/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* clean up enabled and disabled fork logic in ef tests

* Merge branch 'ef-tests-electra' of https://github.com/sigp/lighthouse into electra-devnet-1

* Electra devnet 1 process operations (sigp#5994)

* Implement `process_consolidation_requests()`

* Finish Changes to `process_operations()`

* Fix Lint

* Fix test

* use electra_enabled()

* add light client ssz tests

* Update `engine-api` for `devnet-1`

* disable failing light client ssz tests

* correctly exclude light client struct from accessed files

* Fix Failing Tests

* Fix Calculations Relying on MAX_EB (sigp#5983)

* Fix Calculations Relying on MAX_EB

* Use spec fn for max_effective_balance

* The great renaming receipt -> request

* Address some more review comments

* Electra: Get `devnet-1` ef-tests Working

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra-engine-api

* Update beacon_node/beacon_chain/src/electra_readiness.rs

* Update consensus/types/src/chain_spec.rs

* Update pending balance deposit processing (sigp#6005)

* Update pending balance deposit processing

* Update single_pass.rs

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge pull request sigp#6001 from ethDreamer/electra-devnet-1-engine-api

Update `engine-api` for `devnet-1`

* Merge pull request sigp#6019 from ethDreamer/electra-devnet-1-ef-tests

Electra: Get `devnet-1` ef-tests Working

* Fix Bug in `engine_api` by Renaming Field

* Merge pull request sigp#6047 from ethDreamer/electra-devnet-1-engine-api-fix

Fix Bug in `engine_api` by Renaming Field

* one more stupid rename

* update GET requests

* update POST requests

* add client updates and test updates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* compile after merge

* unwrap -> unwrap_err

* self review

* fix tests

* convert op pool messages to electra in electra

* remove methods to post without content header

* Add BeaconBlocksByRange v3

* Revert "Add BeaconBlocksByRange v3"

This reverts commit e3ce7fc.

* filter instead of convert

* Add range sync metrics to track efficiency (sigp#6095)

* Add more range sync metrics to track efficiency

* Add ignored blocks metrics

* Enable the outbound rate limiter by default, and update blobs method quotas (sigp#6093)

* Enable the outbound rate limiter by default, and update blobs method quotas.

* Lint and book updates.

* Beacon api + validator electra (sigp#5744)

* Attestation superstruct changes for EIP 7549 (sigp#5644)

* update

* experiment

* superstruct changes

* revert

* superstruct changes

* fix tests

* indexed attestation

* indexed attestation superstruct

* updated TODOs

* `superstruct` the `AttesterSlashing` (sigp#5636)

* `superstruct` Attester Fork Variants

* Push a little further

* Deal with Encode / Decode of AttesterSlashing

* not so sure about this..

* Stop Encode/Decode Bounds from Propagating Out

* Tons of Changes..

* More Conversions to AttestationRef

* Add AsReference trait (sigp#15)

* Add AsReference trait

* Fix some snafus

* Got it Compiling! :D

* Got Tests Building

* Get beacon chain tests compiling

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes

* Make EF Tests Fork-Agnostic (sigp#5713)

* Finish EF Test Fork Agnostic (sigp#5714)

* Superstruct `AggregateAndProof` (sigp#5715)

* Upgrade `superstruct` to `0.8.0`

* superstruct `AggregateAndProof`

* Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes

* cargo fmt

* Merge pull request sigp#5726 from realbigsean/electra_attestation_changes

Merge unstable into Electra attestation changes

* process withdrawals updates

* cleanup withdrawals processing

* update `process_operations` deposit length check

* add apply_deposit changes

* add execution layer withdrawal request processing

* process deposit receipts

* add consolidation processing

* update process operations function

* exit updates

* clean up

* update slash_validator

* EIP7549 `get_attestation_indices` (sigp#5657)

* get attesting indices electra impl

* fmt

* get tests to pass

* fmt

* fix some beacon chain tests

* fmt

* fix slasher test

* fmt got me again

* fix more tests

* fix tests

* Some small changes (sigp#5739)

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* cargo fmt (sigp#5740)

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* fix attestation verification

* Add new engine api methods

* Fix the versioning of v4 requests

* Handle new engine api methods in mock EL

* Note todo

* Fix todos

* Add support for electra fields in getPayloadBodies

* Add comments for potential versioning confusion

* udpates for aggregate attestation endpoint

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Sketch op pool changes

* fix get attesting indices (sigp#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Ef test fixes (sigp#5753)

* attestation related ef test fixes

* delete commented out stuff

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Fix Aggregation Pool for Electra (sigp#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (sigp#5755)

* Get `electra_op_pool` up to date (sigp#5756)

* fix get attesting indices (sigp#5742)

* fix get attesting indices

* better errors

* fix compile

* only get committee index once

* Ef test fixes (sigp#5753)

* attestation related ef test fixes

* delete commented out stuff

* Fix Aggregation Pool for Electra (sigp#5754)

* Fix Aggregation Pool for Electra

* Remove Outdated Interface

* fix ssz (sigp#5755)

---------

Co-authored-by: realbigsean <sean@sigmaprime.io>

* Revert "Get `electra_op_pool` up to date (sigp#5756)" (sigp#5757)

This reverts commit ab9e58a.

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into electra_op_pool

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Compute on chain aggregate impl (sigp#5752)

* add compute_on_chain_agg impl to op pool changes

* fmt

* get op pool tests to pass

* update beacon api aggregate attestationendpoint

* update the naive agg pool interface (sigp#5760)

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* updates after merge

* Fix bugs in cross-committee aggregation

* Add comment to max cover optimisation

* Fix assert

* Electra epoch processing

* add deposit limit for old deposit queue

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge pull request sigp#5749 from sigp/electra_op_pool

Optimise Electra op pool aggregation

* don't fail on empty consolidations

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* update committee offset

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* update committee offset

* update committee offset

* update committee offset

* only increment the state deposit index on old deposit flow

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* use correct max eb in epoch cache initialization

* drop initiate validator ordering optimization

* fix initiate exit for single pass

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* accept new payload v4 in mock el

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Fix Electra Fork Choice Tests (sigp#5764)

* Fix Electra Fork Choice Tests (sigp#5764)

* Fix Electra Fork Choice Tests (sigp#5764)

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Fix Consolidation Sigs & Withdrawals

* Merge pull request sigp#5766 from ethDreamer/two_fixes

Fix Consolidation Sigs & Withdrawals

* Merge branches 'block-processing-electra' and 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Send unagg attestation based on fork

* Fix ser/de

* Merge branch 'electra-engine-api' into beacon-api-electra

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* Subscribe to the correct subnets for electra attestations (sigp#5782)

* subscribe to the correct att subnets for electra

* subscribe to the correct att subnets for electra

* cargo fmt

* update electra readiness with new endpoints

* fix slashing handling

* Fix Bug In Block Processing with 0x02 Credentials

* Merge remote-tracking branch 'upstream/unstable'

* Send unagg attestation based on fork

* Publish all aggregates

* just one more check bro plz..

* Merge pull request sigp#5832 from ethDreamer/electra_attestation_changes_merge_unstable

Merge `unstable` into `electra_attestation_changes`

* Merge pull request sigp#5835 from realbigsean/fix-validator-logic

Fix validator logic

* Merge pull request sigp#5816 from realbigsean/electra-attestation-slashing-handling

Electra slashing handling

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* fix: serde rename camle case for execution payload body (sigp#5846)

* Merge branch 'electra-engine-api' into beacon-api-electra

* Electra attestation changes rm decode impl (sigp#5856)

* Remove Crappy Decode impl for Attestation

* Remove Inefficient Attestation Decode impl

* Implement Schema Upgrade / Downgrade

* Update beacon_node/beacon_chain/src/schema_change/migration_schema_v20.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Fix failing attestation tests and misc electra attestation cleanup (sigp#5810)

* - get attestation related beacon chain tests to pass
- observed attestations are now keyed off of data + committee index
- rename op pool attestationref to compactattestationref
- remove unwraps in agg pool and use options instead
- cherry pick some changes from ef-tests-electra

* cargo fmt

* fix failing test

* Revert dockerfile changes

* make committee_index return option

* function args shouldnt be a ref to attestation ref

* fmt

* fix dup imports

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* fix some todos (sigp#5817)

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* add consolidations to merkle calc for inclusion proof

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Remove Duplicate KZG Commitment Merkle Proof Code (sigp#5874)

* Remove Duplicate KZG Commitment Merkle Proof Code

* s/tree_lists/fields/

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* fix compile

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Fix slasher tests (sigp#5906)

* Fix electra tests

* Add electra attestations to double vote tests

* Update superstruct to 0.8

* Merge remote-tracking branch 'origin/unstable' into electra_attestation_changes

* Small cleanup in slasher tests

* Clean up Electra observed aggregates (sigp#5929)

* Use consistent key in observed_attestations

* Remove unwraps from observed aggregates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* De-dup attestation constructor logic

* Remove unwraps in Attestation construction

* Dedup match_attestation_data

* Remove outdated TODO

* Use ForkName Ord in fork-choice tests

* Use ForkName Ord in BeaconBlockBody

* Make to_electra not fallible

* Remove TestRandom impl for IndexedAttestation

* Remove IndexedAttestation faulty Decode impl

* Drop TestRandom impl

* Add PendingAttestationInElectra

* Indexed att on disk (sigp#35)

* indexed att on disk

* fix lints

* Update slasher/src/migrate.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* add electra fork enabled fn to ForkName impl (sigp#36)

* add electra fork enabled fn to ForkName impl

* remove inadvertent file

* Update common/eth2/src/types.rs

Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>

* Dedup attestation constructor logic in attester cache

* Use if let Ok for committee_bits

* Dedup Attestation constructor code

* Diff reduction in tests

* Fix beacon_chain tests

* Diff reduction

* Use Ord for ForkName in pubsub

* Resolve into_attestation_and_indices todo

* Remove stale TODO

* Fix beacon_chain tests

* Test spec invariant

* Use electra_enabled in pubsub

* Remove get_indexed_attestation_from_signed_aggregate

* Use ok_or instead of if let else

* committees are sorted

* remove dup method `get_indexed_attestation_from_committees`

* Merge pull request sigp#5940 from dapplion/electra_attestation_changes_lionreview

Electra attestations sigp#5712 review

* update default persisted op pool deserialization

* ensure aggregate and proof uses serde untagged on ref

* Fork aware ssz static attestation tests

* Electra attestation changes from Lions review (sigp#5971)

* dedup/cleanup and remove unneeded hashset use

* remove irrelevant TODOs

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* Fix Compilation Break

* Merge pull request sigp#5973 from ethDreamer/beacon-api-electra

Fix Compilation Break

* Electra attestation changes sean review (sigp#5972)

* instantiate empty bitlist in unreachable code

* clean up error conversion

* fork enabled bool cleanup

* remove a couple todos

* return bools instead of options in `aggregate` and use the result

* delete commented out code

* use map macros in simple transformations

* remove signers_disjoint_from

* get ef tests compiling

* get ef tests compiling

* update intentionally excluded files

* Avoid changing slasher schema for Electra

* Delete slasher schema v4

* Fix clippy

* Fix compilation of beacon_chain tests

* Update database.rs

* Update per_block_processing.rs

* Add electra lightclient types

* Update slasher/src/database.rs

* fix imports

* Merge pull request sigp#5980 from dapplion/electra-lightclient

Add electra lightclient types

* Merge pull request sigp#5975 from michaelsproul/electra-slasher-no-migration

Avoid changing slasher schema for Electra

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Update beacon_node/beacon_chain/src/attestation_verification.rs

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra_attestation_changes

* Merge branch 'electra_attestation_changes' of https://github.com/realbigsean/lighthouse into block-processing-electra

* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-epoch-proc

* Merge branch 'electra-epoch-proc' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* The great renaming receipt -> request

* Address some more review comments

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra-engine-api

* Update beacon_node/beacon_chain/src/electra_readiness.rs

* Update consensus/types/src/chain_spec.rs

* update GET requests

* update POST requests

* add client updates and test updates

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra-engine-api

* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra

* compile after merge

* unwrap -> unwrap_err

* self review

* fix tests

* convert op pool messages to electra in electra

* remove methods to post without content header

* filter instead of convert

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into beacon-api-electra

* Merge branch 'beacon-api-electra' of https://github.com/sigp/lighthouse into ef-tests-electra

* fix ef test compile

* use `enabled` fork methods

* get tests passing

* last updates

* Fix bug on engine api with consolidations (sigp#6101)

* Fix small bug in engine_api

* Merge pull request sigp#6116 from ethDreamer/electra-devnet-1

Fix payload_bodies bug in engine_api

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into ef-tests-electra

* add deposit request file

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into ef-tests-electra

* Merge branch 'ef-tests-electra' of https://github.com/sigp/lighthouse into electra-devnet-1

* clean up after merge

* Fix Broken Test on `devnet-1`

* another small bug

* Merge pull request sigp#6132 from ethDreamer/devnet-1-fix-test

Fix Broken Test on `devnet-1`

* Fix Deserialization Bug in `engine-api`

* Merge pull request sigp#6235 from ethDreamer/fix_deserialization

Fix Deserialization Bug in `engine-api`

* Merge branch 'unstable' into electra-devnet-1

* remove import

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into electra-devnet-1

* Remove false negative checks on `shuffling_is_compatible`

* Fix flaky validator monitor test

beacon_node/beacon_chain/src/beacon_block_streamer.rs

@@ -1,5 +1,5 @@
 use crate::{metrics, BeaconChain, BeaconChainError, BeaconChainTypes, BlockProcessStatus};
-use execution_layer::{ExecutionLayer, ExecutionPayloadBodyV1};
+use execution_layer::{ExecutionLayer, ExecutionPayloadBody};
 use slog::{crit, debug, error, Logger};
 use std::collections::HashMap;
 use std::sync::Arc;
@@ -57,7 +57,7 @@ struct BodiesByRange<E: EthSpec> {
 struct BlockParts<E: EthSpec> {
     blinded_block: Box<SignedBlindedBeaconBlock<E>>,
     header: Box<ExecutionPayloadHeader<E>>,
-    body: Option<Box<ExecutionPayloadBodyV1<E>>>,
+    body: Option<Box<ExecutionPayloadBody<E>>>,
 }
 
 impl<E: EthSpec> BlockParts<E> {

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -5531,8 +5531,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
                             bls_to_execution_changes: bls_to_execution_changes.into(),
                             blob_kzg_commitments: kzg_commitments
                                 .ok_or(BlockProductionError::InvalidPayloadFork)?,
-                            // TODO(electra): finish consolidations when they're more spec'd out
-                            consolidations: Vec::new().into(),
                         },
                     }),
                     maybe_blobs_and_proofs,

beacon_node/beacon_chain/tests/store_tests.rs

@@ -1437,18 +1437,16 @@ fn check_shuffling_compatible(
                 head_state.current_epoch(),
                 |committee_cache, _| {
                     let state_cache = head_state.committee_cache(RelativeEpoch::Current).unwrap();
+                    // We used to check for false negatives here, but had to remove that check
+                    // because `shuffling_is_compatible` does not guarantee their absence.
+                    //
+                    // See: https://github.com/sigp/lighthouse/issues/6269
                     if current_epoch_shuffling_is_compatible {
                         assert_eq!(
                             committee_cache,
                             state_cache.as_ref(),
                             "block at slot {slot}"
                         );
-                    } else {
-                        assert_ne!(
-                            committee_cache,
-                            state_cache.as_ref(),
-                            "block at slot {slot}"
-                        );
                     }
                     Ok(())
                 },
@@ -1479,8 +1477,6 @@ fn check_shuffling_compatible(
                     let state_cache = head_state.committee_cache(RelativeEpoch::Previous).unwrap();
                     if previous_epoch_shuffling_is_compatible {
                         assert_eq!(committee_cache, state_cache.as_ref());
-                    } else {
-                        assert_ne!(committee_cache, state_cache.as_ref());
                     }
                     Ok(())
                 },

beacon_node/beacon_chain/tests/validator_monitor.rs

@@ -4,7 +4,7 @@ use beacon_chain::test_utils::{
 use beacon_chain::validator_monitor::{ValidatorMonitorConfig, MISSED_BLOCK_LAG_SLOTS};
 use logging::test_logger;
 use std::sync::LazyLock;
-use types::{Epoch, EthSpec, Keypair, MainnetEthSpec, PublicKeyBytes, Slot};
+use types::{Epoch, EthSpec, ForkName, Keypair, MainnetEthSpec, PublicKeyBytes, Slot};
 
 // Should ideally be divisible by 3.
 pub const VALIDATOR_COUNT: usize = 48;
@@ -128,7 +128,7 @@ async fn produces_missed_blocks() {
     let initial_blocks = slots_per_epoch * nb_epoch_to_simulate.as_u64() - 1;
 
     // The validator index of the validator that is 'supposed' to miss a block
-    let mut validator_index_to_monitor = 1;
+    let validator_index_to_monitor = 1;
 
     // 1st scenario //
     //
@@ -201,34 +201,22 @@ async fn produces_missed_blocks() {
     // Missed block happens when slot and prev_slot are not in the same epoch
     // making sure that the cache reloads when the epoch changes
     // in that scenario the slot that missed a block is the first slot of the epoch
-    validator_index_to_monitor = 7;
     // We are adding other validators to monitor as these ones will miss a block depending on
     // the fork name specified when running the test as the proposer cache differs depending on
     // the fork name (cf. seed)
     //
     // If you are adding a new fork and seeing errors, print
     // `validator_indexes[slot_in_epoch.as_usize()]` and add it below.
-    let validator_index_to_monitor_altair = 2;
-    // Same as above but for the merge upgrade
-    let validator_index_to_monitor_bellatrix = 4;
-    // Same as above but for the capella upgrade
-    let validator_index_to_monitor_capella = 11;
-    // Same as above but for the deneb upgrade
-    let validator_index_to_monitor_deneb = 3;
-    // Same as above but for the electra upgrade
-    let validator_index_to_monitor_electra = 6;
-
-    let harness2 = get_harness(
-        validator_count,
-        vec![
-            validator_index_to_monitor,
-            validator_index_to_monitor_altair,
-            validator_index_to_monitor_bellatrix,
-            validator_index_to_monitor_capella,
-            validator_index_to_monitor_deneb,
-            validator_index_to_monitor_electra,
-        ],
-    );
+    let validator_index_to_monitor = match harness1.spec.fork_name_at_slot::<E>(Slot::new(0)) {
+        ForkName::Base => 7,
+        ForkName::Altair => 2,
+        ForkName::Bellatrix => 4,
+        ForkName::Capella => 11,
+        ForkName::Deneb => 3,
+        ForkName::Electra => 1,
+    };
+
+    let harness2 = get_harness(validator_count, vec![validator_index_to_monitor]);
     let advance_slot_by = 9;
     harness2
         .extend_chain(
@@ -300,6 +288,12 @@ async fn produces_missed_blocks() {
         duplicate_block_root = *_state2.block_roots().get(idx as usize).unwrap();
         validator_indexes = _state2.get_beacon_proposer_indices(&harness2.spec).unwrap();
         let not_monitored_validator_index = validator_indexes[slot_in_epoch.as_usize()];
+        // This could do with a refactor: https://github.com/sigp/lighthouse/issues/6293
+        assert_ne!(
+            not_monitored_validator_index,
+            validator_index_to_monitor,
+            "this test has a fragile dependency on hardcoded indices. you need to tweak some settings or rewrite this"
+        );
 
         assert_eq!(
             _state2.set_block_root(prev_slot, duplicate_block_root),