Upgrade the Beacon API e2e evaluator (#13868)

* GET

* POST

* Revert "Auxiliary commit to revert individual files from 615feb1"

This reverts commit 55cf071c684019f3d6124179154c10b2277fda49.

* comment fix

* deepsource

config values

block protos

get_committee_indices

proto and ssz

attestation interface

Revert "Auxiliary commit to revert individual files from deadb21"

This reverts commit 32ad5009537bc5ec0e6caf9f52143d380d00be85.

todos

get_attesting_indices

Revert "Auxiliary commit to revert individual files from dd27897"

This reverts commit f39644ed3cb6f3964fc6c86fdf4bd5de2a9668c8.

beacon spec changes

Fix pending attestation. Build ok

Don't return error that can be internally handled (#13887)

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>

consistent auth token for validator apis (#13747)

* wip

* fixing tests

* adding more tests especially to handle legacy

* fixing linting

* fixing deepsource issues and flags

* fixing some deepsource issues,pathing issues, and logs

* some review items

* adding additional review feedback

* updating to follow updates from ethereum/keymanager-APIs#74

* adjusting functions to match changes in keymanagers PR

* Update validator/rpc/auth_token.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update validator/rpc/auth_token.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* Update validator/rpc/auth_token.go

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

* review feedback

---------

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

Use correct port for health check in Beacon API e2e evaluator (#13892)

Change example.org DNS record (#13904)

DNS Changed for this record causing tests to fail

Co-authored-by: Radosław Kapka <rkapka@wp.pl>

Electra beacon config (#13907)

* Update spectests to v1.5.0

* Add electra config

* Fix tests in beacon-chain/rpc/eth/config

* gofmt

Simplify prune invalid by reusing existing fork choice store call (#13878)

Do not remove blobs DB in slasher. (#13881)

Refactor beacon-chain/core/helpers tests to be black box (#13906)

spectests: fail hard on missing test folders (#13913)

Revert "zig: Update zig to recent main branch commit (#13142)" (#13908)

This reverts commit b24b60d.

Remove EnableEIP4881 flag (#13826)

* Remove EnableEIP4881 flag

* Gaz

* Fix missing error handler

* Remove old tree and fix tests

* Gaz

* Fix build import

* Replace depositcache

* Add pendingDeposit tests

* Nishant's fix

* Fix unsafe uint64 to int

* Fix other unsafe uint64 to int

* Remove: RemovePendingDeposit

* Deprecate and remove DisableEIP4881 flag

* Check: index not greater than deposit count

* Move index check

Protobufs for Electra devnet-0  (#13905)

* block protos

* proto and ssz

* stubs

* Enable Electra spec test

* Pull in EIP-7251 protobuf changes

From: #13903

* All EIP7549 containers are passing

* All EIP7251 containers passing

* including changes from eip7002

* Everything passing except for beacon state hash tree root

* fixing eletra state to use electra payload

* Fix minimal test. Skip beacon state test

* Perston's feedback

---------

Co-authored-by: rkapka <radoslaw.kapka@gmail.com>
Co-authored-by: james-prysm <james@prysmaticlabs.com>

use [32]byte keys in the filesystem cache (#13885)

Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>

Electra: full beacon config (#13918)

* Electra: full beacon config

Fix TestGetSpec

* Fix beacon config spec compliance test so that it properly loads the config from spec tests. Tests failing for now.

* fix tests and comply with spec presets

beacon-chain/cache: Convert tests to cache_test blackbox testing (#13920)

* beacon-chain/cache: convert to blackbox tests (package cache_test)

* Move balanceCacheKey to its own file to satisify go fuzz build

Electra: Minor proto changes, cloner additions (#13923)

* Electra: more proto changes

* Roundtrip test for cloners

Electra: HTR util for DepositReceipt and ExecutionLayerWithdrawalRequest (#13924)

* Electra: HTR utils for DepositReceipts and ExecutionLayerWithdrawalRequests

* Tests for HTR utils

Electra: beacon-chain/core/helpers  (#13921)

* Electra helpers

* Electra helper tests and other fixes

* @terencechain feedback

Electra: add electra version

Electra: consensus types

gocognit exclusion

@potuz's suggestion

build fix

interfaces for indexed att and slashing

indexed att usage

BuildSignedBeaconBlockFromExecutionPayload

slashing usage

grpc stubs

remove unused methods

Electra attestation interfaces

WORKSPACE

@@ -29,23 +29,7 @@ http_archive(
 
 load("@hermetic_cc_toolchain//toolchain:defs.bzl", zig_toolchains = "toolchains")
 
-# Temporarily use a nightly build until 0.12.0 is released.
-# See: https://github.com/prysmaticlabs/prysm/issues/13130
-zig_toolchains(
-    host_platform_sha256 = {
-        "linux-aarch64": "45afb8e32adde825165f4f293fcea9ecea503f7f9ec0e9bf4435afe70e67fb70",
-        "linux-x86_64": "f136c6a8a0f6adcb057d73615fbcd6f88281b3593f7008d5f7ed514ff925c02e",
-        "macos-aarch64": "05d995853c05243151deff47b60bdc2674f1e794a939eaeca0f42312da031cee",
-        "macos-x86_64": "721754ba5a50f31e8a1f0e1a74cace26f8246576878ac4a8591b0ee7b6db1fc1",
-        "windows-x86_64": "93f5248b2ea8c5ee8175e15b1384e133edc1cd49870b3ea259062a2e04164343",
-    },
-    url_formats = [
-        "https://ziglang.org/builds/zig-{host_platform}-{version}.{_ext}",
-        "https://mirror.bazel.build/ziglang.org/builds/zig-{host_platform}-{version}.{_ext}",
-        "https://prysmaticlabs.com/mirror/ziglang.org/builds/zig-{host_platform}-{version}.{_ext}",
-    ],
-    version = "0.12.0-dev.1349+fa022d1ec",
-)
+zig_toolchains()
 
 # Register zig sdk toolchains with support for Ubuntu 20.04 (Focal Fossa) which has an EOL date of April, 2025.
 # For ubuntu glibc support, see https://launchpad.net/ubuntu/+source/glibc
@@ -243,7 +227,7 @@ filegroup(
     url = "https://github.com/ethereum/EIPs/archive/5480440fe51742ed23342b68cf106cefd427e39d.tar.gz",
 )
 
-consensus_spec_version = "v1.4.0"
+consensus_spec_version = "v1.5.0-alpha.0"
 
 bls_test_version = "v0.1.1"
 
@@ -259,7 +243,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    sha256 = "c282c0f86f23f3d2e0f71f5975769a4077e62a7e3c7382a16bd26a7e589811a0",
+    sha256 = "33c5547772b6d8d6f041dff7e7d26b0358c2392daed34394a3aa81147812a81c",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/general.tar.gz" % consensus_spec_version,
 )
 
@@ -275,7 +259,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    sha256 = "4649c35aa3b8eb0cfdc81bee7c05649f90ef36bede5b0513e1f2e8baf37d6033",
+    sha256 = "06f286199cf2fedd4700487fb8feb0904e0ae18daaa4b3f70ea430ca9c388167",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/minimal.tar.gz" % consensus_spec_version,
 )
 
@@ -291,7 +275,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    sha256 = "c5a03f724f757456ffaabd2a899992a71d2baf45ee4db65ca3518f2b7ee928c8",
+    sha256 = "5f2a4452b323075eba6bf950003f7d91fd04ebcbde5bd087beafb5d6f6325ad4",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/mainnet.tar.gz" % consensus_spec_version,
 )
 
@@ -306,7 +290,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    sha256 = "cd1c9d97baccbdde1d2454a7dceb8c6c61192a3b581eee12ffc94969f2db8453",
+    sha256 = "fd7e83e8cbeb3e297f2aeb93776305f7d606272c97834d8d9be673984501ed36",
     strip_prefix = "consensus-specs-" + consensus_spec_version[1:],
     url = "https://github.com/ethereum/consensus-specs/archive/refs/tags/%s.tar.gz" % consensus_spec_version,
 )

api/BUILD.bazel

@@ -1,11 +1,24 @@
-load("@prysm//tools/go:def.bzl", "go_library")
+load("@prysm//tools/go:def.bzl", "go_library", "go_test")
 
 go_library(
     name = "go_default_library",
     srcs = [
         "constants.go",
         "headers.go",
+        "jwt.go",
     ],
     importpath = "github.com/prysmaticlabs/prysm/v5/api",
     visibility = ["//visibility:public"],
+    deps = [
+        "//crypto/rand:go_default_library",
+        "@com_github_ethereum_go_ethereum//common/hexutil:go_default_library",
+        "@com_github_pkg_errors//:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["jwt_test.go"],
+    embed = [":go_default_library"],
+    deps = ["//testing/require:go_default_library"],
 )

api/constants.go

@@ -4,4 +4,6 @@ const (
 	WebUrlPrefix        = "/v2/validator/"
 	WebApiUrlPrefix     = "/api/v2/validator/"
 	KeymanagerApiPrefix = "/eth/v1"
+
+	AuthTokenFileName = "auth-token"
 )

api/jwt.go

@@ -0,0 +1,32 @@
+package api
+
+import (
+	"github.com/ethereum/go-ethereum/common/hexutil"
+	"github.com/pkg/errors"
+	"github.com/prysmaticlabs/prysm/v5/crypto/rand"
+)
+
+// GenerateRandomHexString generates a random hex string that follows the standards for jwt token
+// used for beacon node -> execution client
+// used for web client -> validator client
+func GenerateRandomHexString() (string, error) {
+	secret := make([]byte, 32)
+	randGen := rand.NewGenerator()
+	n, err := randGen.Read(secret)
+	if err != nil {
+		return "", err
+	} else if n != 32 {
+		return "", errors.New("rand: unexpected length")
+	}
+	return hexutil.Encode(secret), nil
+}
+
+// ValidateAuthToken validating auth token for web
+func ValidateAuthToken(token string) error {
+	b, err := hexutil.Decode(token)
+	// token should be hex-encoded and at least 256 bits
+	if err != nil || len(b) < 32 {
+		return errors.New("invalid auth token: token should be hex-encoded and at least 256 bits")
+	}
+	return nil
+}

api/jwt_test.go

@@ -0,0 +1,13 @@
+package api
+
+import (
+	"testing"
+
+	"github.com/prysmaticlabs/prysm/v5/testing/require"
+)
+
+func TestGenerateRandomHexString(t *testing.T) {
+	token, err := GenerateRandomHexString()
+	require.NoError(t, err)
+	require.NoError(t, ValidateAuthToken(token))
+}

beacon-chain/blockchain/BUILD.bazel

@@ -137,7 +137,7 @@ go_test(
         "//async/event:go_default_library",
         "//beacon-chain/blockchain/testing:go_default_library",
         "//beacon-chain/cache:go_default_library",
-        "//beacon-chain/cache/depositcache:go_default_library",
+        "//beacon-chain/cache/depositsnapshot:go_default_library",
         "//beacon-chain/core/blocks:go_default_library",
         "//beacon-chain/core/feed/state:go_default_library",
         "//beacon-chain/core/helpers:go_default_library",

beacon-chain/blockchain/chain_info.go

@@ -509,13 +509,6 @@ func (s *Service) Ancestor(ctx context.Context, root []byte, slot primitives.Slo
 	return ar[:], nil
 }
 
-// SetOptimisticToInvalid wraps the corresponding method in forkchoice
-func (s *Service) SetOptimisticToInvalid(ctx context.Context, root, parent, lvh [32]byte) ([][32]byte, error) {
-	s.cfg.ForkChoiceStore.Lock()
-	defer s.cfg.ForkChoiceStore.Unlock()
-	return s.cfg.ForkChoiceStore.SetOptimisticToInvalid(ctx, root, parent, lvh)
-}
-
 // SetGenesisTime sets the genesis time of beacon chain.
 func (s *Service) SetGenesisTime(t time.Time) {
 	s.genesisTime = t

beacon-chain/blockchain/execution_engine.go

@@ -256,7 +256,7 @@ func (s *Service) notifyNewPayload(ctx context.Context, preStateVersion int,
 // reportInvalidBlock deals with the event that an invalid block was detected by the execution layer
 func (s *Service) pruneInvalidBlock(ctx context.Context, root, parentRoot, lvh [32]byte) error {
 	newPayloadInvalidNodeCount.Inc()
-	invalidRoots, err := s.SetOptimisticToInvalid(ctx, root, parentRoot, lvh)
+	invalidRoots, err := s.cfg.ForkChoiceStore.SetOptimisticToInvalid(ctx, root, parentRoot, lvh)
 	if err != nil {
 		return err
 	}

beacon-chain/blockchain/head.go

@@ -401,7 +401,7 @@ func (s *Service) saveOrphanedOperations(ctx context.Context, orphanedRoot [32]b
 		}
 		for _, a := range orphanedBlk.Block().Body().Attestations() {
 			// if the attestation is one epoch older, it wouldn't been useful to save it.
-			if a.Data.Slot+params.BeaconConfig().SlotsPerEpoch < s.CurrentSlot() {
+			if a.GetData().Slot+params.BeaconConfig().SlotsPerEpoch < s.CurrentSlot() {
 				continue
 			}
 			if helpers.IsAggregated(a) {

beacon-chain/blockchain/metrics.go

@@ -369,6 +369,6 @@ func reportEpochMetrics(ctx context.Context, postState, headState state.BeaconSt
 
 func reportAttestationInclusion(blk interfaces.ReadOnlyBeaconBlock) {
 	for _, att := range blk.Body().Attestations() {
-		attestationInclusionDelay.Observe(float64(blk.Slot() - att.Data.Slot))
+		attestationInclusionDelay.Observe(float64(blk.Slot() - att.GetData().Slot))
 	}
 }

beacon-chain/blockchain/process_attestation.go

@@ -6,6 +6,7 @@ import (
 
 	"github.com/pkg/errors"
 	"github.com/prysmaticlabs/prysm/v5/beacon-chain/core/helpers"
+	"github.com/prysmaticlabs/prysm/v5/consensus-types/interfaces"
 	"github.com/prysmaticlabs/prysm/v5/encoding/bytesutil"
 	ethpb "github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1"
 	"github.com/prysmaticlabs/prysm/v5/proto/prysm/v1alpha1/attestation"
@@ -36,17 +37,17 @@ import (
 //
 //	 # Update latest messages for attesting indices
 //	 update_latest_messages(store, indexed_attestation.attesting_indices, attestation)
-func (s *Service) OnAttestation(ctx context.Context, a *ethpb.Attestation, disparity time.Duration) error {
+func (s *Service) OnAttestation(ctx context.Context, a interfaces.Attestation, disparity time.Duration) error {
 	ctx, span := trace.StartSpan(ctx, "blockChain.onAttestation")
 	defer span.End()
 
 	if err := helpers.ValidateNilAttestation(a); err != nil {
 		return err
 	}
-	if err := helpers.ValidateSlotTargetEpoch(a.Data); err != nil {
+	if err := helpers.ValidateSlotTargetEpoch(a.GetData()); err != nil {
 		return err
 	}
-	tgt := ethpb.CopyCheckpoint(a.Data.Target)
+	tgt := ethpb.CopyCheckpoint(a.GetData().Target)
 
 	// Note that target root check is ignored here because it was performed in sync's validation pipeline:
 	// validate_aggregate_proof.go and validate_beacon_attestation.go
@@ -67,20 +68,20 @@ func (s *Service) OnAttestation(ctx context.Context, a *ethpb.Attestation, dispa
 	}
 
 	// Verify attestation beacon block is known and not from the future.
-	if err := s.verifyBeaconBlock(ctx, a.Data); err != nil {
+	if err := s.verifyBeaconBlock(ctx, a.GetData()); err != nil {
 		return errors.Wrap(err, "could not verify attestation beacon block")
 	}
 
 	// Note that LMD GHOST and FFG consistency check is ignored because it was performed in sync's validation pipeline:
 	// validate_aggregate_proof.go and validate_beacon_attestation.go
 
 	// Verify attestations can only affect the fork choice of subsequent slots.
-	if err := slots.VerifyTime(genesisTime, a.Data.Slot+1, disparity); err != nil {
+	if err := slots.VerifyTime(genesisTime, a.GetData().Slot+1, disparity); err != nil {
 		return err
 	}
 
 	// Use the target state to verify attesting indices are valid.
-	committee, err := helpers.BeaconCommitteeFromState(ctx, baseState, a.Data.Slot, a.Data.CommitteeIndex)
+	committee, err := helpers.BeaconCommitteeFromState(ctx, baseState, a.GetData().Slot, a.GetData().CommitteeIndex)
 	if err != nil {
 		return err
 	}
@@ -97,7 +98,7 @@ func (s *Service) OnAttestation(ctx context.Context, a *ethpb.Attestation, dispa
 	// We assume trusted attestation in this function has verified signature.
 
 	// Update forkchoice store with the new attestation for updating weight.
-	s.cfg.ForkChoiceStore.ProcessAttestation(ctx, indexedAtt.AttestingIndices, bytesutil.ToBytes32(a.Data.BeaconBlockRoot), a.Data.Target.Epoch)
+	s.cfg.ForkChoiceStore.ProcessAttestation(ctx, indexedAtt.GetAttestingIndices(), bytesutil.ToBytes32(a.GetData().BeaconBlockRoot), a.GetData().Target.Epoch)
 
 	return nil
 }

beacon-chain/blockchain/process_block.go

@@ -366,17 +366,17 @@ func (s *Service) handleEpochBoundary(ctx context.Context, slot primitives.Slot,
 func (s *Service) handleBlockAttestations(ctx context.Context, blk interfaces.ReadOnlyBeaconBlock, st state.BeaconState) error {
 	// Feed in block's attestations to fork choice store.
 	for _, a := range blk.Body().Attestations() {
-		committee, err := helpers.BeaconCommitteeFromState(ctx, st, a.Data.Slot, a.Data.CommitteeIndex)
+		committee, err := helpers.BeaconCommitteeFromState(ctx, st, a.GetData().Slot, a.GetData().CommitteeIndex)
 		if err != nil {
 			return err
 		}
-		indices, err := attestation.AttestingIndices(a.AggregationBits, committee)
+		indices, err := attestation.AttestingIndices(a.GetAggregationBits(), committee)
 		if err != nil {
 			return err
 		}
-		r := bytesutil.ToBytes32(a.Data.BeaconBlockRoot)
+		r := bytesutil.ToBytes32(a.GetData().BeaconBlockRoot)
 		if s.cfg.ForkChoiceStore.HasNode(r) {
-			s.cfg.ForkChoiceStore.ProcessAttestation(ctx, indices, r, a.Data.Target.Epoch)
+			s.cfg.ForkChoiceStore.ProcessAttestation(ctx, indices, r, a.GetData().Target.Epoch)
 		} else if err := s.cfg.AttPool.SaveBlockAttestation(a); err != nil {
 			return err
 		}
@@ -387,7 +387,7 @@ func (s *Service) handleBlockAttestations(ctx context.Context, blk interfaces.Re
 // InsertSlashingsToForkChoiceStore inserts attester slashing indices to fork choice store.
 // To call this function, it's caller's responsibility to ensure the slashing object is valid.
 // This function requires a write lock on forkchoice.
-func (s *Service) InsertSlashingsToForkChoiceStore(ctx context.Context, slashings []*ethpb.AttesterSlashing) {
+func (s *Service) InsertSlashingsToForkChoiceStore(ctx context.Context, slashings []interfaces.AttesterSlashing) {
 	for _, slashing := range slashings {
 		indices := blocks.SlashableAttesterIndices(slashing)
 		for _, index := range indices {