Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepared Capella beacon block is blinded when requesting a full block #12950

Closed
michaelsproul opened this issue Sep 24, 2023 · 11 comments
Closed
Labels
Bug Something isn't working

Comments

@michaelsproul
Copy link

Describe the bug

We (Lighthouse) received a report on Discord of a failed proposal due this error being returned from the /eth/v2/beacon/blocks endpoint:

CRIT Error whilst producing block info: proposal did not use a builder, block_slot: Slot(X), message: Recoverable("Some endpoints failed, num_failed: 1 http://192.168.1.2:5052/ => RequestFailed(Recoverable("Error from beacon node when producing block: ServerMessage(ErrorMessage { code: 500, message: \"Prepared Capella beacon block is blinded\", stacktraces: [] })"))"), service: block

This is a Lighthouse VC connected to a Prysm BN, and the inner error Prepared Capella beacon block is blinded comes from the Prysm BN. I don't understand what this means, because on this code path the Lighthouse VC is definitely hitting the /eth/v2/beacon/blocks endpoint. Is this the result of a misconfiguration where the Prysm BN wants to use a builder, and therefore can't build local blocks? Or is this likely related to the recent Geth block-building bug?

Has this worked before in a previous version?

No response

🔬 Minimal Reproduction

No response

Error

No response

Platform(s)

No response

What version of Prysm are you running? (Which release)

No response

Anything else relevant (validator index / public key)?

No response

@michaelsproul michaelsproul added the Bug Something isn't working label Sep 24, 2023
@james-prysm
Copy link
Contributor

It's a misconfiguration
what's happening is internally prysm has another endpoint that it's using a grpc endpoint that can return all kind of block types. because the endpoint mentioned that it's a blinded block ( despite wanting a non blinded block) what happened is it called the internal gRPC endpoint and got a blinded block. blinded blocks are returned when

  1. the mev builder is turned on through the --http-mev-relay flag
  2. the validator is registered ( either through the proposer settings or by having the --enable-builder flag)
  3. circuit breaker is not triggered.
    if those are triggered only the blinded block endpoint will work and the regular one won't we may be able to change this in the future as we are changing the internal endpoints to all be rest. there is no clean way for us to unblind a blinded block in this endpoint currently.

also tangentially related :#12103

@james-prysm
Copy link
Contributor

aware of the issue, will address as we clean up the api. closing for now

@terencechain
Copy link
Member

Reopening the issue. The /eth/v2/beacon/blocks API should always return a full block, regardless of MEV-Boost settings. A straightforward fix is in the works, utilizing the existing skipMevBoost flag in the gRPC endpoint. A PR will be opened today to address this.

@terencechain
Copy link
Member

@michaelsproul, the latest prysm develop branch should have this issue fixed. /eth/v2/beacon/blocks will never return a blind block. Please let me know if you see more issues 🙏🏼

@michaelsproul
Copy link
Author

Fantastic, thanks!

This wasn't something I ran into personally, but I'll let you know if I see any more reports!

@marioevz
Copy link
Contributor

Hey @james-prysm, this is currently reproducing in hive when I changed the prysm validator client to switch to use the rest API of prysm beacon node:

time="2023-10-26 19:54:25" level=error msg="Failed to request block from beacon node" blockSlot=2 error="failed to query GET REST endpoint: error 500: Prepared Capella beacon block is blinded" prefix=validator pubKey=0xa80e6e09cadb

Then the builder is unable to build anything throughout the entire epoch.

These are the parameters I'm currently using:

prysm beacon node:

--verbosity=trace
--accept-terms-of-use=true
--datadir=/data/beacon
--chain-config-file=/hive/input/config.yaml
--genesis-state=/hive/input/genesis.ssz
--p2p-tcp-port=900
--p2p-udp-port=9000
--p2p-host-ip=172.19.0.5
--p2p-local-ip=172.19.0.5
--execution-endpoint=http://172.19.0.3:8551
--jwt-secret=/jwtsecret
--min-sync-peers=1
--subscribe-all-subnets=true
--enable-debug-rpc-endpoints=true
--disable-monitoring=false
--monitoring-host=0.0.0.0
--monitoring-port=8080
--http-mev-relay=http://0x95fde78acd5f6886ddaf5d0056610167c513d09c1c0efabbc7cdcc69beea113779c4a81e2d24daafc5387dbf6ac5fe48@172.19.0.3:18550
--deposit-contract=0x4242424242424242424242424242424242424242
--contract-deployment-block=0
--rpc-host=0.0.0.0
--rpc-port=4001
--grpc-gateway-host=0.0.0.0
--grpc-gateway-port=4000
--grpc-gateway-corsdomain=*
--suggested-fee-recipient=0xa94f5374Fce5edBC8E2a8697C15331677e6EbF0B

validator client:

--verbosity=trace
--accept-terms-of-use=true
--prater
--enable-beacon-rest-api=true
--beacon-rest-api-provider=http://172.19.0.5:4000
--datadir=/data/vc
--wallet-dir=/data/validators
--wallet-password-file=/wallet.pass
--chain-config-file=/hive/input/config.yaml
--enable-builder
--suggested-fee-recipient=0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b

This is on devnet-10 and develop branches.

@james-prysm
Copy link
Contributor

james-prysm commented Oct 26, 2023

@marioevz will investigate perhaps it's related to this bug radek was researching #13116

@james-prysm james-prysm reopened this Oct 26, 2023
@rkapka
Copy link
Contributor

rkapka commented Oct 31, 2023

Hi @marioevz , the initial issue was related to the /beacon/blocks endpoint, but I expect you are running into issues when producing the block? In that case it's another endpoint that's being triggered. Nonetheless, #13116 takes a stab at fixing the problem with producing blinded blocks with the REST validator client.

@marioevz
Copy link
Contributor

Thanks @rkapka, let me run with these changes to see if anything changes 👍

@marioevz
Copy link
Contributor

Hey @rkapka, the problematic behavior is no longer happening after running with #13116, I think this can be closed again, thanks!

@rkapka
Copy link
Contributor

rkapka commented Nov 2, 2023

@marioevz I'm glad to hear this. Closing.

@rkapka rkapka closed this as completed Nov 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants