--suggested-fee-recipient for validator not overriding beacon's address in Prysm v5.0.3

[haoei]

Describe the bug

Our beacon client sets --suggested-fee-recipient to address 1, while our validator client sets it to address 2. However, when our validator (1058425) proposed a block in slot 9430590, the fee recipient was address 1 instead of address 2.

According to the documentation, when both the beacon and validator clients have different fee recipient addresses set using this flag, the validator's address should override the beacon's address. However, in my case, the beacon's address is occasionally taking precedence over the validator's address.

I compiled the logs of the beacon and validator around proposed.
Please let us know if you need additional details or logs to debug this issue.

validator logs

[2024-07-03 07:17:39]  INFO client: Submitted new sync message blockRoot=0xccd908be2130 slot=9430586 slotStartTime=2024-07-03 07:17:35 +0000 UTC timeSinceSlotStart=4.035887758s validatorIndex=1058509
[2024-07-03 07:17:43]  INFO client: Submitted new sync contribution and proof aggregatorIndex=1058509 bitsCount=126 blockRoot=0xccd908be2130 slot=9430586 slotStartTime=2024-07-03 07:17:35 +0000 UTC subcommitteeIndex=0 timeSinceSlotStart=8.013208517s
[2024-07-03 07:17:43]  INFO client: Submitted new attestations blockRoot=0xccd908be2130 committeeIndices=[55 6 48 45 44 17 10 12 35 46 62 20 52 35 31 10 6 56 52 12 63 9 56 53 54 11 56 22 16] pubkeys=[0xaf2e9c50fce8 0xb2b3b2ba45b4 0xad71d4f5f4b7 0x92afcc3c49cf 0xaebea0514f72 0x8695b6b541d6 0x8f313a2af51d 0xaad09fe09a8f 0x8843f3abe63e 0x8acbffc86220 0xad344730149e 0x977ad0530b0e 0xb5a70f23993a 0xa523752e88fa 0x8b36d9988bc7 0x870ae7454d76 0xa1954713ac38 0xa6c0ac41769b 0x83c86bc0ddc3 0xb89a7caa0037 0xb7cc7164461f 0xb192ff7e06fe 0x87357d270a17 0xad97d8fb7085 0x9864884d94ee 0x8462f1d1df8e 0xb89b9d889436 0xa21e600f7ddc 0x8d924909944b] slot=9430586 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:17:43]  INFO client: Submitted new aggregate attestations blockRoot=0xccd908be2130 committeeIndices=[46 35] pubkeys=[0x8acbffc86220 0xa523752e88fa] slot=9430586 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:17:45] ERROR client: Event stream interrupted error=rpc error: code = Canceled desc = Context canceled: could not connect
[2024-07-03 07:17:51]  INFO client: Submitted new sync message blockRoot=0xc3ef20ed8c6f slot=9430587 slotStartTime=2024-07-03 07:17:47 +0000 UTC timeSinceSlotStart=4.030138408s validatorIndex=1058509
[2024-07-03 07:17:51]  INFO client: Event stream reconnecting...
[2024-07-03 07:17:51]  INFO client: Starting event stream topics=[head]
[2024-07-03 07:17:55]  INFO client: Submitted new sync contribution and proof aggregatorIndex=1058509 bitsCount=127 blockRoot=0xc3ef20ed8c6f slot=9430587 slotStartTime=2024-07-03 07:17:47 +0000 UTC subcommitteeIndex=0 timeSinceSlotStart=8.01349886s
[2024-07-03 07:17:55]  INFO client: Submitted new attestations blockRoot=0xc3ef20ed8c6f committeeIndices=[26 23 8 47 58 59 35 19 53 32 24 32 5 27 21 20 40 15 1 59 63 10 58 36 49 48 13 46 33 17 38 60 0 27 51 14 2 38 5 16 51 14 53] pubkeys=[0xb674eba0039b 0x8767bdf5d59c 0x8f9a621fce3e 0x8292b1bbdfce 0xb7dd15466e06 0x9765a54048b9 0xaff7381facb3 0x919c3586bc2b 0xa4e1574475e0 0x939b736822ef 0xb43b785e749b 0x8b866b322ea9 0x87bb42e9ebe5 0x8b3c2b8bc258 0x93facefd7404 0xaebd9b16fc65 0xa64d7ee08ea9 0xb9ef6d7f61d5 0x9466850f96db 0x88b78ea01efb 0xa0113f376743 0xb2fb1c6ca18c 0xabaa17adaf26 0xb8be37cedeaf 0x808489d8b722 0x8d922d2a2490 0x92a78e6aac1b 0xa6b7f4c24483 0xb1928e611ede 0x889387e756f2 0x90d003124984 0x906ba2a78e61 0xaf7605abea84 0x83dcc3660f28 0x85181c94ce5f 0x83624fa05007 0x990b64d86f6c 0xa9908ba05b44 0xab884a5c146d 0xae04cdf0c7b1 0x8835f25111f2 0xb1edb5e262e5 0x94f57bab178d] slot=9430587 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:18:03]  INFO client: Submitted new sync message blockRoot=0xaf6d0e9e302a slot=9430588 slotStartTime=2024-07-03 07:17:59 +0000 UTC timeSinceSlotStart=4.024759565s validatorIndex=1058509
[2024-07-03 07:18:03]  INFO client: Submitted new attestations blockRoot=0xaf6d0e9e302a committeeIndices=[3 12 34 53 35 10 7 33 25 39 21 24 24 15 3 48 40 59 60 40 12 13 9 15 22 15 35 26 15 17] pubkeys=[0xb5df203f68b9 0x99711de13f42 0xa3998090e455 0x9032bab875cb 0x939fad284439 0x97516451e1ae 0xa57d0f8eb32b 0x97e1f31d0907 0xab084b1a3187 0xae1b1b2637dc 0xa9ed0a233606 0xb02b3ccd0eb5 0x800ff7dd4ed1 0xace5b08a6218 0xac574332732e 0xa06fbcb85ac0 0xb5742f82d5a1 0x899005e18c37 0x84a343603981 0xabbf44a67ac3 0xaf51401299e2 0x997603a8b456 0xb600d60dcd86 0x84d1b90a15d7 0xa189405e6732 0x8a1351899133 0x9336ed16a92c 0xb6144995f0d0 0x85ac1609952c 0x953a8c5f0761] slot=9430588 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:18:15]  INFO client: Submitted new sync message blockRoot=0x7672d006fcdb slot=9430589 slotStartTime=2024-07-03 07:18:11 +0000 UTC timeSinceSlotStart=4.010208094s validatorIndex=1058509
[2024-07-03 07:18:19]  INFO client: Submitted new attestations blockRoot=0x7672d006fcdb committeeIndices=[5 42 46 21 39 30 6 26 7 14 20 20 18 61] pubkeys=[0xa5c9ff28172c 0x855c774d0d1a 0xb0b4fe7a0961 0x87096f465635 0xb08dbf8fb100 0x884afcf9d289 0x934118ca24d0 0x89323d6ed75d 0x991a287dbfac 0x9345c4963f85 0x8551dd04a88d 0xa542abbdb0ad 0x8b0718c33bf0 0x981b768cb706] slot=9430589 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:18:19]  INFO client: Submitted new aggregate attestations blockRoot=0x7672d006fcdb committeeIndices=[42] pubkeys=[0x855c774d0d1a] slot=9430589 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:18:23]  INFO client: Submitted new block attestationCount=128 blockRoot=0xc98864e20cac depositCount=0 fork=deneb gasUtilized=0.2634832 graffiti=ebunker.io kzgCommitmentCount=6 parentHash=0xe0cbd742bb2d payloadHash=0x9c7e46063361 pubkey=0xa46b792d507a slot=9430590 txCount=86 withdrawalCount=16
[2024-07-03 07:18:27]  INFO client: Submitted new sync message blockRoot=0xc98864e20cac slot=9430590 slotStartTime=2024-07-03 07:18:23 +0000 UTC timeSinceSlotStart=4.028274204s validatorIndex=1058509
[2024-07-03 07:18:31]  INFO client: Submitted new attestations blockRoot=0xc98864e20cac committeeIndices=[48 55 39 50 17 27 22 20 15 48 39 1 57 25 41 48 0 12 56 49 20 50 44 44 50 22 30 39 20 56 28 57 7 62 63] pubkeys=[0xb6a75c3afbfa 0xb4dabaa7aab2 0x8708033dfa9a 0xb14b8790ba68 0x92a50a75b45e 0x95f93cbc1fa9 0x85fe9c63b398 0xa2e5f6d333e3 0xb3c434433912 0x98c456e52219 0xa40dcefd4f81 0x97c99ffb1c91 0x8286c383af6b 0xab0677ac911c 0xb92cddc8dc6b 0x8e9583710980 0x8b624b5d1b4c 0x85796272106f 0xb04a5e3bc56f 0xa6758ec8d166 0x860351ab82b0 0xb698efb82f29 0x90bbf58d29c2 0x943ace5b7547 0xb059186fea9f 0xa4b9fb7c790a 0xb636316c7be1 0x922141f830a3 0xa64e9fa5f140 0x840d117470f8 0xb54bf9e260a3 0xaecaf0f8ebdb 0xb16a6837dce5 0x8220554c1120 0xb9c00bad8ae0] slot=9430590 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:18:31]  INFO client: Submitted new aggregate attestations blockRoot=0xc98864e20cac committeeIndices=[63] pubkeys=[0xb9c00bad8ae0] slot=9430590 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:18:39]  INFO client: Submitted new sync message blockRoot=0x27db9644da8e slot=9430591 slotStartTime=2024-07-03 07:18:35 +0000 UTC timeSinceSlotStart=4.016845818s validatorIndex=1058509
[2024-07-03 07:18:43]  INFO client: Submitted new attestations blockRoot=0x27db9644da8e committeeIndices=[27 61 58 21 62 22 43 50 10 32 22 5 18 24 29 27 18 29 23 46 26 1 57 42 54 57 43] pubkeys=[0x8b2a0e5b94d1 0xb48cbc101690 0x88867b8bffaf 0x8831b701283d 0xb4c6ed121348 0xb3e88a9b721d 0x8c006ec1bc63 0x8831e5112d8a 0x97cff39e6136 0x8f504c1d08c2 0xb005b76a0fb0 0xb6de808bf6e9 0x96ebb6873385 0xac73892628c0 0xaef4f8c8ad39 0x87b739f11299 0xac16cbd13188 0xa58669806f0a 0xa4cc93621ae7 0x8e24c442982e 0x8d17cf668d03 0xa24d57120c54 0xafd9b9f3b7fe 0xa621d59c424b 0x8bd102871e91 0xb34c8341ab22 0x88e7e3add5a0] slot=9430591 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8
[2024-07-03 07:18:43]  INFO client: Submitted new aggregate attestations blockRoot=0x27db9644da8e committeeIndices=[43] pubkeys=[0x8c006ec1bc63] slot=9430591 sourceEpoch=294704 sourceRoot=0x924eca7e98bb targetEpoch=294705 targetRoot=0x6574642636c8

beacon logs

Jul 03 07:18:12 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:12" level=info msg="Synced new block" block=0x7672d006... epoch=294705 finalizedEpoch=294703 finalizedRoot=0x04883763... prefix=blockchain slot=9430589
Jul 03 07:18:12 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:12" level=info msg="Finished applying state transition" attestations=66 kzgCommitmentCount=6 payloadHash=0xe0cbd742bb2d prefix=blockchain slot=9430589 syncBitsCount=507 txCount=140
Jul 03 07:18:23 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:23" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=23.65034ms slot=9430590
Jul 03 07:18:23 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:23" level=warning msg="could not find tracked proposer index" headRoot=0x7672d006fcdbc8163acd5e142f0ad2f8a7228a7afeed44ab1265f98931912cba slot=9430590 validatorIndex=1058425
Jul 03 07:18:23 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:23" level=info msg="Finished building block" prefix="rpc/validator" sinceSlotStartTime=254.698143ms slot=9430590 validator=1058425
Jul 03 07:18:23 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:23" level=info msg="Synced new block" block=0xc98864e2... epoch=294705 finalizedEpoch=294703 finalizedRoot=0x04883763... prefix=blockchain slot=9430590
Jul 03 07:18:23 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:23" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=6 payloadHash=0x9c7e46063361 prefix=blockchain slot=9430590 syncBitsCount=507 txCount=86
Jul 03 07:18:37 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:37" level=info msg="Synced new block" block=0x27db9644... epoch=294705 finalizedEpoch=294703 finalizedRoot=0x04883763... prefix=blockchain slot=9430591
Jul 03 07:18:37 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:37" level=info msg="Finished applying state transition" attestations=128 kzgCommitmentCount=1 payloadHash=0xc75edd1e4f52 prefix=blockchain slot=9430591 syncBitsCount=509 txCount=190
Jul 03 07:18:38 eth-cl prysm.sh[2741]: time="2024-07-03 07:18:38" level=info msg="Peer summary" activePeers=218 inbound=183 outbound=35 prefix=p2p

Has this worked before in a previous version?

This problem has occurred in a previous version, but I don't remember which version

🔬 Minimal Reproduction

Steps to Reproduce:

Set up the beacon client with --suggested-fee-recipient pointing to fee address 1.
Set up the validator client with --suggested-fee-recipient pointing to fee address 2.
Run both clients and observe the fee recipient address in use.
Expected Behavior:

The fee recipient address set for the validator (fee address 2) should consistently override the beacon's address (fee address 1).

Actual Behavior:

The fee recipient address set for the beacon (fee address 1) is intermittently being used instead of the validator's address.

beacon script

USE_PRYSM_VERSION=v5.0.3 /usr/local/prysm/prysm.sh beacon-chain \
    --accept-terms-of-use \
    --checkpoint-sync-url=https://beaconstate.ethstaker.cc \
    --genesis-beacon-api-url=https://beaconstate.ethstaker.cc \
    --monitoring-host=0.0.0.0 \
    --execution-endpoint=http://127.0.0.1:8551 \
    --jwt-secret=jwt.hex \
    --suggested-fee-recipient=<address1> \
    --http-mev-relay=http://127.0.0.1:18550

validator script

USE_PRYSM_VERSION=v5.0.3 ../prysm.sh validator \
    --beacon-rpc-provider 127.0.0.1:4000 \
    --monitoring-host 0.0.0.0 \
    --suggested-fee-recipient <address2> \
    --enable-builder=true

Error

No response

Platform(s)

Linux (x86)

What version of Prysm are you running? (Which release)

v5.0.3

Anything else relevant (validator index / public key)?

Validator: https://beaconcha.in/validator/1058425
Proposed slot: https://beaconcha.in/slot/9430590

[james-prysm]

I'd like to better understand whether these validators are being added to the client via API right before they are proposing or changing their status right before they are proposing?

is the validator falling back onto some other beacon node ( some kind of bad connection causing the validator to connect to a fall back beacon node?)

[haoei]

I'd like to better understand whether these validators are being added to the client via API right before they are proposing or changing their status right before they are proposing?

These validators have been added to the validator client and activated for a long time. After the activation of the validator, the validator client was restarted several times for upgrades.

is the validator falling back onto some other beacon node ( some kind of bad connection causing the validator to connect to a fall back beacon node?)

Yes. The validator client connects to the beacon nodes through a load balancer. When a beacon node is being upgraded, the validator client will automatically switch to another beacon node. An upgrade was just carried out not long ago.

[james-prysm]

Yes. The validator client connects to the beacon nodes through a load balancer. When a beacon node is being upgraded, the validator client will automatically switch to another beacon node. An upgrade was just carried out not long ago.

I'm guessing this is the root cause in this case, I assume the load balance moving to another beacon node during the epoch of the of the validator proposing but before the next epoch basically caused the proposer settings which contain the fee recipient not to register, there is a flaw in our design for this gap and I am investigating if we can update these settings every slot without too much overhead ( need to bench and see how it goes). we have a longer term strategy but requires more design discussion.

if a more immediate solution is needed you can update the loadbalanced beacon node manually using the push proposer settings / validator registration endpoints manually.

i was also experimenting with allowing proposer settings file on the beacon node as well but also not merged.