[PROD] Validator miss attestation

[twoeths]

Describe the bug

Validator 270069 missed an attestation at slot 2733664. For more information visit: https://beaconcha.in/validator/270069

Expected behavior

Validator should not miss any attestations

[twoeths]

Validator log shows that the attestation is published on time with correct head

Dec-16 04:13:14.649 []                ^[[34mdebug^[[39m: Signed attestation slot=2733664, index=27, head=0xeb5d6fccd8d86989f576e83b22bac72de531abb858a2314e7283a3bce9b3b17f, validatorIndex=270069

committeesPerSlot = 131, epoch slot = 0, committee index = 27 => attestation subnet = 27

Node log shows that attestation is published by gossipsub

Dec-16 04:13:11.010 [CHAIN]         ^[[36mverbose^[[39m: Clock slot slot=2733664
Dec-16 04:13:14.651 [API]             ^[[34mdebug^[[39m: Req req-6gh 127.0.0.1 POST:/eth/v1/beacon/pool/attestations
Dec-16 04:13:14.770 [NETWORK]       ^[[36mverbose^[[39m: Publish to topic topic=/eth2/afcaaba0/beacon_attestation_27/ssz_snappy

maybe it's an issue of having attestation subnet 27 peers at that time to spread the attestation to the network

[dapplion]

@tuyennhv We should have some internal log of an attestation's journey.

1. Submitted by validator
2. Received in some aggregate
3. Received in block

[twoeths]

one reason could be we submit attestation too early sigp/lighthouse#635 (comment) , and the other peer (could be another lodestar node) ignore the attestation since they haven't seen the voted block

[dapplion]

one reason could be we submit attestation too early sigp/lighthouse#635 (comment) , and the other peer (could be another lodestar node) ignore the attestation since they haven't seen the voted block

That seems extremely unlikely, there are multiple aggregators per subnet and Lodestar represents a tiny fraction of the network

[twoeths]

one reason could be we submit attestation too early sigp/lighthouse#635 (comment) , and the other peer (could be another lodestar node) ignore the attestation since they haven't seen the voted block

That seems extremely unlikely, there are multiple aggregators per subnet and Lodestar represents a tiny fraction of the network

I mean if the original node only propagate subnet attestation to the other lodestar node and that lodestar node just ignore the subnet attestation (since it does not know the root), no aggregator could include it and it becomes missed

[dapplion]

@tuyennhv Nimbus latest release specifically optimize for situations where emitting to no peers:

Tuned peer management: reduces the likelihood of missed attestations

If you've seen frequent "No peers for topic" in your logs, this release will help

https://github.com/status-im/nimbus-eth2/releases/tag/v1.6.0

You should investigate what they've changed or reach out

[twoeths]

#3781 helps us gain good gossip peer score, hence more mesh peers. This could help us mitigate the issue.

[twoeths]

There were a lot of enhancements/improvements since I open the issue, we rarely missed attestations in the last 7 days

the known issues were resolved so closing the issue for now, we should open new issues for any specific missed attesttations