Validator Client Could Shut Down Prematurely

[nisdas]

🐞 Bug Report

Description

Depending on when a user starts up their validator client and how long their beacon node takes to sync/get ready a certain fatal error path could be triggered which leads to the validator client being shut down and a user being offline. The root cause of this error is how we apply our deadline here in PushProposerSettings

https://github.com/prysmaticlabs/prysm/blob/develop/validator/client/validator.go#L993

We round up the slot to the end of the epoch here in RoundUpToNearestEpoch in the majority of the cases. However if our current slot is the first slot of the epoch, we will not round it up. Instead it will apply the deadline to the end of the current slot. While this is fine if the current time is earlier on in the slot, this becomes an issue if our current time falls towards the end of a slot. If our current time is already close to the end of a slot and the deadline is at the end of it, it might give us a very short time period to execute all the calls in PushProposerSettings. In this case our deadline will very likely expire and the validator client will produce a fatal log and shut down.

The correct solution would be to apply a fixed time period to execute all these calls rather than relying on rounded up slots as being a marker of how long we have to execute these calls.

Found by Antithesis

cc @0xTylerHolmes

Has this worked before in a previous version?

N.A

🔬 Minimal Reproduction

This error is difficult to reproduce as it requires the validator client to start up in a short time period in an epoch (1 second).

🔥 Error

level=fatal msg="Failed to update proposer settings" error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" prefix=validator

🌍 Your Environment

Operating System:

x86_64 GNU/Linux

What version of Prysm are you running? (Which release)

v3.2.1

Anything else relevant (validator index / public key)?

N.A