Checkpoint Submission stopped saying INVALID_EPOCH

[kr-advait]

Checkpoint submission stopped INVALID_EPOCH

We are running polygon supernet from past 3 months and chain was working fine until 6 days ago, out of nowhere, checkpoint submission have been stopped. According to what i feel, one of the checkpoint was missed to be submitted due to some reason which created this gap between checkpoint in blockchain as it moved ahead with other blocks and checkpoint manager contract left behind.

Now the error i am facing is INVALID_EPOCH from checkpoint manager contract and that requires checkpoint to be either same or current epoch + 1. I have tried debugging and have added logs and i can see that when it goes to send pending checkpoints the epoch is current epoch on contract + 1.

eg : Current epoch on contract = 29660, so supernet is trying to send checkpoint for epoch = 29661

but still getting error INVALID_EPOCH

Your environment

• OS and version. : Linux Ubuntu 22.04
• The version of the Polygon Edge. : polygon edge 1.3.2
• Please confirm if the validators are running under containerized environment (K8s, Docker, etc.). : no validators are running locally on VPS.

Steps to reproduce

• Where the issue is, if you know. : consensus / polybft / checkpoint_manager.go => submitCheckpoint

Expected behavior

• Tell us what should happen. : checkpoints should be submitted if pending and then normally
• Tell us what happened instead. : pending checkpoints are not being submitted which leads to current epoch submit checkpoint error

Logs

2024-09-18T11:18:39.242Z [DEBUG] polygon.server.polybft.consensus_runtime: on block inserted already handled: current=3836289 block=3836289
2024-09-18T11:18:39.369Z [DEBUG] polygon.server.polybft: validator message received: type=PREPARE height=3836289 round=0 addr=0x0B9aC1dc962e2a1131eaF05Fc103222f9b71409f
2024-09-18T11:18:39.369Z [DEBUG] polygon.server.polybft: validator message received: type=COMMIT height=3836289 round=0 addr=0x0B9aC1dc962e2a1131eaF05Fc103222f9b71409f
2024-09-18T11:18:39.370Z [DEBUG] polygon.server.polybft: validator message received: type=COMMIT height=3836289 round=0 addr=0x0B9aC1dc962e2a1131eaF05Fc103222f9b71409f
2024-09-18T11:18:39.370Z [DEBUG] polygon.server.polybft: validator message received: type=PREPARE height=3836289 round=0 addr=0x0B9aC1dc962e2a1131eaF05Fc103222f9b71409f
2024-09-18T11:18:39.545Z [DEBUG] polygon.server.polybft.checkpoint_manager: submitCheckpoint invoked...: latest checkpoint block=3766820 checkpoint block=3836289

[Encode & Send Checkpoint for Epoch] : 29661

2024-09-18T11:18:39.926Z [DEBUG] polygon.server.polybft.checkpoint_manager: send checkpoint txn...: block number=3766947

[Submit Checkpoint for BlockNumber] : 3766947

[Submit Checkpoint for Epoch] : 29661

[Sending Transaction For currentExtra Epoch from address 0xa31110B4c8A7d0eDBc4269B4a5d21F3C71CC711a] : 29661

[Sending Transaction For param Extra Epoch from address 0xa31110B4c8A7d0eDBc4269B4a5d21F3C71CC711a] : 29661

[Error in Sending Epoch To Polygon] : failed to estimate gas: {"code":3,"message":"execution reverted: INVALID_EPOCH","data":"0x08c379a00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000d494e56414c49445f45504f434800000000000000000000000000000000000000"}

2024-09-18T11:18:40.448Z [WARN] polygon.server.polybft.checkpoint_manager: failed to submit checkpoint: checkpoint block=3836289 epoch number=30207 error="failed to estimate gas: {"code":3,"message":"execution reverted: INVALID_EPOCH","data":"0x08c379a00000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000d494e56414c49445f45504f434800000000000000000000000000000000000000"}"

[kr-advait]

Fixed the issue!

I believe the main issue is with estimate gas in send transaction.

• When the checkpoint has been prepared to send to the smart contract, before executing the call, polygon edge calls estimate gas function to determine the required amount of gas to be passed for smart contract call.
• However, this estimate gas assumed that the smart contract call is going to fail and thus returned
  execution reverted : INVALID_EPOCH but also it gave failed to estimate gas error.
• Which means that we might have to provide custom gas when facing error during gas estimation.
• This fix was enough to send all pending checkpoints and sync again with current state and epoch of the blockchain.

Here is the fix :

Location : txrelayer / txrelayer.go => sendTransactionLocked() function

Replace below code :

with this code :