Could not send block production result to proposer

[mclyk]

Hi, I am testing runtime upgrade, however meet an error, parachain stop producing block after doing runtime upgrade.
does anyone know the reason?

below is the main log:

aura: [Parachain] ⌛️ Discarding proposal for slot 136537288; block production took too long
TRACE tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] Could not send block production result to proposer!    

here is the detailed log:

021-12-02 12:17:42.056  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0x6ce0c0b5b4809e8556bd95b4f76b9b0396e14afe6dca4b654d8ee0137ca22d95    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ⚠️ System declares internal migrations (which *might* execute). On-chain `<wasm:stripped>` vs current storage version `<wasm:stripped>`    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for CurrencyAdapter    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for AMMRoute    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for AMM    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ValidatorFeedersMembership    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for LiquidStakingAgentMembership    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for OracleMembership    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TechnicalCommitteeMembership    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for GeneralCouncilMembership    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for NomineeElection    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for LiquidStaking    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Prices    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Loans    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Vesting    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for OrmlXcm    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for XTokens    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Oracle    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for AuraExt    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Aura    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Session    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for CollatorSelection    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Authorship    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for CumulusXcm    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ⚠️ PolkadotXcm declares internal migrations (which *might* execute). On-chain `<wasm:stripped>` vs current storage version `<wasm:stripped>`    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for DmpQueue    
2021-12-02 12:17:42.563  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for XcmpQueue    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainInfo    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for ParachainSystem    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Scheduler    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Treasury    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TechnicalCommittee    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for GeneralCouncil    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Democracy    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Sudo    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Proxy    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Assets    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for TransactionPayment    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Balances    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Multisig    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Utility    
2021-12-02 12:17:42.564  INFO tokio-runtime-worker runtime::frame-support: [Parachain] ✅ no migration for Timestamp    
2021-12-02 12:17:42.600 DEBUG tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] Attempting to push transactions from the pool.    
2021-12-02 12:17:42.600 DEBUG tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] Pool status: PoolStatus { ready: 0, ready_bytes: 0, future: 0, future_bytes: 0 }    
2021-12-02 12:17:42.798  INFO tokio-runtime-worker aura: [Parachain] ⌛️ Discarding proposal for slot 136537288; block production took too long    
2021-12-02 12:17:42.920  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🎁 Prepared block for proposing at 89 [hash: 0x3b18fb467b3817132c3d2745cdc10f90ab3c4f18ff4bd0de15bc89c5cbb9e99c; parent_hash: 0x6ce0…2d95; extrinsics (2): [0x7fbe…cf4f, 0xfb38…e94e]]    
2021-12-02 12:17:42.928 TRACE tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] Could not send block production result to proposer!    
2021-12-02 12:17:45.292  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (6 peers), best: #440 (0x2fc6…b623), finalized #437 (0x29bf…e345), ⬇ 2.6kiB/s ⬆ 2.2kiB/s    
2021-12-02 12:17:46.556  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (2 peers), best: #88 (0x6ce0…2d95), finalized #88 (0x6ce0…2d95), ⬇ 0.1kiB/s ⬆ 91 B/s    
2021-12-02 12:17:46.903 TRACE tokio-runtime-worker txpool: [Relaychain] Attempting to notify watchers of finalization for 0xac3d…8342    

[bkchr]

aura: [Parachain] ⌛️ Discarding proposal for slot 136537288; block production took too long

This is the problem. Your block production taking too long.

[mclyk]

Thanks for your reply, if the chain will skip some blocks to get more time, as you mentioned here:
#10064 (comment)

we need to do data migration in on_runtime_upgrade, maybe this leads to the long block production time, do you have any suggestions? @bkchr

[bkchr]

Can you maybe do it over multiple blocks?

[mclyk]

Can you maybe do it over multiple blocks?

multiple blocks seems like to be a scheme under discussion, as mentioned here: #7911
May I ask if we have any solution in the current Substrate codebase?

[bkchr]

I assume the migration is your own?

If yes, then you need to either find a way to speed it up or how you can execute it over multiple blocks.

[mclyk]

Now we upgrade it between two blocks, that succeed locally, migrate data by two times' runtime upgrade.

However failed on production, what should we do next, because it stop producing blocks, the Heiko parachain, @bkchr

[mclyk]

https://medium.com/altair-network/altair-runtime-upgrade-2-a-post-mortem-618d8c3b12d8
works for us, thanks, will close this issue.