x/upgrade segmentation violation crash

[RaspiRepo]

x/upgrade.BeginBlocker: ctx.ConsensusParams().Version.AppVersion <nil>

Here AppVersion is optional field as per tendermint-SDK but cosmos-sdk upgrade module debug lines trying to access AppVersion filed and leading to upgrade process to crash, unable to recover chain application once get into this issue

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5486fd8]

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0xc000e12420, 0x16}, 0xc00114c480, {0x64fdf38, 0xc0011a6570}, {0x65637a0, 0xc000e16df0}, 0xc00021e090, {0x64ebca0, 0xc000103a00}, ...}, ...)
	/Users/skuchain/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/x/upgrade/abci.go:38 +0x1238
github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)

trace

12:21PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
12:21PM INF Starting EventBus service impl=EventBus module=events
12:21PM INF Starting PubSub service impl=PubSub module=pubsub
12:21PM INF Starting IndexerService service impl=IndexerService module=txindex
12:21PM INF ABCI Handshake App Info hash="5��8��e��{�Wq�-��t\x19}\x10���}��'�`kJ" height=3075 module=consensus protocol-version=0 software-version=v0.10
12:21PM INF ABCI Replay Blocks appHeight=3075 module=consensus stateHeight=3075 storeHeight=3076
12:21PM INF Replay last block using real app module=consensus
upgrae beginblk:  ctx.ConsensusParams().Version...... block:<max_bytes:22020096 max_gas:-1 > evidence:<max_age_num_blocks:100000 max_age_duration:<seconds:172800 > max_bytes:1048576 > validator:<pub_key_types:"ed25519" >
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5486124]

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0xc00022afa8, 0x16}, 0xc001115620, {0x64fdef8, 0xc000f0a570}, {0x6563760, 0xc0001e4590}, 0xc00112c8a0, {0x64ebc60, 0xc00122d860}, ...}, ...)
	/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/x/upgrade/abci.go:38 +0x664
github.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)
	/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/x/upgrade/module.go:130
github.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x65391e8, 0xc000128008}, {0x6569620, 0xc001191000}, {{0xb, 0x0}, {0xc000d82ad8, 0x5}, 0xc04, ...}, ...}, ...)
	/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/types/module/module.go:481 +0x218
github.com/raspirepo/hello/app.(*App).BeginBlocker(...)
	/Users/raspirepo/Documents/personal/cosmos-dev/upgrade-demo/hello/app/app.go:641
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc000fc3ae0, 0x20, 0x20}, {{0xb, 0x0}, {0xc000d82ad8, 0x5}, 0xc04, {0x28f80d68, ...}, ...}, ...})
	/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/abci.go:194 +0x9dc
github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc000fc3ae0, 0x20, 0x20}, {{0xb, 0x0}, {0xc000d82ad8, 0x5}, 0xc04, {0x28f80d68, ...}, ...}, ...})
	/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/abci/client/local_client.go:280 +0x118
github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc000fc3ae0, 0x20, 0x20}, {{0xb, 0x0}, {0xc000d82ad8, 0x5}, 0xc04, {0x28f80d68, ...}, ...}, ...})
	/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/proxy/app_conn.go:81 +0x55
github.com/tendermint/tendermint/state.execBlockOnProxyApp({0x653a1e0, 0xc001235260}, {0x6550bc0, 0xc001127d30}, 0xc00000c5a0, {0x6564090, 0xc0011264e0}, 0xc03)
	/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/state/execution.go:307 +0x3dd
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc000d82bc8, 0x7}}, {0xc000d82c00, 0x5}, 0x1, 0xc03, {{0xc000fc2020, ...}, ...}, ...}, ...)
	/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/state/execution.go:140 +0x171
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc000d82bc8, 0x7}}, {0xc000d82c00, 0x5}, 0x1, 0xc03, {{0xc000fc2020, ...}, ...}, ...}, ...)
	/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/consensus/replay.go:503 +0x23c
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc000d82bc8, 0x7}}, {0xc000d82c00, 0x5}, 0x1, 0xc03, {{0xc000fc2020, ...}, ...}, ...}, ...)
	/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/consensus/replay.go:416 +0x7ae
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc000c1df90, {0x656b7a0, 0xc000d86000})
	/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/consensus/replay.go:268 +0x3c8

Version

go: go version go1.17.6 darwin/amd64
build_deps:
tendermint : github.com/tendermint/tendermint@v0.34.19
cosmos_sdk_version: v0.45.4
commit hash : ad9e562

cosmos-sdk/x/upgrade/abci.go

Line 38 in ad9e562

panic(fmt.Sprintf("Wrong app version %d, upgrade handler is missing for %s upgrade plan", ctx.ConsensusParams().Version.AppVersion, lastAppliedPlan))

Steps to Reproduce

1. Deploy chain app with version v0.1 (using cosmovisor start)
2. Build a chain binary for version v0.2
3. copy binary to ~/.chainapp/cosmovisor/upgrades/v0.2/bin/
4. apply upgrade process ---> success
5. Prepare another version binary say v0.3 on local folder,
6. (Don't copy to cosmovisor/upgrades/v0.3/bin/) i.e skip step 3. NOTE: by mistake coping binary to upgrade folder is missed.
7. Apply upgrade process v0.3, once block height reached as per upgrade plan, cosmovisor try to replace the new version binary and stop with error like one below and application stopped.

11:43AM ERR UPGRADE "v0.3" NEEDED at height: 3075: version-upgrade v0.3
binary not present, downloading disabled: cannot stat dir ~/.hello/cosmovisor/upgrades/v0.3/bin/hellod: stat ~/.hello/cosmovisor/upgrades/v0.3/bin/hellod: no such file or directory

8. Now copy version v0.3 binary to cosmovisor/upgrades/v0.3/bin/
9. restart chain app cosmovisor start Now we will see the log, upgrade new binary replace is good

11:45AM INF Replay last block using real app module=consensus
11:45AM ERR UPGRADE "v0.03" NEEDED at height: 3075: version-upgrade v0.03
11:45AM INF starting node with ABCI Tendermint in-process
11:45AM INF Starting multiAppConn service impl=multiAppConn module=proxy
11:45AM INF Starting localClient service connection=query impl=localClient module=abci-client
11:45AM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
11:45AM INF Starting localClient service connection=mempool impl=localClient module=abci-client
11:45AM INF Starting localClient service connection=consensus impl=localClient module=abci-client
11:45AM INF Starting EventBus service impl=EventBus module=events
11:45AM INF Starting PubSub service impl=PubSub module=pubsub
11:45AM INF Starting IndexerService service impl=IndexerService module=txindex
11:45AM INF ABCI Handshake App Info hash="�f|���\x16n\x13W�E���(VkVC}E\\4���C}\x18\a�" height=3074 module=consensus protocol-version=0 software-version=v0.03
11:45AM INF ABCI Replay Blocks appHeight=3074 module=consensus stateHeight=3074 storeHeight=3075
11:45AM INF Replay last block using real app module=consensus
11:45AM INF applying upgrade "v0.10" at height: 3075
11:45AM INF adding a new module: hello
11:45AM INF created new capability module=ibc name=ports/monitoring
11:45AM INF port binded module=x/ibc/port port=monitoring
..................
........
11:45AM INF Timed out dur=4871.09 height=3076 module=consensus round=0 step=1
11:45AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"81DC2358C928D13D83CF74260B985791324E912CD7E41CB7504123C50753FC0C","parts":{"hash":"A3E1E400F17FC6E778440C482CED600E8256C43013B1C57EA7F27AAF1022C4F4","total":1}},"height":3076,"pol_round":-1,"round":0,"signature":"gmcMdKcafL4fCvCT9nNPzJpeEQzc8Pjtor6bPVkeXDnsRCnbQtwVsUA4k/O988sYcuqUb8xhikOqGf1bG5FSDw==","timestamp":"2022-05-13T18:45:25.876384Z"}
11:45AM INF received complete proposal block hash=81DC2358C928D13D83CF74260B985791324E912CD7E41CB7504123C50753FC0C height=3076 module=consensus
11:45AM INF finalizing commit of block hash=81DC2358C928D13D83CF74260B985791324E912CD7E41CB7504123C50753FC0C height=3076 module=consensus num_txs=0 root=35E3DE38C7F16586E17BDD57719C2DC1B574197D10FAF5807D9FB6279B606B4A
11:45AM ERR CONSENSUS FAILURE!!! err="runtime error: invalid memory address or nil pointer dereference" module=consensus stack="goroutine 64 [running]:
runtime/debug.Stack()\n\t/usr/local/Cellar/go/1.17.6/libexec/src/runtime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()
/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/consensus/state.go:727 +0x4c\npanic({0x57b2180, 0x76146d0})\n\t/usr/local/Cellar/go/1.17.6/libexec/src/runtime/panic.go:1038 +0x215\ngithub.com/cosmos/cosmos-sdk/x/upgrade.BeginBlocker({{0xc0010e6438, 0x16}, 0xc001325230, {0x64fdfb8, 0xc0001fe020}, {0x6563820, 0xc0010df580}, 0xc001324e70, {0x64ebd20, 0xc00119d860}, ...}, ...)\n\t/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/x/upgrade/abci.go:38 +0x1093\ngithub.com/cosmos/cosmos-sdk/x/upgrade.AppModule.BeginBlock(...)\n\t/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/x/upgrade/module.go:130\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).BeginBlock(_, {{0x65392a8, 0xc000128008}, {0x6569760, 0xc000ebf980}, {{0xb, 0x0}, {0xc002542729, 0x5}, 0xc04, ...}, ...}, ...)\n\t/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/types/module/module.go:481 +0x218\ngithub.com/raspirepo/hello/app.(*App).BeginBlocker(...)\n\t/Users/raspirepo/Documents/personal/cosmos-dev/upgrade-demo/hello/app/app.go:641\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(_, {{0xc000d2a720, 0x20, 0x20}, {{0xb, 0x0}, {0xc002542729, 0x5}, 0xc04, {0x28f80d68, ...}, ...}, ...})\n\t/Users/raspirepo/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.45.4/baseapp/abci.go:194 +0x9dc\ngithub.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(_, {{0xc000d2a720, 0x20, 0x20}, {{0xb, 0x0}, {0xc002542729, 0x5}, 0xc04, {0x28f80d68, ...}, ...}, ...})\n\t/Users/raspirepo/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/abci/client/local_client.go:280 +0x118\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(_, {{0xc000d2a720, 0x20, 0x20}, {{0xb, 0x0}, {0xc002542729, 0x5}, 0xc04, {0x28f80d68, ...}, ...}, ...})
tendermint@v0.34.19/consensus/state.go:1949 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000d78e00, {{0x64ece60, 0xc004e3c1e8}, {0x0, 0x0}})\n\t/Users/skuchain/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/consensus/state.go:856 +0x45f\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000d78e00, 0x0)\n\t/Users/skuchain/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/consensus/state.go:783 +0x512\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/Users/skuchain/go/pkg/mod/github.com/tendermint/tendermint@v0.34.19/consensus/state.go:379 +0x13e\n"
11:45AM INF Stopping baseWAL service impl={"Logger":{}} module=consensus wal=/Users/skuchain/.hello/data/cs.wal/wal
11:45AM INF Stopping Group service impl={"Dir":"/Users/skuchain/.hello/data/cs.wal","Head":{"ID":"HPsaskddVKJm:/Users/skuchain/.hello/data/cs.wal/wal","Path":"/Users/skuchain/.hello/data/cs.wal/wal"},"ID":"group:HPsaskddVKJm:/Users/skuchain/.hello/data/cs.wal/wal","Logger":{}} module=consensus wal=/Users/skuchain/.hello/data/cs.wal/wal
11:45AM INF Timed out dur=3000 height=3076 module=consensus round=0 step=3
11:45AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
11:45AM INF No addresses to dial. Falling back to seeds module=pex
11:46AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
11:46AM INF No addresses to dial. Falling back to seeds module=pex
11:46AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
11:46AM INF No addresses to dial. Falling back to seeds module=pex

10. Now chain-app not accept any transactions after this above steps, once reached this state there is no option to recover the application back. Can not downgrade or upgrade due to inconsistent state.

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned