Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consensus failure on Gaia-7001a - Slashing bug #1797

Closed
4 tasks
hendrikhofstadt opened this issue Jul 23, 2018 · 5 comments
Closed
4 tasks

Consensus failure on Gaia-7001a - Slashing bug #1797

hendrikhofstadt opened this issue Jul 23, 2018 · 5 comments
Assignees
@cwgoes

Comments

@hendrikhofstadt
Copy link
Contributor

hendrikhofstadt commented Jul 23, 2018 •
edited
Loading

Consensus failed because of a bug in the slashing logic

I[07-23|19:24:00.842] Absent validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 at height 2512, 0 signed, threshold 50 module=x/slashing
I[07-23|19:24:00.842] Validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 past min height of 2510 and below signed blocks threshold of 50 module=x/slashing
E[07-23|19:24:00.842] WARNING: Ignored attempt to slash a nonexistent validator with address DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7, we recommend you investigate immediately module=x/stake
E[07-23|19:24:00.863] CONSENSUS FAILURE!!!                         module=consensus err="Validator with pubkey PubKeyEd25519{162125EDAF7A35A6836951F9995F442BFEAD39458BE669244F8E804512805888} not found, cannot set revoked to true" stack="goroutine 1333 [running]:
runtime/debug.Stack(0xc423de9458, 0xcc1c80, 0xc423b3b070)
	/usr/lib/golang/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func1(0xc4200fc600)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:558 +0x57
panic(0xcc1c80, 0xc423b3b070)
	/usr/lib/golang/src/runtime/panic.go:505 +0x229
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.setRevoked(0xfd9480, 0xc42006ffa0, 0xc4208bc150, 0xfd9480, 0xc42006ff80, 0xf63e28, 0xc4208bc150, 0x4, 0xfe0b80, 0xc4232953b0, ...)
	go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/slash.go:140 +0x26f
github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Revoke(0xfd9480, 0xc42006ffa0, 0xc4208bc150, 0xfd9480, 0xc42006ff80, 0xf63e28, 0xc4208bc150, 0x4, 0xfe0b80, 0xc4232953b0, ...)
	go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/slash.go:120 +0xa5
github.com/cosmos/cosmos-sdk/x/slashing.Keeper.handleValidatorSignature(0xfd9480, 0xc42006ffb0, 0xc4208bc150, 0xfe5c40, 0xc420924440, 0xc4208bc150, 0xfd9480, 0xc42006ffe0, 0xa, 0xfe0b80, ...)
	go/src/github.com/cosmos/cosmos-sdk/x/slashing/keeper.go:107 +0x923
github.com/cosmos/cosmos-sdk/x/slashing.BeginBlocker(0xfe0b80, 0xc4232953b0, 0xc42628eac0, 0x9, 0xc423e63ac0, 0x14, 0x20, 0xc422f53400, 0xa, 0x9d0, ...)
	go/src/github.com/cosmos/cosmos-sdk/x/slashing/tick.go:28 +0x1ec
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).BeginBlocker(0xc4200f05a0, 0xfe0b80, 0xc4232953b0, 0xc42628eac0, 0x9, 0xc423e63ac0, 0x14, 0x20, 0xc422f53400, 0xa, ...)
	go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:136 +0xd0
github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).BeginBlocker-fm(0xfe0b80, 0xc4232953b0, 0xc42628eac0, 0x9, 0xc423e63ac0, 0x14, 0x20, 0xc422f53400, 0xa, 0x9d0, ...)
	go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:108 +0xa0
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc420016240, 0xc423e63ac0, 0x14, 0x20, 0xc422f53400, 0xa, 0x9d0, 0x5b562b50, 0x0, 0x3f, ...)
	go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:431 +0x1ef
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc42089e720, 0xc423e63ac0, 0x14, 0x20, 0xc422f53400, 0xa, 0x9d0, 0x5b562b50, 0x0, 0x3f, ...)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:206 +0xab
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc4207e8ef0, 0xc423e63ac0, 0x14, 0x20, 0xc422f53400, 0xa, 0x9d0, 0x5b562b50, 0x0, 0x3f, ...)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:69 +0x78
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0xfe1180, 0xc4219c45c0, 0xfe5760, 0xc4207e8ef0, 0xc42564cb40, 0xc423455920, 0xfe9880, 0xc4200de2b8, 0x1, 0xc424cbaac0, ...)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:190 +0x547
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc42089e600, 0xc420470770, 0xa, 0x9cf, 0x3f, 0xc425a2e660, 0x14, 0x20, 0x1, 0xc424cbaac0, ...)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:76 +0x12f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc4200fc600, 0x9d0)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1289 +0xbb2
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc4200fc600, 0x9d0)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1220 +0x468
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc4200fc600, 0x0, 0x9d0)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1168 +0x98
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc4200fc600, 0x9d0, 0x0)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1197 +0x802
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc4200fc600, 0xc4233dd860, 0xc424176660, 0x28, 0xf9, 0x0, 0x0)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1600 +0xbb4
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc4200fc600, 0xc4233dd860, 0xc424176660, 0x28, 0xf9, 0xf1)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1458 +0x56
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc4200fc600, 0xcc46e0, 0xc422ac2ad8, 0xc424176660, 0x28)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:628 +0x64f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc4200fc600, 0x0)
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:580 +0x6e0
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
	go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:295 +0x140

For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@adrianbrink
Copy link
Contributor

Another stacktrace:

I[07-23|19:24:00.842] Absent validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 at height 2512, 0 signed, threshold 50 module=x/slashing
I[07-23|19:24:00.842] Validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 past min height of 2510 and below signed blocks threshold of 50 module=x/slashing
E[07-23|19:24:00.842] WARNING: Ignored attempt to slash a nonexistent validator with address DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7, we recommend you investigate immediately module=x/stake
E[07-23|19:24:00.843] CONSENSUS FAILURE!!!                         module=consensus err="Validator with pubkey PubKeyEd25519{162125EDAF7A35A6836951F9995F442BFEAD39458BE669244F8E804512805888} not found, cannot
 set revoked to true" stack="goroutine 1102 [running]:\nruntime/debug.Stack(0xc4279f3458, 0xcc0d20, 0xc426f62eb0)\n\t/home/adrian/.gvm/gos/go1.10.3/src/runtime/debug/stack.go:24 +0xa7\ngithub.com/cosmos/cosmo
s-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func1(0xc4200ee900)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/ten
dermint/tendermint/consensus/state.go:558 +0x57\npanic(0xcc0d20, 0xc426f62eb0)\n\t/home/adrian/.gvm/gos/go1.10.3/src/runtime/panic.go:502 +0x229\ngithub.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.setRevoked(
0xfd8220, 0xc42093fab0, 0xc4208517a0, 0xfd8220, 0xc42093fa90, 0xf62e38, 0xc4208517a0, 0x4, 0xfdf920, 0xc422fac990, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/x/stake/ke
eper/slash.go:140 +0x26f\ngithub.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.Revoke(0xfd8220, 0xc42093fab0, 0xc4208517a0, 0xfd8220, 0xc42093fa90, 0xf62e38, 0xc4208517a0, 0x4, 0xfdf920, 0xc422fac990, ...)\n\t/
home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/slash.go:120 +0xa5\ngithub.com/cosmos/cosmos-sdk/x/slashing.Keeper.handleValidatorSignature(0xfd8220, 0xc42093fac0, 0xc
4208517a0, 0xfe49e0, 0xc4208c30c0, 0xc4208517a0, 0xfd8220, 0xc42093faf0, 0xa, 0xfdf920, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/x/slashing/keeper.go:107 +0x923\ngith
ub.com/cosmos/cosmos-sdk/x/slashing.BeginBlocker(0xfdf920, 0xc422fac990, 0xc429afb140, 0x9, 0xc4231e1400, 0x14, 0x20, 0xc422b5aab0, 0xa, 0x9d0, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com
/cosmos/cosmos-sdk/x/slashing/tick.go:28 +0x1ec\ngithub.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).BeginBlocker(0xc420010f00, 0xfdf920, 0xc422fac990, 0xc429afb140, 0x9, 0xc4231e1400, 0x14, 0x20, 0xc422b5aa
b0, 0xa, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:136 +0xd0\ngithub.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).BeginBlocker-fm(0xfdf920, 0xc422
fac990, 0xc429afb140, 0x9, 0xc4231e1400, 0x14, 0x20, 0xc422b5aab0, 0xa, 0x9d0, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:108 +0xa0\ngithub.com/cosm
os/cosmos-sdk/baseapp.(*BaseApp).BeginBlock(0xc4200aa480, 0xc4231e1400, 0x14, 0x20, 0xc422b5aab0, 0xa, 0x9d0, 0x5b562b50, 0x0, 0x3f, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cos
mos-sdk/baseapp/baseapp.go:431 +0x1ef\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync(0xc4200849c0, 0xc4231e1400, 0x14, 0x20, 0xc422b5aab0, 0xa,
 0x9d0, 0x5b562b50, 0x0, 0x3f, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:206 +0xab\ngithub.com/cosm
os/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync(0xc42007b630, 0xc4231e1400, 0x14, 0x20, 0xc422b5aab0, 0xa, 0x9d0, 0x5b562b50, 0x0, 0x3f, ...)\n\t/home/adrian/.gv
m/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:69 +0x78\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBl
ockOnProxyApp(0xfdff20, 0xc420946e20, 0xfe4500, 0xc42007b630, 0xc425208960, 0xc42502fbc0, 0xfe8620, 0xc42000e1d0, 0x1, 0xc4291ad7e0, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cos
mos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:190 +0x547\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc4208e1500, 0xc4208a3440
, 0xa, 0x9cf, 0x3f, 0xc424bb9e40, 0x14, 0x20, 0x1, 0xc4291ad7e0, ...)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:7
6 +0x12f\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).finalizeCommit(0xc4200ee900, 0x9d0)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmo
s/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1289 +0xbb2\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryFinalizeCommit(0xc4200e
e900, 0x9d0)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1220 +0x468\ngithub.com/cosmos/cosmos-sdk/vendor/github.co
m/tendermint/tendermint/consensus.(*ConsensusState).enterCommit.func1(0xc4200ee900, 0x0, 0x9d0)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tende
rmint/consensus/state.go:1168 +0x98\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterCommit(0xc4200ee900, 0x9d0, 0x0)\n\t/home/adrian/.gvm/pkgsets/go1.10.
3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1197 +0x802\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusStat
e).addVote(0xc4200ee900, 0xc425fcf2c0, 0xc420717530, 0x28, 0xf9, 0x0, 0x0)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state
.go:1600 +0xbb4\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc4200ee900, 0xc425fcf2c0, 0xc420717530, 0x28, 0xf9, 0xf1)\n\t/home/adrian/.gvm/pk
gsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1458 +0x56\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*C
onsensusState).handleMsg(0xc4200ee900, 0xcc3780, 0xc42595ad58, 0xc420717530, 0x28)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consens
us/state.go:628 +0x64f\ngithub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc4200ee900, 0x0)\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/gith
ub.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:580 +0x6e0\ncreated by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnSt
art\n\t/home/adrian/.gvm/pkgsets/go1.10.3/global/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:295 +0x140\n"

@cwgoes cwgoes self-assigned this Jul 23, 2018
@cwgoes
Copy link
Contributor

cwgoes commented Jul 23, 2018 •
edited
Loading

The slashing logic (at the time of consensus fault, at least) is behaving as intended: if validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 was in the Tendermint validator set and should have signed the block, it should have been in the staking store. Looks like that invariant was broken previous to the fault somehow.

edit: Looks like validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 was slashed and revoked at block 2511, passed back to Tendermint as a validator set update, but still in the validator set for block 2512:

10642 I[07-24|00:16:56.681] Absent validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 at height 2511, 0 signed, threshold 50 module=x/slashing
10643 I[07-24|00:16:56.681] Validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 past min height of 2510 and below signed blocks threshold of 50 module=x/slashing
10644 I[07-24|00:16:56.682] Slashing at current height 2511, not scanning unbonding delegations & redelegations module=x/stake
10645 I[07-24|00:16:56.689] Validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 slashed by slashFactor 1/100, burned 3/10 tokens module=x/stake
10646 I[07-24|00:16:56.695] Validator DF54B862410BCD9B5C516FBB6C56BDC826ED6FA7 revoked module=x/stake
10648 I[07-24|00:16:56.709] Executed block                               module=state height=2511 validTxs=1 invalidTxs=0
10649 I[07-24|00:16:56.709] Updates to validators                        module=state updates="[{\"address\":\"\",\"pub_key\":\"FiEl7a96NaaDaVH5mV9EK/6tOUWL5mkkT46ARRKAWIg=\",\"power\":0}]"
10650 I[07-24|00:16:56.722] Committed state                              module=state height=2511 txs=1 appHash=F6382E15AA2DD5A3E5D3C2FD67859505D032BAF4

The store record was deleted since the validator was unbonded from in block 2511 - as expected - but the validator should have never been in the Tendermint validator set for block 2512.

Strangely, the validator sets as queried directly (2511, 2512) are correct.

@gin
Copy link
Contributor

gin commented Jul 24, 2018

🤔

@cwgoes
Copy link
Contributor

cwgoes commented Jul 24, 2018 •
edited
Loading

The issue appears to be that Tendermint sends signing info for the last block on abci.RequestBeginBlock.SigningValidators, while the SDK expected signing info for the current block.

@cwgoes cwgoes mentioned this issue Jul 24, 2018
Merged
9 tasks
@cwgoes
Copy link
Contributor

cwgoes commented Jul 24, 2018

Fixed by #1805. Thanks all for the bug reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cwgoes cwgoes

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gin @hendrikhofstadt @adrianbrink @cwgoes