Slashing: supply invariance broken

[cwgoes]

Caught by #2617 (multi-seed simulation ranges over random genesis parameter combinations).

Replicable with:

go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=989182 -v -timeout 24h

on 3884b13

Fails with:

I[10-29|00:14:53.709] Absent validator 027E8AF0A5D7BC9508C550BA4097BA219703B32F at height 3, 3 missed, threshold 276 module=x/slashing
I[10-29|00:14:53.709] Absent validator 0F8DCCE06C47BC5313C334DCA3DC5092FB4AA140 at height 3, 1 missed, threshold 276 module=x/slashing 
I[10-29|00:14:53.709] Confirmed double sign from 33A2A55325BB76E45D801F94D8A853FE40FB687D at height 3, age of 0 less than max age of 214588000000000 module=x/slashing
I[10-29|00:14:53.709] Fraction slashed capped by slashing period from 500000000 to 500000000 module=x/slashing 
I[10-29|00:14:53.710] validator cosmosvaloper1xw3225e9hdmwghvqr72d32znleq0k6ra44kg0y slashed by slash factor of 0.0500000000; burned 223980000000000 tokens module=x/stake
I[10-29|00:14:53.710] validator cosmosvalcons1xw3225e9hdmwghvqr72d32znleq0k6rapx95r9 jailed module=x/stake 
I[10-29|00:14:53.710] Confirmed double sign from 0F8DCCE06C47BC5313C334DCA3DC5092FB4AA140 at height 0, age of 14568000000000 less than max age of 214588000000000 module=x/slashing
I[10-29|00:14:53.710] Fraction slashed capped by slashing period from 500000000 to 500000000 module=x/slashing 
I[10-29|00:14:53.710] validator cosmosvaloper1p7xuecrvg779xy7rxnw28hzsjta54g2qltg0m4 slashed by slash factor of 0.0500000000; burned 203604000000000 tokens module=x/stake
I[10-29|00:14:53.710] validator cosmosvalcons1p7xuecrvg779xy7rxnw28hzsjta54g2qtcmnh5 jailed module=x/stake 
Invariants broken after BeginBlock
bonded token invariance:
  pool.BondedTokens: 188465635000000000
  sum of account tokens: 188467622500000000
Begin block 0

[alexanderbez]

welp, I cannot reproduce this with the given seed/command. I created a new branch off of the latest develop and merged in your changes and it passed every time (I tried 6 or so times).

I did however, rerun the CI pipeline and failures still do occur in the multi-seed and the fast sim. I do not see this error however. So I will be closing for now unless it occurs again.

Perhaps we have some non-determinism again.

[cwgoes]

welp, I cannot reproduce this with the given seed/command. I created a new branch off of the latest develop and merged in your changes and it passed every time (I tried 6 or so times).

Did you try the specific commit referenced? Likely the specific seeds required on develop have changed, since we've merged some PRs since which change the simulation a bit (but that didn't address whatever the underlying cause of this problem is).

Like so:

git checkout 3884b13c004d1cd1936575f03b3e31d5bd07feb9
go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationBlockSize=200 -SimulationCommit=true -SimulationSeed=989182 -v -timeout 24h

I really don't think this is non-determinism, I can replicate it multiple times on different machines.

[alexanderbez]

You're correct. I should work off of that exact commit. I don't think anything committed recently went into addressing this potentially.

[rigelrozanski]

I would just run through a whole bunch of different seeds - I bet a a failing seed exists in < 100