Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Racing conditions on UnsavedFastIterator & MutableTree #696

Closed
GAtom22 opened this issue Mar 2, 2023 · 8 comments
Closed

Racing conditions on UnsavedFastIterator & MutableTree #696

GAtom22 opened this issue Mar 2, 2023 · 8 comments
Assignees
@tac0turtle @cool-develope

Comments

@GAtom22
Copy link

GAtom22 commented Mar 2, 2023

Detected an issue with concurrent map operations to the unsavedFastNodeAdditions & unsavedFastNodeRemovals maps on the UnsavedFastIterator and MutableTree structs.

I tried adding a mutex and use the lock when these operations happen and seems to solve the issue.

More info here cosmos/cosmos-sdk#10568 (comment)

Error with the stack trace:

�[90m12:54PM�[0m �[32mINF�[0m executed block �[36mheight=�[0m9522889 �[36mmodule=�[0mstate �[36mnum_invalid_txs=�[0m0 �[36mnum_valid_txs=�[0m3 �[36mserver=�[0mnode
fatal error: concurrent map iteration and map write

goroutine 963386 [running]:
runtime.throw({0x27ef174?, 0x8?})
	runtime/panic.go:992 +0x71 fp=0xc0a2e49fb0 sp=0xc0a2e49f80 pc=0x43f0b1
runtime.mapiternext(0xc0a2e4a0a0?)
	runtime/map.go:871 +0x4eb fp=0xc0a2e4a020 sp=0xc0a2e49fb0 pc=0x41698b
github.com/cosmos/iavl.NewUnsavedFastIterator({0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1, 0xc0012f2880, 0xc07132da40, 0xc07132da70)
	github.com/cosmos/iavl@v0.19.4/unsaved_fast_iterator.go:55 +0x236 fp=0xc0a2e4a110 sp=0xc0a2e4a020 pc=0xe6f2d6
github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0018040f0, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0xe0?)
	github.com/cosmos/iavl@v0.19.4/mutable_tree.go:216 +0x10c fp=0xc0a2e4a178 sp=0xc0a2e4a110 pc=0xe55f2c
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc0705fe500?, {0xc09c5e3520?, 0x20?, 0xc0705f6720?}, {0xc09c5e3540?, 0x18?, 0xc09d8b26e0?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/iavl/store.go:244 +0x31 fp=0xc0a2e4a1c8 sp=0xc0a2e4a178 pc=0xe902d1
github.com/cosmos/cosmos-sdk/store/cache.(*CommitKVStoreCache).Iterator(0xc05efd0e94?, {0xc09c5e3520?, 0xc0452f3d40?, 0x195?}, {0xc09c5e3540?, 0x343ff98?, 0xc0019ca7e0?})
	<autogenerated>:1 +0x34 fp=0xc0a2e4a210 sp=0xc0a2e4a1c8 pc=0x1104b34
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc07134b0c0, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:180 +0x126 fp=0xc0a2e4a2b0 sp=0xc0a2e4a210 pc=0xe44246
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x64?, {0xc09c5e3520?, 0xc8?, 0x3?}, {0xc09c5e3540?, 0x343e980?, 0xc0000600c0?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:165 +0x29 fp=0xc0a2e4a300 sp=0xc0a2e4a2b0 pc=0xe44029
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0a2b20880, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:180 +0x126 fp=0xc0a2e4a3a0 sp=0xc0a2e4a300 pc=0xe44246
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc0705fe460?, {0xc09c5e3520?, 0x20?, 0xc0705fe480?}, {0xc09c5e3540?, 0x20?, 0xc0705fe4a0?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:165 +0x29 fp=0xc0a2e4a3f0 sp=0xc0a2e4a3a0 pc=0xe44029
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0a2b214c0, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:180 +0x126 fp=0xc0a2e4a490 sp=0xc0a2e4a3f0 pc=0xe44246
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc1507b9800?, {0xc09c5e3520?, 0x0?, 0xc09cd6ae61?}, {0xc09c5e3540?, 0x456ef4?, 0x20?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:165 +0x29 fp=0xc0a2e4a4e0 sp=0xc0a2e4a490 pc=0xe44029
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc09d7b5560, {0xc09c5e3520?, 0x64?, 0x15?}, {0xc09c5e3540?, 0x20?, 0xc94e910000000000?}, 0x40?)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/gaskv/store.go:98 +0x56 fp=0xc0a2e4a588 sp=0xc0a2e4a4e0 pc=0xca7d56
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc09d7bd530?, {0xc09c5e3520?, 0x344e8f0?, 0xc0a2b21880?}, {0xc09c5e3540?, 0x0?, 0xc05efd0e94?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/gaskv/store.go:74 +0x29 fp=0xc0a2e4a5d8 sp=0xc0a2e4a588 pc=0xca7b89
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/store.go:337 +0x156 fp=0xc0a2e4a908 sp=0xc0a2e4a5d8 pc=0x16ada36
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/delegation.go:80 +0x2d9 fp=0xc0a2e4b308 sp=0xc0a2e4a908 pc=0x16969f9
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/delegation.go:147 +0x1e5 fp=0xc0a2e4c170 sp=0xc0a2e4b308 pc=0x16973c5
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawDelegationRewards({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/keeper.go:94 +0x198 fp=0xc0a2e4c548 sp=0xc0a2e4c170 pc=0x16a2638
github.com/cosmos/cosmos-sdk/x/distribution/keeper.msgServer.WithdrawDelegatorReward({{{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/msg_server.go:63 +0x335 fp=0xc0a2e4da40 sp=0xc0a2e4c548 pc=0x16a4295
github.com/cosmos/cosmos-sdk/x/distribution/keeper.(*msgServer).WithdrawDelegatorReward(0x41196b?, {0x343e9f0?, 0xc0a2b39fb0?}, 0x2776600?)
	<autogenerated>:1 +0xc5 fp=0xc0a2e4dbd8 sp=0xc0a2e4da40 pc=0x16b5bc5
github.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler.func1({0x343e9f0, 0xc0a2b39fb0}, {0x26db920?, 0xc0a2b1d920})
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/types/tx.pb.go:657 +0x78 fp=0xc0a2e4dc18 sp=0xc0a2e4dbd8 pc=0x166b498
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x343ffd0, 0xc0a2b50580}, {0xc0a2e4df50?, 0x41196b?}, 0x2776600?, 0xc0a2b134b8)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/msg_service_router.go:113 +0xd2 fp=0xc0a2e4df08 sp=0xc0a2e4dc18 pc=0x11183b2
github.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler({0x2777080?, 0xc000017ec0}, {0x343ffd0, 0xc0a2b50580}, 0x3013700, 0xc0a2b54720)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/types/tx.pb.go:659 +0x138 fp=0xc0a2e4df60 sp=0xc0a2e4df08 pc=0x166b358
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x343e980, 0xc0000600c0}, {0x344e8f0, 0xc0a2b21880}, {{0xb, 0x0}, {0xc05efd0e94, 0xc}, 0x914ec8, {0x3c658c6, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/msg_service_router.go:126 +0x344 fp=0xc0a2e4ed18 sp=0xc0a2e4df60 pc=0x1118124
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x343e980, 0xc0000600c0}, {0x344e8f0, 0xc0a2b21880}, {{0xb, 0x0}, {0xc05efd0e94, 0xc}, 0x914ec8, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/baseapp.go:779 +0x625 fp=0xc0a2e4f3c8 sp=0xc0a2e4ed18 pc=0x11155e5
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc001854380, 0x2, {0xc0452f3d40, 0x195, 0x1a0})
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/baseapp.go:719 +0xc45 fp=0xc0a2e527f8 sp=0xc0a2e4f3c8 pc=0x1114385
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(0x4142e7?, {0xc0452f3d40?, 0x2543ec0?, 0x1?})
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/test_helpers.go:25 +0x2c fp=0xc0a2e52830 sp=0xc0a2e527f8 pc=0x111abcc
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate-fm({0xc0452f3d40?, 0x20?, 0xc09c7b9600?})
	<autogenerated>:1 +0x37 fp=0xc0a2e52860 sp=0xc0a2e52830 pc=0x1c15957
github.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x345a070, 0xc18b37c280}, 0x0, {0xc000152360, 0xc}, {0x3453450, 0xc0012237b0}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/auth/tx/service.go:131 +0x12e fp=0xc0a2e52910 sp=0xc0a2e52860 pc=0x1186b2e
github.com/cosmos/cosmos-sdk/x/auth/tx.(*txServer).Simulate(0x7?, {0x343e9f0?, 0xc0a2b1e780?}, 0x343e9f0?)
	<autogenerated>:1 +0xa5 fp=0xc0a2e52cb8 sp=0xc0a2e52910 pc=0x118a205
github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x343e9f0, 0xc0a2b1e780}, {0x2698920?, 0xc09ced3a00})
	github.com/cosmos/cosmos-sdk@v0.46.7/types/tx/service.pb.go:933 +0x78 fp=0xc0a2e52cf8 sp=0xc0a2e52cb8 pc=0xe1f838
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x343e9f0, 0xc09d36ac60}, {0x2698920, 0xc09ced3a00}, 0xc09ced3a20?, 0xc09d34c9d8)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/grpcserver.go:68 +0x3e2 fp=0xc0a2e53808 sp=0xc0a2e52cf8 pc=0x1117642
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x343e9f0?, 0xc09d36ac60?}, {0x2698920?, 0xc09ced3a00?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a fp=0xc0a2e53848 sp=0xc0a2e53808 pc=0x1109e5a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x343e9f0?, 0xc09d36ac60?}, {0x2698920?, 0xc09ced3a00?}, 0x7f3676742d98?, 0xc09ced3a20?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4 fp=0xc0a2e538d8 sp=0xc0a2e53848 pc=0x110a064
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x343e9f0?, 0xc09d36ac60?}, {0x2698920?, 0xc09ced3a00?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a fp=0xc0a2e53918 sp=0xc0a2e538d8 pc=0x1109e5a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x343e9f0, 0xc09d36ac60}, {0x2698920, 0xc09ced3a00}, 0xc0e8075901?, 0xc0e80759b8?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbf fp=0xc0a2e53970 sp=0xc0a2e53918 pc=0x1109cff
github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x2630240?, 0xc18df0a540}, {0x343e9f0, 0xc09d36ac60}, 0xc0804b9570, 0xc09d36ac90)
	github.com/cosmos/cosmos-sdk@v0.46.7/types/tx/service.pb.go:935 +0x138 fp=0xc0a2e539c8 sp=0xc0a2e53970 pc=0xe1f6f8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2({0x2630240, 0xc18df0a540}, {0x343e9f0, 0xc09d36ac60}, 0x270fd80?, 0xc09d3630e0?)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/grpcserver.go:82 +0xdc fp=0xc0a2e53a28 sp=0xc0a2e539c8 pc=0x11171fc
google.golang.org/grpc.(*Server).processUnaryRPC(0xc002282000, {0x344c468, 0xc0ed00da00}, 0xc09d3630e0, 0xc0e9ef0330, 0xc066cdab00, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13 fp=0xc0a2e53e48 sp=0xc0a2e53a28 pc=0xa676b3
google.golang.org/grpc.(*Server).handleStream(0xc002282000, {0x344c468, 0xc0ed00da00}, 0xc09d3630e0, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b fp=0xc0a2e53f68 sp=0xc0a2e53e48 pc=0xa6c6bb
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.51.0/server.go:965 +0x98 fp=0xc0a2e53fe0 sp=0xc0a2e53f68 pc=0xa65158
runtime.goexit()
	runtime/asm_amd64.s:1571 +0x1 fp=0xc0a2e53fe8 sp=0xc0a2e53fe0 pc=0x472b41
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.51.0/server.go:963 +0x28a
@0xphilipp
Copy link

0xphilipp commented Apr 25, 2023 •
edited
Loading

We found that our node also had this issue when simulating some things in parallel. (On Terra)

Apr 25 09:14:24

INF executed block height=4785411 module=state num_invalid_txs=0 num_valid_txs=3
 fatal error: concurrent map iteration and map write
 goroutine 1842358 [running, locked to thread]:
 github.com/cosmos/iavl.NewUnsavedFastIterator({0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x1, 0xc000555a80, 0xc016e5ecc0, 0xc016e5ecf0)
         github.com/cosmos/iavl@v0.19.5/unsaved_fast_iterator.go:55 +0x236
 github.com/cosmos/iavl.(*MutableTree).Iterator(0xc001631ae0, {0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x20?)
         github.com/cosmos/iavl@v0.19.5/mutable_tree.go:216 +0x10c
 github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0x1fd8740?, {0xc018c10660?, 0x597b14?, 0x3108270?}, {0xc018c106a0?, 0x1fd8740?, 0x30df400?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/iavl/store.go:249 +0x31
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc018363040, {0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x1)
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:185 +0x134
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x7f56ea9e73c8?, {0xc018c10660?, 0xc0000c0c00?, 0xc01aa85bc0?}, {0xc018c106a0?, 0xc01a89f1f0?, 0x415b07?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:170 +0x29
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc01952e500, {0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x1)
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:185 +0x134
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x9?, {0xc018c10660?, 0xe308253?, 0xedbd990e4?}, {0xc018c106a0?, 0xc00b895920?, 0x20?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:170 +0x29
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc01952f640, {0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x1)
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:185 +0x134
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc016208160?, {0xc018c10660?, 0x20?, 0xc010ec4930?}, {0xc018c106a0?, 0x18?, 0xc001974000?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:170 +0x29
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc01abb4b80, {0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x1)
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:185 +0x134
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x3?, {0xc018c10660?, 0x1e?, 0x1e?}, {0xc018c106a0?, 0x64?, 0x64?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:170 +0x29
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc01abb5c00, {0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x1)
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:185 +0x134
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc00b895d00?, {0xc018c10660?, 0x20?, 0xc00b895fc0?}, {0xc018c106a0?, 0x20?, 0xc016208020?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:170 +0x29
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc01abd2100, {0xc018c10660, 0x1e, 0x20}, {0xc018c106a0, 0x1e, 0x20}, 0x1)
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:185 +0x134
 github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x7f56ea9e73c8?, {0xc018c10660?, 0xc0000c0c00?, 0xc018c106a0?}, {0xc018c106a0?, 0xc01a89f750?, 0x458e96?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/cachekv/store.go:170 +0x29
 github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc01aa85bc0, {0xc018c10660?, 0x64?, 0x15?}, {0xc018c106a0?, 0x20?, 0x305490000000000?}, 0xa0?)
         github.com/cosmos/cosmos-sdk@v0.46.11/store/gaskv/store.go:98 +0x56
 github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc00c4b33b0?, {0xc018c10660?, 0x3119420?, 0xc01abd26c0?}, {0xc018c106a0?, 0x0?, 0xc016d48281?})
         github.com/cosmos/cosmos-sdk@v0.46.11/store/gaskv/store.go:74 +0x29
 github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x30efc08, 0xc001666540}, {0x31189a0, 0xc000ed34d0}, {{0x31189a0, 0xc000ed34d0}, 0xc000537a08, {0x30efc08, 0xc001666570}, {0x30efc58, ...}, ...}, ...}, ...)
         github.com/cosmos/cosmos-sdk@v0.46.11/x/distribution/keeper/store.go:337 +0x155
 github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x30efc08, 0xc001666540}, {0x31189a0, 0xc000ed34d0}, {{0x31189a0, 0xc000ed34d0}, 0xc000537a08, {0x30efc08, 0xc001666570}, {0x30efc58, ...}, ...}, ...}, ...)
         github.com/cosmos/cosmos-sdk@v0.46.11/x/distribution/keeper/delegation.go:80 +0x2d9
 github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.DelegationRewards({{0x30efc08, 0xc001666540}, {0x31189a0, 0xc000ed34d0}, {{0x31189a0, 0xc000ed34d0}, 0xc000537a08, {0x30efc08, 0xc001666570}, {0x30efc58, ...}, ...}, ...}, ...)
         github.com/cosmos/cosmos-sdk@v0.46.11/x/distribution/keeper/grpc_query.go:149 +0x385
 github.com/CosmWasm/wasmd/x/wasm/keeper.getAccumulatedRewards({{0x3108270, 0xc01abbe780}, {0x3119420, 0xc01abb5e80}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, {0xe308253, ...}, ...}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:455 +0x1a2
 github.com/CosmWasm/wasmd/x/wasm/keeper.sdkToFullDelegation({{0x3108270, 0xc01abbe780}, {0x3119420, 0xc01abb5e80}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, {0xe308253, ...}, ...}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:432 +0x3b3
 github.com/CosmWasm/wasmd/x/wasm/keeper.StakingQuerier.func1({{0x3108270, 0xc01abbe780}, {0x3119420, 0xc01abb5e80}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, {0xe308253, ...}, ...}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:358 +0x285
 github.com/CosmWasm/wasmd/x/wasm/keeper.QueryPlugins.HandleQuery({_, _, _, _, _, _}, {{0x3108270, 0xc01abbe780}, {0x3119420, 0xc01abb5e80}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:154 +0x195
 github.com/CosmWasm/wasmd/x/wasm/keeper.QueryHandler.Query({{{0x3108270, 0xc01abbe780}, {0x3119420, 0xc01abb4f00}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, {0xe308253, ...}, ...}, ...}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:55 +0x2ca
 github.com/CosmWasm/wasmvm/types.RustQuery({0x30f1be8, 0xc01ab9c600}, {0xc01a96a8f0, 0xad, 0xad}, 0x10f?)
         github.com/CosmWasm/wasmvm@v1.1.2/types/queries.go:50 +0x18b
 github.com/CosmWasm/wasmvm/internal/api.cQueryExternal(0x20?, 0x20?, 0x7f56c1c25e70, {0x20?, 0x7f56b8019450?, 0xc00b895d00?}, 0x7f56c1c25e78, 0x7f56c1c25f00)
         github.com/CosmWasm/wasmvm@v1.1.2/internal/api/callbacks.go:455 +0x13f
 github.com/CosmWasm/wasmvm/internal/api._C2func_query(0x7f569c01a0e0, {0x0, 0xc018c103e0, 0x20}, {0x0, 0xc01a956cc0, 0xbd}, {0x0, 0xc003b88a50, 0xc}, ...)
         _cgo_gotypes.go:587 +0x72
 github.com/CosmWasm/wasmvm/internal/api.Query.func1({0xc0066a23a0?}, {0xc0?, 0xc018c103e0?, 0xc0018a9b00?}, {0x40?, 0xc01a956cc0?, 0x1c80100?}, {0x0, 0xc003b88a50, 0xc}, ...)
         github.com/CosmWasm/wasmvm@v1.1.2/internal/api/lib.go:359 +0x21e
 github.com/CosmWasm/wasmvm/internal/api.Query({0xc01a8a4a90?}, {0xc018c103e0?, 0xc01ac12d40?, 0x20?}, {0xc01a956cc0?, 0x219b0a0?, 0x3e8?}, {0xc003b88a50, 0xc, 0xc}, ...)
         github.com/CosmWasm/wasmvm@v1.1.2/internal/api/lib.go:359 +0x5ab
 github.com/CosmWasm/wasmvm.(*VM).Query(0xc001667180, {0xc018c103e0, 0x20, 0x20}, {{0x490502, 0x175922aaacc66a53, {0xc016d48281, 0x9}}, 0x0, {{0xc00d4ba1c0, ...}}}, ...)
         github.com/CosmWasm/wasmvm@v1.1.2/lib.go:226 +0x205
 github.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.QuerySmart({{0x30efc08, 0xc001666640}, {0x311f4f8, 0xc000ed34d0}, {0x3106f70, 0xc00188ce60}, {0x30ebb00, 0xc0016671a0}, {0x30e7840, 0xc00057e1c0}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/keeper.go:716 +0x983
 github.com/CosmWasm/wasmd/x/wasm/keeper.WasmQuerier.func1({{0x3108200, 0xc000056090}, {0x3119420, 0xc01abb4f00}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, {0xe308253, ...}, ...}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:483 +0x4bd
 github.com/CosmWasm/wasmd/x/wasm/keeper.QueryPlugins.HandleQuery({_, _, _, _, _, _}, {{0x3108200, 0xc000056090}, {0x3119420, 0xc01abb4f00}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:160 +0xfa
 github.com/CosmWasm/wasmd/x/wasm/keeper.QueryHandler.Query({{{0x3108200, 0xc000056090}, {0x3119420, 0xc01952f9c0}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, {0xe308253, ...}, ...}, ...}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/query_plugins.go:55 +0x2ca
 github.com/CosmWasm/wasmvm/types.RustQuery({0x30f1be8, 0xc00a207b00}, {0xc01a964b80, 0x80, 0x80}, 0x1a?)
         github.com/CosmWasm/wasmvm@v1.1.2/types/queries.go:50 +0x18b
 github.com/CosmWasm/wasmvm/internal/api.cQueryExternal(0x0?, 0xc8?, 0x7f56c1c275d0, {0x3?, 0x7f56b8921ec0?, 0xc000056090?}, 0x7f56c1c275d8, 0x7f56c1c27660)
         github.com/CosmWasm/wasmvm@v1.1.2/internal/api/callbacks.go:455 +0x13f
 github.com/CosmWasm/wasmvm/internal/api._C2func_execute(0x7f569c01a0e0, {0x0, 0xc00ad5c120, 0x20}, {0x0, 0xc01847f500, 0xbd}, {0x0, 0xc0195205a0, 0x44}, ...)
         _cgo_gotypes.go:304 +0x72
 github.com/CosmWasm/wasmvm/internal/api.Execute.func1({0x1?}, {0x97?, 0xc00ad5c120?, 0x1?}, {0x90?, 0xc01847f500?, 0x2?}, {0x0, 0xc0195205a0, 0x44}, ...)
         github.com/CosmWasm/wasmvm@v1.1.2/internal/api/lib.go:211 +0x247
 github.com/CosmWasm/wasmvm/internal/api.Execute({0x30dcb78?}, {0xc00ad5c120?, 0x1?, 0xc01a8a94e0?}, {0xc01847f500?, 0xc00eb27160?, 0x20?}, {0xc0195205a0, 0x44, 0x50}, ...)
         github.com/CosmWasm/wasmvm@v1.1.2/internal/api/lib.go:211 +0x6f1
 github.com/CosmWasm/wasmvm.(*VM).Execute(0xc001667180, {0xc00ad5c120, 0x20, 0x20}, {{0x490502, 0x175922aaacc66a53, {0xc016d48281, 0x9}}, 0x0, {{0xc00dd412c0, ...}}}, ...)
         github.com/CosmWasm/wasmvm@v1.1.2/lib.go:186 +0x272
 github.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.execute({{0x30efc08, 0xc001666640}, {0x311f4f8, 0xc000ed34d0}, {0x3106f70, 0xc00188ce60}, {0x30ebb00, 0xc0016671a0}, {0x30e7840, 0xc00057e1c0}, ...}, ...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/keeper.go:426 +0xbbd
 github.com/CosmWasm/wasmd/x/wasm/keeper.PermissionedKeeper.Execute(...)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/contract_keeper.go:95
 github.com/CosmWasm/wasmd/x/wasm/keeper.msgServer.ExecuteContract({{0x3119778?, 0xc00059c2e0?}}, {0x3108270, 0xc019564e10}, 0xc0022ad810)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/keeper/msg_server.go:139 +0x71b
 github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler.func1({0x3108270, 0xc019564e10}, {0x23269e0?, 0xc0022ad810})
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/types/tx.pb.go:926 +0x78
 github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x31094d0, 0xc018bc7b80}, {0xc01a8ac538?, 0x41312b?}, 0x2389520?, 0xc011ccc1c8)
         github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/msg_service_router.go:113 +0x90
 github.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler({0x21f3f40?, 0xc000e3ca30}, {0x31094d0, 0xc018bc7b80}, 0x2cc6b00, 0xc00eb27060)
         github.com/CosmWasm/wasmd@v0.30.0/x/wasm/types/tx.pb.go:928 +0x138
 github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x3108200, 0xc000056090}, {0x3119420, 0xc01952f9c0}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, {0xe308253, ...}, ...}, ...}, ...)
         github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/msg_service_router.go:126 +0x323
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x3108200, 0xc000056090}, {0x3119420, 0xc01952f9c0}, {{0xb, 0x0}, {0xc016d48281, 0x9}, 0x490502, ...}, ...}, ...)
         github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/baseapp.go:778 +0x265
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc000de3a40, 0x2, {0xc018b8ce00, 0x35d, 0x380})
         github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/baseapp.go:718 +0xd45
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(...)
         github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/test_helpers.go:25
 github.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x3129bd8, 0xc008b12080}, 0x0, {0x0, 0x0}, {0x311f4f8, 0xc000ed34d0}, ...}, ...}, ...)
         github.com/cosmos/cosmos-sdk@v0.46.11/x/auth/tx/service.go:131 +0x12e
 github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x3108270, 0xc019523aa0}, {0x22cc400?, 0xc00eb26440})
         github.com/cosmos/cosmos-sdk@v0.46.11/types/tx/service.pb.go:933 +0x78
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x3108270, 0xc01950d7d0}, {0x22cc400, 0xc00eb26440}, 0xc00eb26460?, 0xc011493620)
         github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/grpcserver.go:68 +0x3b0
 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3108270?, 0xc01950d7d0?}, {0x22cc400?, 0xc00eb26440?})
         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3108270?, 0xc01950d7d0?}, {0x22cc400?, 0xc00eb26440?}, 0x212b560?, 0xc00eb26460?)
         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x3108270?, 0xc01950d7d0?}, {0x22cc400?, 0xc00eb26440?})
         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x3108270, 0xc01950d7d0}, {0x22cc400, 0xc00eb26440}, 0xc0124a2901?, 0xc0124a29f0?)
         github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
 github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x226da20?, 0xc008e52000}, {0x3108270, 0xc01950d7d0}, 0xc018bec150, 0xc01950d800)
         github.com/cosmos/cosmos-sdk@v0.46.11/types/tx/service.pb.go:935 +0x138
 github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2({0x226da20, 0xc008e52000}, {0x3108270, 0xc01950d7d0}, 0x23374a0?, 0xc00a261560?)
         github.com/cosmos/cosmos-sdk@v0.46.11/baseapp/grpcserver.go:82 +0xdc
 google.golang.org/grpc.(*Server).processUnaryRPC(0xc0090805a0, {0x3116aa0, 0xc00dfeb6c0}, 0xc00a261560, 0xc008be9500, 0xc007133180, 0x0)
         google.golang.org/grpc@v1.53.0/server.go:1336 +0xd33
 google.golang.org/grpc.(*Server).handleStream(0xc0090805a0, {0x3116aa0, 0xc00dfeb6c0}, 0xc00a261560, 0x0)
         google.golang.org/grpc@v1.53.0/server.go:1704 +0xa36
 google.golang.org/grpc.(*Server).serveStreams.func1.2()
         google.golang.org/grpc@v1.53.0/server.go:965 +0x98
 created by google.golang.org/grpc.(*Server).serveStreams.func1
         google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

@0xphilipp
Copy link

Is there any update to this? I can point our script to any terra node if you need to reproduce the issue. Usually nodes die very quickly on it.

@tac0turtle
Copy link
Member

We fixed a few things with this issue but we will look into it

@cool-develope
Copy link
Collaborator

@GAtom22 please share the detailed context of how to reproduce the above issue.
Theoretically, the current iavl doesn't provide the concurrency, I am thinking it is an issue of simulation.

@GAtom22
Copy link
Author

GAtom22 commented Jul 11, 2023 •
edited
Loading

@cool-develope this issue was reported by Keplr team. They mentioned they got this error message on their Evmos node and seems to be related to simulation. In the stack trace provided on the issue description, you can see it invokes github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate

@javiersuweijie
Copy link

javiersuweijie commented Jul 12, 2023 •
edited
Loading

Was looking into this issue and the Simulate API call does not hold any lock on the iavl so calling simulate concurrently leads to referencing the store in multiple go routines.

@giskook made a comment here that make sense https://github.com/cosmos/iavl/pull/468/files#r959060203

Can consider to wrap unsavedFastNodeAdditions iterations and updates around using a lock. javiersuweijie@7fb4bb1

This seems to fix it but since iavl doesn't support concurrency, I'm not sure if the fix should be to make sure cosmos-sdk does not use it in parallel in the first place.

This was referenced Jul 31, 2023
Closed
Closed
Merged
@adizere
Copy link

adizere commented Oct 4, 2023

Very similar to cometbft/cometbft#815 (comment)

@tac0turtle
Copy link
Member

we have been testing on mainnets iavlv1 and have not run into this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tac0turtle tac0turtle

@cool-develope cool-develope

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@adizere @javiersuweijie @0xphilipp @tac0turtle @cool-develope @GAtom22