Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concurrent map read and map write in GRPC server #10568

Closed
4 tasks
danil-lashin opened this issue Nov 18, 2021 · 8 comments
Closed
4 tasks

Concurrent map read and map write in GRPC server #10568

danil-lashin opened this issue Nov 18, 2021 · 8 comments
Assignees
@odeke-em

Comments

@danil-lashin
Copy link

Summary of Bug

Concurrent map read and map write sometimes kills application when accessing GRPC server. Related to #8591, #8549

fatal error: concurrent map read and map write

goroutine 115946 [running]:
runtime.throw({0x10230bc0a, 0x21})
	/usr/local/go/src/runtime/panic.go:1198 +0x54 fp=0x140043e7540 sp=0x140043e7510 pc=0x100916af4
runtime.mapaccess2_fast64(0x1027a3380, 0x1400029f650, 0xd)
	/usr/local/go/src/runtime/map_fast64.go:61 +0x25c fp=0x140043e75b0 sp=0x140043e7540 pc=0x1008ee46c
github.com/cosmos/iavl.(*MutableTree).VersionExists(0x140010b18b0, 0xd)
	/Users/daniillashin/go/pkg/mod/github.com/cosmos/iavl@v0.17.1/mutable_tree.go:72 +0x108 fp=0x140043e7680 sp=0x140043e75b0 pc=0x101558368
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).VersionExists(0x1400129a270, 0xd)
	/Users/daniillashin/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/store/iavl/store.go:145 +0x48 fp=0x140043e76b0 sp=0x140043e7680 pc=0x10157f568
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).GetImmutable(0x1400129a270, 0xd)
	/Users/daniillashin/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/store/iavl/store.go:93 +0x38 fp=0x140043e7750 sp=0x140043e76b0 pc=0x10157ee68
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).CacheMultiStoreWithVersion(0x140000c0460, 0xd)
	/Users/daniillashin/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/store/rootmulti/store.go:473 +0x1a0 fp=0x140043e7ab0 sp=0x140043e7750 pc=0x10158a2e0
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).createQueryContext(0x14001290820, 0xd, 0x0)
	/Users/daniillashin/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.3/baseapp/abci.go:637 +0x1b8 fp=0x140043e8800 sp=0x140043e7ab0 pc=0x1016dfbe8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x102a45418, 0x14004978960}, {0x102913920, 0x1038f19f0}, 0x1400497c000, 0x140049766c0)

Version

github.com/cosmos/cosmos-sdk v0.44.3

For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@tac0turtle
Copy link
Member

Version exists is a read mutex but it seems to mutate state. We will need to change it to a mutex.

@tac0turtle
Copy link
Member

This has been fixed in a new release of iavl.

@sgerogia sgerogia mentioned this issue Feb 4, 2022
Closed
4 tasks
@njmurarka
Copy link
Contributor

Is this resolved in the new Cosmos? I am on Cosmos 0.44.x and see this happening.

9:09PM INF executed block height=185035 module=state num_invalid_txs=0 num_valid_txs=0
fatal error: concurrent map read and map write

goroutine 4267938119 [running]:
runtime.throw({0x2654adf, 0x7f3a3e2bf108})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc031a842d8 sp=0xc031a842a8 pc=0x435a91
runtime.mapaccess2_fast64(0xc001cbb790, 0x21f82a0, 0x2d2c9)```

@alexanderbez
Copy link
Contributor

Hi @njmurarka, a few things:

  1. v0.44.x is not really supported, would strongly suggest updating to 0.46 or 0.47 in the coming days
  2. A sufficient stacktrace will be needed in order to help you
  3. We have not seen any of these issues on >= 0.45.x

@GAtom22
Copy link
Contributor

GAtom22 commented Mar 2, 2023 •
edited
Loading

Hi @alexanderbez @tac0turtle, we got a report from a keplr node that this is happening using cosmos-sdk v0.46.7 and iavl v0.19.4
I understand these versions should have the fix, right?

The issue is on the UnsavedFastIterator. Below is the stack trace

�[90m12:54PM�[0m �[32mINF�[0m executed block �[36mheight=�[0m9522889 �[36mmodule=�[0mstate �[36mnum_invalid_txs=�[0m0 �[36mnum_valid_txs=�[0m3 �[36mserver=�[0mnode
fatal error: concurrent map iteration and map write

goroutine 963386 [running]:
runtime.throw({0x27ef174?, 0x8?})
	runtime/panic.go:992 +0x71 fp=0xc0a2e49fb0 sp=0xc0a2e49f80 pc=0x43f0b1
runtime.mapiternext(0xc0a2e4a0a0?)
	runtime/map.go:871 +0x4eb fp=0xc0a2e4a020 sp=0xc0a2e49fb0 pc=0x41698b
github.com/cosmos/iavl.NewUnsavedFastIterator({0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1, 0xc0012f2880, 0xc07132da40, 0xc07132da70)
	github.com/cosmos/iavl@v0.19.4/unsaved_fast_iterator.go:55 +0x236 fp=0xc0a2e4a110 sp=0xc0a2e4a020 pc=0xe6f2d6
github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0018040f0, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0xe0?)
	github.com/cosmos/iavl@v0.19.4/mutable_tree.go:216 +0x10c fp=0xc0a2e4a178 sp=0xc0a2e4a110 pc=0xe55f2c
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc0705fe500?, {0xc09c5e3520?, 0x20?, 0xc0705f6720?}, {0xc09c5e3540?, 0x18?, 0xc09d8b26e0?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/iavl/store.go:244 +0x31 fp=0xc0a2e4a1c8 sp=0xc0a2e4a178 pc=0xe902d1
github.com/cosmos/cosmos-sdk/store/cache.(*CommitKVStoreCache).Iterator(0xc05efd0e94?, {0xc09c5e3520?, 0xc0452f3d40?, 0x195?}, {0xc09c5e3540?, 0x343ff98?, 0xc0019ca7e0?})
	<autogenerated>:1 +0x34 fp=0xc0a2e4a210 sp=0xc0a2e4a1c8 pc=0x1104b34
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc07134b0c0, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:180 +0x126 fp=0xc0a2e4a2b0 sp=0xc0a2e4a210 pc=0xe44246
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x64?, {0xc09c5e3520?, 0xc8?, 0x3?}, {0xc09c5e3540?, 0x343e980?, 0xc0000600c0?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:165 +0x29 fp=0xc0a2e4a300 sp=0xc0a2e4a2b0 pc=0xe44029
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0a2b20880, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:180 +0x126 fp=0xc0a2e4a3a0 sp=0xc0a2e4a300 pc=0xe44246
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc0705fe460?, {0xc09c5e3520?, 0x20?, 0xc0705fe480?}, {0xc09c5e3540?, 0x20?, 0xc0705fe4a0?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:165 +0x29 fp=0xc0a2e4a3f0 sp=0xc0a2e4a3a0 pc=0xe44029
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0a2b214c0, {0xc09c5e3520, 0x1e, 0x20}, {0xc09c5e3540, 0x1e, 0x20}, 0x1)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:180 +0x126 fp=0xc0a2e4a490 sp=0xc0a2e4a3f0 pc=0xe44246
github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc1507b9800?, {0xc09c5e3520?, 0x0?, 0xc09cd6ae61?}, {0xc09c5e3540?, 0x456ef4?, 0x20?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/cachekv/store.go:165 +0x29 fp=0xc0a2e4a4e0 sp=0xc0a2e4a490 pc=0xe44029
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc09d7b5560, {0xc09c5e3520?, 0x64?, 0x15?}, {0xc09c5e3540?, 0x20?, 0xc94e910000000000?}, 0x40?)
	github.com/cosmos/cosmos-sdk@v0.46.7/store/gaskv/store.go:98 +0x56 fp=0xc0a2e4a588 sp=0xc0a2e4a4e0 pc=0xca7d56
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc09d7bd530?, {0xc09c5e3520?, 0x344e8f0?, 0xc0a2b21880?}, {0xc09c5e3540?, 0x0?, 0xc05efd0e94?})
	github.com/cosmos/cosmos-sdk@v0.46.7/store/gaskv/store.go:74 +0x29 fp=0xc0a2e4a5d8 sp=0xc0a2e4a588 pc=0xca7b89
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/store.go:337 +0x156 fp=0xc0a2e4a908 sp=0xc0a2e4a5d8 pc=0x16ada36
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/delegation.go:80 +0x2d9 fp=0xc0a2e4b308 sp=0xc0a2e4a908 pc=0x16969f9
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/delegation.go:147 +0x1e5 fp=0xc0a2e4c170 sp=0xc0a2e4b308 pc=0x16973c5
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawDelegationRewards({{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/keeper.go:94 +0x198 fp=0xc0a2e4c548 sp=0xc0a2e4c170 pc=0x16a2638
github.com/cosmos/cosmos-sdk/x/distribution/keeper.msgServer.WithdrawDelegatorReward({{{0x34236a8, 0xc00298c550}, {0x344e0a0, 0xc0012237b0}, {{0x344e0a0, 0xc0012237b0}, 0xc00011d1e0, {0x34236a8, 0xc00298c580}, {0x34236f8, ...}, ...}, ...}}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/keeper/msg_server.go:63 +0x335 fp=0xc0a2e4da40 sp=0xc0a2e4c548 pc=0x16a4295
github.com/cosmos/cosmos-sdk/x/distribution/keeper.(*msgServer).WithdrawDelegatorReward(0x41196b?, {0x343e9f0?, 0xc0a2b39fb0?}, 0x2776600?)
	<autogenerated>:1 +0xc5 fp=0xc0a2e4dbd8 sp=0xc0a2e4da40 pc=0x16b5bc5
github.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler.func1({0x343e9f0, 0xc0a2b39fb0}, {0x26db920?, 0xc0a2b1d920})
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/types/tx.pb.go:657 +0x78 fp=0xc0a2e4dc18 sp=0xc0a2e4dbd8 pc=0x166b498
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x343ffd0, 0xc0a2b50580}, {0xc0a2e4df50?, 0x41196b?}, 0x2776600?, 0xc0a2b134b8)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/msg_service_router.go:113 +0xd2 fp=0xc0a2e4df08 sp=0xc0a2e4dc18 pc=0x11183b2
github.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler({0x2777080?, 0xc000017ec0}, {0x343ffd0, 0xc0a2b50580}, 0x3013700, 0xc0a2b54720)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/distribution/types/tx.pb.go:659 +0x138 fp=0xc0a2e4df60 sp=0xc0a2e4df08 pc=0x166b358
github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x343e980, 0xc0000600c0}, {0x344e8f0, 0xc0a2b21880}, {{0xb, 0x0}, {0xc05efd0e94, 0xc}, 0x914ec8, {0x3c658c6, ...}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/msg_service_router.go:126 +0x344 fp=0xc0a2e4ed18 sp=0xc0a2e4df60 pc=0x1118124
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x343e980, 0xc0000600c0}, {0x344e8f0, 0xc0a2b21880}, {{0xb, 0x0}, {0xc05efd0e94, 0xc}, 0x914ec8, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/baseapp.go:779 +0x625 fp=0xc0a2e4f3c8 sp=0xc0a2e4ed18 pc=0x11155e5
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc001854380, 0x2, {0xc0452f3d40, 0x195, 0x1a0})
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/baseapp.go:719 +0xc45 fp=0xc0a2e527f8 sp=0xc0a2e4f3c8 pc=0x1114385
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(0x4142e7?, {0xc0452f3d40?, 0x2543ec0?, 0x1?})
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/test_helpers.go:25 +0x2c fp=0xc0a2e52830 sp=0xc0a2e527f8 pc=0x111abcc
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate-fm({0xc0452f3d40?, 0x20?, 0xc09c7b9600?})
	<autogenerated>:1 +0x37 fp=0xc0a2e52860 sp=0xc0a2e52830 pc=0x1c15957
github.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x345a070, 0xc18b37c280}, 0x0, {0xc000152360, 0xc}, {0x3453450, 0xc0012237b0}, ...}, ...}, ...)
	github.com/cosmos/cosmos-sdk@v0.46.7/x/auth/tx/service.go:131 +0x12e fp=0xc0a2e52910 sp=0xc0a2e52860 pc=0x1186b2e
github.com/cosmos/cosmos-sdk/x/auth/tx.(*txServer).Simulate(0x7?, {0x343e9f0?, 0xc0a2b1e780?}, 0x343e9f0?)
	<autogenerated>:1 +0xa5 fp=0xc0a2e52cb8 sp=0xc0a2e52910 pc=0x118a205
github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x343e9f0, 0xc0a2b1e780}, {0x2698920?, 0xc09ced3a00})
	github.com/cosmos/cosmos-sdk@v0.46.7/types/tx/service.pb.go:933 +0x78 fp=0xc0a2e52cf8 sp=0xc0a2e52cb8 pc=0xe1f838
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x343e9f0, 0xc09d36ac60}, {0x2698920, 0xc09ced3a00}, 0xc09ced3a20?, 0xc09d34c9d8)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/grpcserver.go:68 +0x3e2 fp=0xc0a2e53808 sp=0xc0a2e52cf8 pc=0x1117642
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x343e9f0?, 0xc09d36ac60?}, {0x2698920?, 0xc09ced3a00?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a fp=0xc0a2e53848 sp=0xc0a2e53808 pc=0x1109e5a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x343e9f0?, 0xc09d36ac60?}, {0x2698920?, 0xc09ced3a00?}, 0x7f3676742d98?, 0xc09ced3a20?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4 fp=0xc0a2e538d8 sp=0xc0a2e53848 pc=0x110a064
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x343e9f0?, 0xc09d36ac60?}, {0x2698920?, 0xc09ced3a00?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a fp=0xc0a2e53918 sp=0xc0a2e538d8 pc=0x1109e5a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x343e9f0, 0xc09d36ac60}, {0x2698920, 0xc09ced3a00}, 0xc0e8075901?, 0xc0e80759b8?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbf fp=0xc0a2e53970 sp=0xc0a2e53918 pc=0x1109cff
github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x2630240?, 0xc18df0a540}, {0x343e9f0, 0xc09d36ac60}, 0xc0804b9570, 0xc09d36ac90)
	github.com/cosmos/cosmos-sdk@v0.46.7/types/tx/service.pb.go:935 +0x138 fp=0xc0a2e539c8 sp=0xc0a2e53970 pc=0xe1f6f8
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2({0x2630240, 0xc18df0a540}, {0x343e9f0, 0xc09d36ac60}, 0x270fd80?, 0xc09d3630e0?)
	github.com/cosmos/cosmos-sdk@v0.46.7/baseapp/grpcserver.go:82 +0xdc fp=0xc0a2e53a28 sp=0xc0a2e539c8 pc=0x11171fc
google.golang.org/grpc.(*Server).processUnaryRPC(0xc002282000, {0x344c468, 0xc0ed00da00}, 0xc09d3630e0, 0xc0e9ef0330, 0xc066cdab00, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13 fp=0xc0a2e53e48 sp=0xc0a2e53a28 pc=0xa676b3
google.golang.org/grpc.(*Server).handleStream(0xc002282000, {0x344c468, 0xc0ed00da00}, 0xc09d3630e0, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b fp=0xc0a2e53f68 sp=0xc0a2e53e48 pc=0xa6c6bb
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.51.0/server.go:965 +0x98 fp=0xc0a2e53fe0 sp=0xc0a2e53f68 pc=0xa65158
runtime.goexit()
	runtime/asm_amd64.s:1571 +0x1 fp=0xc0a2e53fe8 sp=0xc0a2e53fe0 pc=0x472b41
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.51.0/server.go:963 +0x28a

@GAtom22
Copy link
Contributor

GAtom22 commented Mar 2, 2023 •
edited
Loading

I managed to reproduce the error using concurrency on this test, replacing these lines for

       // first delegator withdraws
	wg.Add(1)
	go func() {
		defer wg.Done()
		expRewards := sdk.Coins{sdk.NewCoin(sdk.DefaultBondDenom, math.NewInt(initial*3/4))}
		bankKeeper.EXPECT().SendCoinsFromModuleToAccount(ctx, disttypes.ModuleName, addr, expRewards)
		_, err = distrKeeper.WithdrawDelegationRewards(ctx, addr, valAddr)
		require.NoError(t, err)
	}()

	// second delegator withdraws
	wg.Add(1)
	go func() {
		defer wg.Done()
		expRewards := sdk.Coins{sdk.NewCoin(sdk.DefaultBondDenom, math.NewInt(initial*1/4))}
		bankKeeper.EXPECT().SendCoinsFromModuleToAccount(ctx, disttypes.ModuleName, sdk.AccAddress(valConsAddr1), expRewards)
		_, err = distrKeeper.WithdrawDelegationRewards(ctx, sdk.AccAddress(valConsAddr1), valAddr)
		require.NoError(t, err)
	}()

	wg.Wait()

I ran it multiple times, sometimes passes, and sometimes panics.
Here's the stack trace with the same error as above:

fatal error: concurrent map iteration and map write

goroutine 66 [running]:
github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0002f8140, 0x1e, 0x20}, {0xc0002f8180, 0x1e, 0x20}, 0x1, 0xc000340d00, 0xc0002353e0, 0xc000235410)
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/unsaved_fast_iterator.go:57 +0x236
github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0002b7270, {0xc0002f8140, 0x1e, 0x20}, {0xc0002f8180, 0x1e, 0x20}, 0x0?)
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:218 +0x10c
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc0004da400?, {0xc0002f8140?, 0x0?, 0xc00059c091?}, {0xc0002f8180?, 0x454314?, 0x20?})
        /home/tom/evmos/cosmos-sdk/store/iavl/store.go:247 +0x31
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc000300cc0, {0xc0002f8140?, 0x7d0?, 0x15?}, {0xc0002f8180?, 0x20?, 0x400000000000000?}, 0x80?)
        /home/tom/evmos/cosmos-sdk/store/gaskv/store.go:98 +0x56
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc00030c3c0?, {0xc0002f8140?, 0x1b7b150?, 0xc00018afc0?}, {0xc0002f8180?, 0x0?, 0x0?})
        /home/tom/evmos/cosmos-sdk/store/gaskv/store.go:74 +0x29
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x1b60c10, 0xc000694dd0}, {0x7fb48866c740, 0xc000694fa0}, {0x1b71930, 0xc000231bc0}, {0x1b750e0, 0xc000231ba0}, {0x1b788d8, 0xc000231bb0}, ...}, ...)
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/store.go:337 +0x16a
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x1b60c10, 0xc000694dd0}, {0x7fb48866c740, 0xc000694fa0}, {0x1b71930, 0xc000231bc0}, {0x1b750e0, 0xc000231ba0}, {0x1b788d8, 0xc000231bb0}, ...}, ...)
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/delegation.go:82 +0x330
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x1b60c10, 0xc000694dd0}, {0x7fb48866c740, 0xc000694fa0}, {0x1b71930, 0xc000231bc0}, {0x1b750e0, 0xc000231ba0}, {0x1b788d8, 0xc000231bb0}, ...}, ...)
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/delegation.go:149 +0x23b
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawDelegationRewards({{0x1b60c10, 0xc000694dd0}, {0x7fb48866c740, 0xc000694fa0}, {0x1b71930, 0xc000231bc0}, {0x1b750e0, 0xc000231ba0}, {0x1b788d8, 0xc000231bb0}, ...}, ...)
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/keeper.go:95 +0x1ce
github.com/cosmos/cosmos-sdk/x/distribution/keeper_test.TestCalculateRewardsMultiDelegatorMultWithdraw.func1()
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/delegation_test.go:799 +0x368
created by github.com/cosmos/cosmos-sdk/x/distribution/keeper_test.TestCalculateRewardsMultiDelegatorMultWithdraw
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/delegation_test.go:795 +0x1c4a

While doing this, also observed other concurrency issues:

fatal error: concurrent map read and map write

goroutine 50 [running]:
github.com/cosmos/iavl.(*MutableTree).Get(0xc000161720, {0xc00033c210, 0x1e, 0x30})
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:155 +0x5f
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Get(0x15?, {0xc00033c210?, 0x1721030?, 0x1e?})
        /home/tom/evmos/cosmos-sdk/store/iavl/store.go:209 +0x88
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Get(0xc000709440, {0xc00033c210, 0x1e, 0x30})
        /home/tom/evmos/cosmos-sdk/store/gaskv/store.go:37 +0x71
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.GetValidatorHistoricalRewards({{0x1b60c10, 0xc000119a00}, {0x7f4cbac93b80, 0xc000119b20}, {0x1b71930, 0xc00022f4b0}, {0x1b750e0, 0xc00022f490}, {0x1b788d8, 0xc00022f4a0}, ...}, ...)
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/store.go:130 +0xf9
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.decrementReferenceCount({{0x1b60c10, 0xc000119a00}, {0x7f4cbac93b80, 0xc000119b20}, {0x1b71930, 0xc00022f4b0}, {0x1b750e0, 0xc00022f490}, {0x1b788d8, 0xc00022f4a0}, ...}, ...)
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/validator.go:79 +0xbb
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x1b60c10, 0xc000119a00}, {0x7f4cbac93b80, 0xc000119b20}, {0x1b71930, 0xc00022f4b0}, {0x1b750e0, 0xc00022f490}, {0x1b788d8, 0xc00022f4a0}, ...}, ...)

fatal error: concurrent map writes

goroutine 68 [running]:
github.com/cosmos/iavl.(*MutableTree).addUnsavedAddition(...)
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:945
github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc000301720, 0xc000368960, {0x26c5092, 0x1, 0x1}, {0x26c54d8, 0x0, 0x0}, 0xc0001f0408)
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:250 +0x151
github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc000301720, 0xc0003399a0, {0x26c5092, 0x1, 0x1}, {0x26c54d8, 0x0, 0x0}, 0xc0001f0408)
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:288 +0x8d1
github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc000301720, 0xc000339900, {0x26c5092, 0x1, 0x1}, {0x26c54d8, 0x0, 0x0}, 0xc0001f0408)
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:288 +0x8d1
github.com/cosmos/iavl.(*MutableTree).recursiveSet(0xc000301720, 0xc000339860, {0x26c5092, 0x1, 0x1}, {0x26c54d8, 0x0, 0x0}, 0xc0001f0408)
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:288 +0x8d1
github.com/cosmos/iavl.(*MutableTree).set(0xc000301720, {0x26c5092, 0x1, 0x1}, {0x26c54d8, 0x0, 0x0})
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:239 +0x350
github.com/cosmos/iavl.(*MutableTree).Set(0xc0001f04a0?, {0x26c5092?, 0xc000297620?, 0xc0001f04d8?}, {0x26c54d8?, 0xc00010afc0?, 0x1b67b10?})
        /home/tom/go/pkg/mod/github.com/cosmos/iavl@v0.20.0-alpha1/mutable_tree.go:136 +0x28
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Set(0xc0000188a0, {0x26c5092?, 0x172530f?, 0xb20ec5?}, {0x26c54d8?, 0x1b67b10?, 0xc000297620?})
        /home/tom/evmos/cosmos-sdk/store/iavl/store.go:200 +0x4d
github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Set(0xc0007ad7a0, {0x26c5092, 0x1, 0x1}, {0x26c54d8, 0x0, 0x0})
        /home/tom/evmos/cosmos-sdk/store/gaskv/store.go:54 +0xff
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.SetFeePool({{0x1b60c10, 0xc00016abc0}, {0x7fbf484edac0, 0xc00016acc0}, {0x1b71930, 0xc000791510}, {0x1b750e0, 0xc0007914f0}, {0x1b788d8, 0xc000791500}, ...}, ...)
        /home/tom/evmos/cosmos-sdk/x/distribution/keeper/store.go:61 +0x158
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x1b60c10, 0xc00016abc0}, {0x7fbf484edac0, 0xc00016acc0}, {0x1b71930, 0xc000791510}, {0x1b750e0, 0xc0007914f0}, {0x1b788d8, 0xc000791500}, ...}, ...)

@GAtom22
Copy link
Contributor

GAtom22 commented Mar 2, 2023 •
edited
Loading

All these seem to be related to concurrent read and write operations to the unsavedFastNodeAdditions & unsavedFastNodeRemovals maps on the UnsavedFastIterator and MutableTree structs.

I tried adding a mutex and use the lock when these operations happen and seems to solve the issue.

@GAtom22 GAtom22 mentioned this issue Mar 2, 2023
Closed
@alexanderbez
Copy link
Contributor

I have no idea if these methods are supposed to be thread-safe or not.

@evan-forbes evan-forbes mentioned this issue Nov 15, 2023
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@odeke-em odeke-em

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@njmurarka @danil-lashin @odeke-em @alexanderbez @tac0turtle @GAtom22