Validator crashes in 2.2.2 #676
Comments
|
Hi @jmcnevin, can you please paste your genesis.json |
{
"config": {
"chainId": 19827658,
"homesteadBlock": 1,
"eip150Block": 2,
"eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"eip155Block": 3,
"eip158Block": 3,
"byzantiumBlock": 4,
"istanbul": {
"epoch": 30000,
"policy": 0
},
"isQuorum": true,
"txnSizeLimit": 64
},
"nonce": "0x0",
"timestamp": "0x5a7fd1b0",
"extraData": "0x0000000000000000000000000000000000000000000000000000000000000000f89af8549408e9e9cd8ca072b435832d00d3a266f6cf7444ac941daf95f9c9129c44c888345ae0ee4bde9152b3fa9446f4512d6cddb8c776046103b1022d5b33058ebd94fdbd79619bee02829e53b0331062ece15367676fb8410000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0",
"gasLimit": "0x47b760",
"difficulty": "0x1",
"mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
"coinbase": "0x0000000000000000000000000000000000000000",
"alloc": {
"08e9e9cd8ca072b435832d00d3a266f6cf7444ac": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
},
"1daf95f9c9129c44c888345ae0ee4bde9152b3fa": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
},
"46f4512d6cddb8c776046103b1022d5b33058ebd": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
},
"fdbd79619bee02829e53b0331062ece15367676f": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
}
},
"number": "0x0",
"gasUsed": "0x0",
"parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
} |
|
Thanks @jmcnevin. What is the TPS when you are seeing the issue. Was this network also upgraded from 2.2.1 to 2.2.2 |
|
Yes this was a network upgraded from 2.2.1. Wasn't able to clock TPS, it happens so quickly, but I did manage to do a screen capture of an incident. |
|
I have been trying to reproduce this, by upgrading the network from v2.2.1 to v2.2.2, but so far no luck even with sustained 500 TPS. The only thing I had to change was increase the gasLimit. Can you increase the gasLimit and try again and see if you get better luck |
|
I faced the same error in v2.2.3. It's a 4-node Istanbul network init-ed in v2.2.3. Doing 1TPS only. One of the 4 nodes returned Genesis as follows: {
"config": {
"chainId": 2017,
"homesteadBlock": 0,
"byzantiumBlock": 0,
"eip150Block": 0,
"eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"eip155Block": 0,
"eip158Block": 0,
"istanbul": {
"epoch": 30000,
"policy": 0
},
"isQuorum": true
},
"nonce": "0x0",
"timestamp": "0x5ca70e02",
"extraData": "0x0000000000000000000000000000000000000000000000000000000000000000f89af854948e8a891401c591089ae6af3adcad81851580a47794cc6b795beb02477975c58615924010881bc4ab229403848249a94f393867a1c2243c791a020b7176d79471b8ea022c3eba13cdc4b3182d7c65d1e59d8849b8410000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0",
"gasLimit": "0x2faf0800",
"difficulty": "0x1",
"mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
"coinbase": "0x0000000000000000000000000000000000000000",
"alloc": {
"03848249a94f393867a1c2243c791a020b7176d7": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
},
"71b8ea022c3eba13cdc4b3182d7c65d1e59d8849": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
},
"8e8a891401c591089ae6af3adcad81851580a477": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
},
"cc6b795beb02477975c58615924010881bc4ab22": {
"balance": "0x446c3b15f9926687d2c40534fdb564000000000000"
}
},
"number": "0x0",
"gasUsed": "0x0",
"parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
}Command to start geth --datadir qdata --nodiscover --verbosity 5 --unlock 0 --password password.txt --gasprice 0 --rpc --rpccorsdomain '*' --rpcaddr 0.0.0.0 --rpcport 21000 --rpcapi admin,db,eth,debug,net,shh,txpool,personal,web3,quorum,istanbul --port 30303 --networkid 2017concurrent-map-read-and-map-write.txt |
|
Hi @thpun, looks like it could be some race condition. I am trying to reproduce it by mimicking my setup similar to yours and submitting transactions at 1 tps, but so far no luck. Can you provide more details like what kind and how you are submitting the transactions. Also, how is the network setup. Are all the nodes running on different hosts? |
|
Actually I was trying the same thing in issue #669 with a clean network set up with v2.2.3. |
|
I should note, too, that the transactions we are testing with range from 800K-1mil gas. |
|
Hi @jmcnevin and @thpun, I have been trying to reproduce the above issue, but have not been able to so far. These are below scenarios I have tried so far;
In the both cases, I sent transactions each of size 64k and with 1, 10 and 20 TPS and it did not see any issues; below is the sample block when running 10 TPS As can be seen from above that avg transaction size is 64k and gas is around 1.4 mil. Block period is 1 sec, which is the default when it is not specified as geth startup argument. I am using the below contract Can you reproduce this issue at will, and if yes, is it possible to give a reproducible test case. |
|
Reproducible for me. I'm using the following contract: Perhaps I used a And keep firing transaction at 1 TPS: And 2 of the nodes got I'm testing in a 4-node network with v2.2.3. Each node is in different host. Each host has 4 vCPUs and 16GB memory. |
|
@thpun, you are right, this could be because of the mapping to store the values. I am able to reproduce, once I added the mapping |
|
I will try testing this branch today |
|
I did have one of my validators crash with this code, but it's a different panic: |
|
Same as @jmcnevin. Got the same panic in one of the validators: |
|
Looking much more stable on my end. 🤞 |
|
Didnt see the same error so far. |
|
Fixed in master |
System information
Geth version: 2.2.2
OS & Version: Linux
Expected behaviour
Validators should not crash.
Actual behaviour
With IBFT, under load, we are able to get validators to crash with errors such as
fatal error: concurrent map writes.Steps to reproduce the behaviour
We have a four-node validator set with validators running the following options:
Backtrace
dump.txt
The text was updated successfully, but these errors were encountered: