Unable to transfer ethers in Raft using web3js due to high gas estimate

[kbhokray]

System information

Geth version: geth version

Geth
Version: 1.7.2-stable
Git Commit: df4267a25637a5497a3db9fbde4603a3dcd6aa14
Quorum Version: 2.0.2
Architecture: amd64
Network Id: 1
Go Version: go1.10.2
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

OS & Version: Windows/Linux/OSX
Ubuntu 16.04.4 LTS

Branch, Commit Hash or Release: git status
Quorum: v2.0.2

Expected behaviour

Gas estimate for ether transfer should be minimal

Actual behaviour

Gas estimate for ether transfer > 3 * 10 ^ 9

Steps to reproduce the behaviour

Download quorum-maker and start a Development Network. Query the started nodes to estimate the gas requirement for ether transfer

curl -X POST --data '{ "jsonrpc":"2.0", "id":2, "method":"eth_estimateGas", "params":[ { "to":"0xE2F384B526C57309e7798B9E8455BF05bE89A465", "value":"0x64", "from":"0x7798673316D6028b7fF2CF7369aa658276e5C1C5" } ] }' http://10.50.0.2:22000

The resulting gas estimate is too high (3754428764).

I'm trying to transfer ethers from one account to another and web3js is not allowing me to do it because the gas estimate is greater than the block gas limit. I am using truffle-hdwallet-provider. Is there any other way to transfer ethers in nodejs?

Other Details

quorum-maker internally uses this command to start the network:

geth --verbosity 6 --datadir qdata --raft --nodiscover --networkid 41418 --rpc --rpcaddr 0.0.0.0 --rpcapi admin,db,eth,debug,miner,net,shh,txpool,personal,web3,quorum,raft --emitcheckpoints --raftport 22003 --rpcport 22000 --port 22001 --nat extip:10.50.0.2

More details here:
https://ethereum.stackexchange.com/q/55587/38431

[fixanoid]

Hi, can you please make sure that genesis file you are using matches this one: https://github.com/jpmorganchase/quorum-examples/blob/master/examples/7nodes/genesis.json

[kbhokray]

Hi, I have copied the genesis into the template that quorum-maker uses. All config except the alloc part is the same now:

{
  "config": {
        "byzantiumBlock": 1,
        "chainId": 10,
        "eip150Block": 1,
        "eip155Block": 0,
        "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "eip158Block": 1,
        "isQuorum":true
    },
  "alloc"      : {
    "0xec4227a4c65d5868fdb82ce707f56319e0c72947":{"balance":"1000000000000000000000000000"},"0x4396535f4a474e72cad5846126a477548783ee3c":{"balance":"1000000000000000000000000000"},"0x6829340b1a79591a40ab17de55450c66b690086a":{"balance":"1000000000000000000000000000"}
  },
  "coinbase"   : "0x0000000000000000000000000000000000000000",
  "difficulty" : "0x0",
  "extraData"  : "0x0000000000000000000000000000000000000000000000000000000000000000",
  "gasLimit"   : "0xE0000000",
  "nonce"      : "0x0",
  "mixhash"    : "0x00000000000000000000000000000000000000647572616c65787365646c6578",
  "parentHash" : "0x0000000000000000000000000000000000000000000000000000000000000000",
  "timestamp"  : "0x00"
}

Still
curl -X POST --data '{ "jsonrpc":"2.0", "id":2, "method":"eth_estimateGas", "params":[ { "to":"0xec4227a4c65d5868fdb82ce707f56319e0c72947", "value":"0x200", "from":"0x4396535f4a474e72cad5846126a477548783ee3c" } ] }' http://10.50.0.2:22000 returns
{"jsonrpc":"2.0","id":2,"result":"0xdfc80951"}

[kbhokray]

I just tried the same thing on the 7 nodes example using raft consensus.

cat qdata/dd2/keystore/key2
> {"address":"ca843569e3427144cead5e4d5999a3d0ccf92b8e" ...

cat qdata/dd1/keystore/key1
> {"address":"ed9d02e382b34818e88b88a309c7fe71e65f419d" ...

curl -X POST --data '{ "jsonrpc":"2.0", "id":2, "method":"eth_estimateGas", "params":[ { "to":"0xed9d02e382b34818e88b88a309c7fe71e65f419d", "value":"0x200", "from":"0xca843569e3427144cead5e4d5999a3d0ccf92b8e" } ] }' http://localhost:22000
> {"jsonrpc":"2.0","id":2,"result":"0xe0000000"}

Even the 7 nodes example gives a high estimate

[viestursr]

Hi, I'm getting the same issue with the genesis file that you provided. A huge gas limit that's limiting transactions.

[kbhokray]

I just tried setting the gasLimit to the puppeth default (0x3b9aca00). It still gives a high estimate

[SatpalSandhu61]

Hi, have you actually tried submitting the transaction to transfer the ether?
I've tried running this on the 7-nodes example, I also see the same high value returned by eth_estimateGas(), however if I submit the transaction with 21000 gas it does actually succeed:

curl -X POST --data '{ "jsonrpc":"2.0", "id":2, "method":"eth_estimateGas", "params":[ { "to":"0xa9e871f88cbeb870d32d88e4221dcfbd36dd635a", "value":"0x200", "from":"0xed9d02e382b34818e88b88a309c7fe71e65f419d" } ] }' http://localhost:22000
{"jsonrpc":"2.0","id":2,"result":"0xe0000000"}

eth.sendTransaction({from:"0xed9d02e382b34818e88b88a309c7fe71e65f419d",to:"0xa9e871f88cbeb870d32d88e4221dcfbd36dd635a",value:0x200,gasPrice:0,gas:21000})
"0xb581617b0223532645984da8416c4f83d14c41a3e0d71ae8eabf9d09d189960a"

[kbhokray]

@SatpalSandhu61 when I do the transfer from one local account to another using cli, it does succeed. But when I try to send a transaction from a non-local account using truffle-hdwallet-provider using its seedwords, it gets stuck in the txpool and all subsequent transactions fail with the error replacement transaction underpriced. After debugging the issue a bit, I figured that the high gas estimate might be the reason.

> txpool.inspect
{
  pending: {
    0xE2F384B526C57309e7798B9E8455BF05bE89A465: {
      6: "0x7798673316D6028b7fF2CF7369aa658276e5C1C5: 10000000000000000000000 wei + 3748019791 × 0 gas"
    }
  },
  queued: {}
}

The sender account has sufficient balance, allocated to it in the genesis. I am facing this issue in quorum-maker which starts the node with the command
geth --verbosity 6 --datadir qdata --raft --nodiscover --networkid 41418 --rpc --rpcaddr 0.0.0.0 --rpcapi admin,db,eth,debug,miner,net,shh,txpool,personal,web3,quorum,raft --emitcheckpoints --raftport 22003 --rpcport 22000 --port 22001 --nat extip:10.50.0.2
Not tried it on 7-nodes yet. Let me know if you would want the exact steps to reproduce the issue and I will.

[SatpalSandhu61]

Hi kbhokray, we are investigating the issue.
If you can give the exact steps to reproduce then it would help.

[kbhokray]

I have added a minimal code here:
https://github.com/kbhokray/quorum-tx-test

The code tries to do two transactions from an account in 1. Ropsten and 2. Custom Quorum network. The quorum-maker default genesis has been changed to include the transferring address

Steps to reproduce

Test on Ropsten

1. Insert Ropsten RPC url in quorum-transfer/index.js
2. Run the code by passing ROPSTEN as env variable

$ cd quorum-transfer
$ ENV=ROPSTEN npm start

Output:

Web3 intialized with default account: 0xe2f384b526c57309e7798b9e8455bf05be89a465
Available balance: 19686595299999850000
Using RPC Url: // RPC_URL
Transaction 1: Transfering 100 wei to 0x1398f371170139894f53cf3b5e266a675d7c657b
Please consult tx: 0x0a848b00da322ee82397100c412d7c0c8102a4a09476372850682615dd316c28
Transaction 2: Transfering 100 wei to 0x1398f371170139894f53cf3b5e266a675d7c657b
Please consult tx: 0x320e9be0df9836ed54b2697cc1037ee81eeaa7dcede428fe96d2a8b65b2292be

The transfers take place without errors

Test on Quorum

1. Start the network in development mode

$ cd quorum-maker
$ ./setup.sh
> option: 4
$ cd TestNetwork
$ docker-compose up

2. Open another terminal and run the code by passing QUORUM as env variable

$ cd quorum-transfer
$ ENV=QUORUM npm start

Output:

Web3 intialized with default account: 0xe2f384b526c57309e7798b9e8455bf05be89a465
Available balance: 1e+50
Using RPC Url: http://10.50.0.2:22000
Transaction 1: Transfering 100 wei to 0x1398f371170139894f53cf3b5e266a675d7c657b
Please consult tx: 0x9db82a25313c1ef6c80628ec4c9a01025898bbf584a6e0c5a7799992e3d79313
Transaction 2: Transfering 100 wei to 0x1398f371170139894f53cf3b5e266a675d7c657b
Error transferring ethers: known transaction: 9db82a25313c1ef6c80628ec4c9a01025898bbf584a6e0c5a7799992e3d79313

The last line shows the error. Now all further transactions error out because the transaction is stuck in pending state, which can be seen by attaching to the container and starting geth console inside it.

> txpool.inspect
{
  pending: {
    0xE2F384B526C57309e7798B9E8455BF05bE89A465: {
      0: "0x1398f371170139894f53CF3B5E266a675d7C657B: 100 wei + 3754428753 × 0 gas"
    }
  },
  queued: {}
}

The genesis is generated dynamically upon running setup.sh and can be seen in the network directory once it is created.

[kegsay]

I mentioned this in another issue, the root cause here is:

It's worth noting that estimateGas is buggy if you use Raft because BlockByNumber was edited to return the latest block instead of the pending block. You generally cannot do this safely due to how the gas limit tends towards the amount of gas actually being used, meaning the returned estimate will be greater than the gas block limit.

To clarify: the block gas limit isn't a constant; it will be slightly different for each block. Ethereum will try to adjust the block gas limit depending on actual usage, in a "use it or lose it" fashion. That means if you only use a bit of gas but have a huge gas limit, the next block's gas limit will be lower than the previous block.

This isn't a problem for Ethereum, but is a problem for Quorum (using Raft) because the BlockByNumber function was modified in an unsafe way. When someone wants the pending block (which comes after the current block) they changed it to return the current block instead, commenting that:

// Pending block is only known by the miner

Unfortunately, estimateGas uses this function in order to estimate the gas ceiling; that being the max amount of gas it can possibly estimate a given transaction will use. Due to the change in behaviour of BlockByNumber, this maximum is now greater than the actual pending block gas limit (in most cases), meaning it will always result in a value which can never possibly work.

In other words, if you set a massive gas limit in your genesis block that ironically will be causing the problem because of the downward trend in block gas limits. We've resolved this by simply not estimating gas if using Raft, but the real (albeit technically distasteful) fix would be to use the same calculation as the miner when estimating gas, since you can't actually share the pending block state when using Raft.

[SatpalSandhu61]

Issue root cause is that the EVM is failing with ErrReadOnlyValueTransfer, so the gas estimate is being increased up to the max value. The EVM failure occurs because private state has not been initialised and the EVM believes we are calling from private to public state, so it sets the evm.quorumReadOnly flag.
Solution is to ensure that private state is correctly initialised.

Code change to fix this issue is being worked on, in order for it to be applied to the v1.8.12 branch.

Note that the v1.8.12 code has been tightened up in this area, so that an EVM failure is caught and reported, rather than a max gas estimate just being returned.