CI End to End Tests Not Operational

[petermetz]

Describe the bug

npm run test:bc is not working.

To Reproduce

Run npm run test:bc observe failure.

Expected behavior

Tests are passing.

Logs/Stack traces

Bunch of host not reachable errors quoting localhost IP addresses (which is suspect because the containers have static, non-localhost private IPs.

Cloud provider or hardware configuration:
Dev machine and also in CI.

Operating system name, version, build:

Linux 4.15.0-70-generic x86_64
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

BIF release version or commit (git rev-parse --short HEAD):

v0.1.0

BIF Plugins/Connectors Used

Fabric, Quorum

Additional context

This is the continuation of the earlier issue related to CI: #36

[petermetz]

New clue from the quorum API logs shows that the quorum API server cannot be reached because it crashes due to the transaction registering the actors failing:

qserver1_1  | 2019-11-22 07:54:53 info register:        Registering actors... [{"name":"VALIDATOR1","type":"VALIDATOR","ethAddress":"0xed9d02e382b34818e88B88a309c7fe71E65f419d","constKey":"BULeR8JyUWhiuuCMU/HLA0Q5pzkYT+cHII3ZKBey3Bo=","host":"localhost","port":5050},{"name":"VALIDATOR2","type":"VALIDATOR","ethAddress":"0xca843569e3427144cead5e4d5999a3d0ccf92b8e","constKey":"QfeDAys9MPDs2XHExtc84jKGHxZg/aj52DTh0vtA3Xc=","host":"localhost","port":5051},{"name":"VALIDATOR3","type":"VALIDATOR","ethAddress":"0x0fbdc686b912d7722dc86510934589e0aaf3b55a","constKey":"1iTZde/ndBHvzhcl7V68x44Vx7pl8nwx9LqnM/AfJUg=","host":"localhost","port":5052},{"name":"VALIDATOR4","type":"VALIDATOR","ethAddress":"0x9186eb3d20cbd1f5f992a950d808c4495153abd5","constKey":"oNspPPgszVUFw0qmGFfWwh1uxVUXgvBxleXORHj07g8=","host":"localhost","port":5053}]
qserver1_1  | 2019-11-22 07:54:53 debug actorWrapper:   registerActor actorName: VALIDATOR1 1 BULeR8JyUWhiuuCMU/HLA0Q5pzkYT+cHII3ZKBey3Bo= 0xed9d02e382b34818e88B88a309c7fe71E65f419d localhost 5050
qserver1_1  | 2019-11-22 07:54:53 debug actorWrapper:   registerActor actorName: VALIDATOR2 1 QfeDAys9MPDs2XHExtc84jKGHxZg/aj52DTh0vtA3Xc= 0xca843569e3427144cead5e4d5999a3d0ccf92b8e localhost 5051
qserver1_1  | 2019-11-22 07:54:53 debug actorWrapper:   registerActor actorName: VALIDATOR3 1 1iTZde/ndBHvzhcl7V68x44Vx7pl8nwx9LqnM/AfJUg= 0x0fbdc686b912d7722dc86510934589e0aaf3b55a localhost 5052
qserver1_1  | 2019-11-22 07:54:53 debug actorWrapper:   registerActor actorName: VALIDATOR4 1 oNspPPgszVUFw0qmGFfWwh1uxVUXgvBxleXORHj07g8= 0x9186eb3d20cbd1f5f992a950d808c4495153abd5 localhost 5053
qserver1_1  | 2019-11-22 07:54:54 error register:       Error: Transaction has been reverted by the EVM:
qserver1_1  | {
qserver1_1  |   "blockHash": "0xbf54d98bb54aa2996b87a107b685afb6bf51756814a6c347172bcf90e062f13f",
qserver1_1  |   "blockNumber": 2,
qserver1_1  |   "contractAddress": null,
qserver1_1  |   "cumulativeGasUsed": 12000000,
qserver1_1  |   "from": "0xed9d02e382b34818e88b88a309c7fe71e65f419d",
qserver1_1  |   "gasUsed": 3000000,
qserver1_1  |   "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
qserver1_1  |   "status": false,
qserver1_1  |   "to": "0x1932c48b2bf8102ba33b4a6b545c32236e342f34",
qserver1_1  |   "transactionHash": "0xfdee6e1137d0eac64ffecf62fb9ff1217d922f93e944a1b74a9709e0ad34eca6",
qserver1_1  |   "transactionIndex": 3,
qserver1_1  |   "events": {}
qserver1_1  | }
qserver1_1  |     at /src/node_modules/web3-core-method/src/index.js:412:37
qserver1_1  |     at <anonymous>
qserver1_1  |     at process._tickCallback (internal/process/next_tick.js:189:7)
qserver1_1  | npm ERR! code ELIFECYCLE

[petermetz]

The actor registration issue can be fixed by upgrading to quorum 2.2.5 which fixes the underlying gas estimate issue: Consensys/quorum#585

But then with the newer quorum comes a newer (> 1.8.2) geth version which breaks contract deployment due to bugs in geth's websocket implementation. This is fixed in geth 1.9.1 and above but quorum doesn't have that version in any of their releases yet.

Requested geth upgrade of quorum here:
Consensys/quorum#890

[petermetz]

Found workaround by using the web3 HTTP provider just for contract deployment which doesn't have the payload size limitations the web socket provider does.

[petermetz]

CI script works pretty well on multiple machines that have at least 16 GB RAM and 6vCPU.
Still have issues with Travis where it's just 8GB and 2vCPU. Traced one of the issues back to chaincode instantiation taking too long and gRPC hanging up the call: The chaincode instantiation can take up to 10 minutes to finish on the smaller hardware and the socket gets hung up after 4 minutes.
I reduced log levels of fabric and ensured in the ci script that quorum is shut down at the build phase of fabric so that it doesn't use the CPU unnecessarily at that point.
Without qourum and the very verbose (thousands of lines per second from what I could tell) logging the chaincode instantiation can finish in 2-3 minutes on the lower spec machines as well.

[petermetz]

Fixed by #70