[BUG] - Conversation with cardano-tx-generator kills the node due to apparent ChainDB consistency check failure

[deepfire]

Internal

Summary

1.16 fails as follows, after a somewhat short (200-400 slots) conversation with cardano-tx-generator:

cardano-node: ExceptionInLinkedThread "ThreadId 37" (ChainDbParseFailure (BlockRef {blockRefPoint = At (Block {blockPointSlot = SlotNo {unSlotNo = 0}, blockPointHash = "?\138\GS\208*#\211g\187\248\175\&5\155\157])q0\147\131k\240$\203\190\DC4\133$\249\134\NAK\235"}), blockRefIsEBB = IsNotEBB}) (DeserialiseFailure 0 "expected list len"))

The setup is a three-node cluster, talking over loopback, starting with fresh genesis and empty database.

The nodes are fed transactions, at approx 10 TPS by the cardano-tx-generator, up to a total of ~1000.

Node genesis and configs attached.

Steps to reproduce

0. Have Nix installed (https://nixos.org/download.html)
   1.get tip of cardano-tx-generator branch cad-1340-txgen-update
1. nix-shell
2. sed -ni '1,/--- 8< ---/ p' "$(git rev-parse --show-toplevel)"/cabal.project
3. cd benchmarks/shelley3pools
4. ./start.sh
5. approximately around slots 200-400 the nodes start to fail

System info (please complete the following information):

• OS: Linux
• Version NixOS 20.04
• Node 1.16, at tag commit (ab846e5e884ac1f3d044a0d15b9e69efc8256036)

Artifacts

• configuration-node-1.json.txt
• genesis.json.txt

[deepfire]

cc @intricate

[edsko]

I suspect this is the bug that was fixed in IntersectMBO/ouroboros-network#2435.

[mrBliss]

I suspect this is the bug that was fixed in input-output-hk/ouroboros-network#2435.

Indeed, I'm pretty sure it is.

[deepfire]

Looks like it's indeed fixed -- as tested with node commit a0eb63646c35e33192aa99a73672757d7f25f246.