Blocks do not finalise in cross-client devnet

[kishansagathiya]

Describe the bug

• If you run a cross-client devnet (2 gossamer nodes, 1 substrate node), blocks never get finalised.

2022-06-16T19:47:13+05:30 DBUG failed to validate vote message &{1 0 stage=precommit hash=0x0000000000000000000000000000000000000000000000000000000000000000 number=0 authorityID=0xd17c2d7823ebf260fd138f2d7e27d114c0145d968b5ff5006125f2414fadae69}: block does not exist	vote_message.go:L64	pkg=grandpa

My feeling is that above error message is the culprit. Most likely we are creating a block header without populating block hash into it.

[kishansagathiya]

• The problem is happening in getPreVotedBlock. There is getPossibleSelectedBlocks which gives us blocks with more than threshold votes,
  getPossibleSelectedBlocks gives 0 blocks. so we go to getGrandpaGHOST which returns a vote without setting any hash.

I am not yet sure, which one is the abnormal part here getGrandpaGHOST or getPossibleSelectedBlocks

[EclesioMeloJunior]

Can you describe how are you running these nodes? Would be nice if you provide the human-readable genesis file

[kishansagathiya]

it does look like we are receiving proper pre vote messages. All prevote messages that we get are good. we are receiving pre votes from substrate

Problem is that we don't send proper pre-commit message.

[kishansagathiya]

@EclesioMeloJunior just use ChainSafe/substrate-node-template#1 to create genesis.json
Use

target/release/gssmr-test-node build-spec --raw > genesis.json

[kishansagathiya]

it does look like we are receiving proper pre vote messages. All prevote messages that we get are good. we are receiving pre votes from substrate

Problem is that we don't send proper pre-commit message.

There are enough prevote counts,

2022-06-17T18:54:54+05:30 DBUG validated vote message hash=0x0fdb1ec614670e21e8a1bf9b4b4f49eeb0e7a827da3fb6ee454b565bd0ec0573 number=1 from 0x439660b36c6c03afafca027b910b4fecf99801834c62a5e6006f27d978de234f, round 1, subround 0, prevote count 3, precommit count 1, votes needed 3	vote_message.go:L69	pkg=grandpa

that means that, problem must be in getPossibleSelectedBlocks. It should get at least block one block, , that is block number 1.

Also, why is voting happening for block number 0, isn't 0th block genesis block and finalised?

2022-06-17T18:54:44+05:30 DBUG validated vote message hash=0xee87cc7ac5e87460094ec4ba29bbf8f28fb7741b69e7edb142233c3dd990b365 number=0 from 0x88dc3417d5058ec4b4503e0c12ea1a0a89be200fe98922423d4334014fa6b0ee, round 1, subround 0, prevote count 2, precommit count 0, votes needed 3	vote_message.go:L69	pkg=grandpa

[EclesioMeloJunior]

I believe you found an investigation path! I think we could transform this issue into an epic as Blocks do not finalize in cross-client devnet can group a couple of related issues, what do you think? @danforbes @timwu20

[danforbes]

@EclesioMeloJunior let's talk more about that idea in standup on Monday 👍🏻

[kishansagathiya]

Seems like blocks are getting finalised. So, after finalising a block we are supposed to write it in the db and chain sync will pick it up and share with rest of the network. For whatever reason chain sync always thinks that blocks are not finalised. So, either we don't write things proper or don't read them or something in between.

After finalising a block we send a commit message to peers. I see another gossamer node getting the commit messages, but not substrate (may be I don't know what log to look for). The gossamer node that gets commit message also doesn't seem to use that info.

[kishansagathiya]

seeing this error sometimes with 2 substrate 1 gossamer

2022-07-19T16:24:20+05:30 EROR block data processing for block with hash 0xf71994272a0ccc2ee46e0e3d90645ed95448c33bd602f29da3431c1bb8cda695 failed: block producer equivocated	chain_processor.go:L86	pkg=sync

[EclesioMeloJunior]

This looks like a block production error, not a finalization issue, see -> #2682 (comment)

[kishansagathiya]

@EclesioMeloJunior we have been tracking them all in this issue

[kishansagathiya]

@danforbes I don't think this issue is relevant anymore. We don't have any finalisation or block building issue as far as I know.

[danforbes]

Closing per the above.