author_submitAndWatchExtrinsic reports incorrect block hash when there are missed authorship slots

[kylezs]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

What we've seen on the Chainflip network:

1. Submit an extrinsic with author_submitAndWatchExtrinsic and eventually get InBlock(A)
2. The finalised stream says the extrinsic was in the block before block A... however the block hash A still exists.

I'm assuming it has to do with the fact there were missed authorship slots at that time. In block A (on the finalised chain) we see that there are two missing authorship events, meaning that the authors of the two previous blocks missed their slots.

This behaviour from the subscription, would still be incorrect though, since even in the case of a reorg, the fact the extrinsic was in a different block would change the hash of block A, but the block with hash A still exists, without the extrinsic we were looking for.

We are using Aura for block authorship.

Steps to reproduce

No response

[bkchr]

So, what you are saying it reports InBlock(A), but the extrinsic is not in block A?

CC @michalkucharczyk

[michalkucharczyk]

Is it fork-aware pool or the old one? (Did you pass --pool-type=fork-aware?)

[michalkucharczyk]

Would you also provide a version of the node used?
What is network configuration: do you send transaction directly to collator, or does it go via dedicated RPC node?

[kylezs]

Thanks for taking a look into this.

So, what you are saying it reports InBlock(A), but the extrinsic is not in block A?

Yes, that's correct.

Would you also provide a version of the node used?

We using polkadot-sdk version 1.15 - in particular this fork: https://github.com/chainflip-io/polkadot-sdk/tree/chainflip-substrate-1.15.2%2B2
There are only very minor changes to this fork, I think the only change is making the event types public, can't recall from memory exactly, but certainly nothing to do with these subscriptions, should be the same as 1.15 in that respect.

do you send transaction directly to collator?

In this case, they are sent directly to one of the authoring nodes via localhost. There's effectively a side-car component reading from the validator/author that is listening and submitting extrinsics to it, though ofc, this node isn't always the author.

Is it fork-aware pool or the old one?

It's not fork-aware. I wasn't aware of fork-aware 😄

[michalkucharczyk]

What is the reproducibility rate?

[kylezs]

I've only seen it once unfortunately. Since the network has had less congestion since, there have been fewer missed authorship slots and so there is already only a small chance that we would even get into a situation it was possible for this to occur.