UnknownUncleParent while syncing

[phahulin]

Hello

I'm running:

• Parity version: Parity/v1.7.0-beta-5f2cabd-20170727/x86_64-linux-gnu/rustc1.18.0
• Operating system: Ubuntu 16.04.1 LTS
• And installed: via binaries

We have a private PoA network with validation via a contract. Yesterday we had a network issue and some nodes ran out of sync for a period of time. Later that day they all synced and network seemed to be working fine. However, today clients can't sync with it, the following error appears in logs:

Stage 3 block verification failed for #880003 (72cd…3db3)
Error: Block(UnknownUncleParent(6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69))

More logs here

Could someone help figure out what might be the origin of this error? Using web3 I can find block 880003, its uncles, their parent blocks... so why are they unknown? what does it mean?

1. Block 880003:

> web3.eth.getBlock(880003)
{ author: '0xdd0bb0e2a1594240fed0c2f2c17c1e9ab4f87126',
  difficulty: { [String: '3.40282366920938463463374607431768211453e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211453 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0x72cd3a741f6d69f98fc14d32bb45d5ff022f0b35feea4be27b04a2c2c8a63db3',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0xdd0bb0e2a1594240fed0c2f2c17c1e9ab4f87126',
  number: 880003,
  parentHash: '0xe6628345aa8b67a828586946ef2ca69073de7a9cd2bf8e444440693576c84c94',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bda0',
     '0xb84110607e64493c5fcc9c6d9e04d43e8b04ebc8c40187c4f3c79c0d1093b118adbe673c845c8ba09f121af6b0b02b38a9075c17ff39309294f281d5980cdb0a6f5300' ],
  sha3Uncles: '0x746f7881ee006ba9a1cb67cabd04d24462a4c079c6d63e3a41ec8bd76307f36f',
  signature: '10607e64493c5fcc9c6d9e04d43e8b04ebc8c40187c4f3c79c0d1093b118adbe673c845c8ba09f121af6b0b02b38a9075c17ff39309294f281d5980cdb0a6f5300',
  size: 1729,
  stateRoot: '0x6a7cbe1ecd84fe8bf3beea54e74785009cc50bda4d46e01bd2e8543a04ad2875',
  step: '301579680',
  timestamp: 1507898399,
  totalDifficulty:
   { [String: '2.99449503737526610663160044663778321083585757e+44']
     s: 1,
     e: 44,
     c: [ 299, 44950373752661, 6631600446637, 78321083585757 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles:
   [ '0xa1486a295a189e9ed6616941c3109aa39e7f5f4e7fddb4789bf01bac0022bbf9',
     '0x654d93e4a7064ca92086788bb62890992acd7098b3d3e7d60124a9dac7cc0fbd' ] }

2. First uncle:

> web3.eth.getBlock('0xa1486a295a189e9ed6616941c3109aa39e7f5f4e7fddb4789bf01bac0022bbf9')
{ author: '0x0e59f9957722ecae11cb65ba46bcac53f6761fe1',
  difficulty: { [String: '3.40282366920938463463374607431768211453e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211453 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0xa1486a295a189e9ed6616941c3109aa39e7f5f4e7fddb4789bf01bac0022bbf9',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x0e59f9957722ecae11cb65ba46bcac53f6761fe1',
  number: 880000,
  parentHash: '0x6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd9c',
     '0xb8416094980d922b8337dbb4fd947e081750ed09f849189d5af100aeb9510f80538777e483ed74984a7138456e2551354de711265f04d5f3e342e153d52842a36e3400' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: '6094980d922b8337dbb4fd947e081750ed09f849189d5af100aeb9510f80538777e483ed74984a7138456e2551354de711265f04d5f3e342e153d52842a36e3400',
  size: 579,
  stateRoot: '0x65af5c00b8be251a46fc94ff30f31110d5c97a8d735c018f5364283fd96ad739',
  step: '301579676',
  timestamp: 1507898380,
  totalDifficulty:
   { [String: '2.99448482890425847847769654539956025778951396e+44']
     s: 1,
     e: 44,
     c: [ 299, 44848289042584, 78477696545399, 56025778951396 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }

3. First uncle's parent:

> web3.eth.getBlock('0x6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69')
{ author: '0x0b55fdba9e288c23cf3eb04c6f237d3bd0f751d6',
  difficulty: { [String: '3.40282366920938463463374607431768211454e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211454 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0x6a1cd18267a8a60f7c86dac2b19ded1e2ad9284efefeace0ae8a376d6f961d69',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x0b55fdba9e288c23cf3eb04c6f237d3bd0f751d6',
  number: 879999,
  parentHash: '0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd9a',
     '0xb84124622d83b323bfb43503c4fcb8228c44e51b8d31da26dff11fe4f292bcc0a02e005233a3592deb3a7d8b6d40c2b97d2d3228a29dfd87768c27e6633a7078f75401' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: '24622d83b323bfb43503c4fcb8228c44e51b8d31da26dff11fe4f292bcc0a02e005233a3592deb3a7d8b6d40c2b97d2d3228a29dfd87768c27e6633a7078f75401',
  size: 579,
  stateRoot: '0x060367fdb4fbb80ea5c794d177983bcea9a7cdae24c47d5717c11b09097a0b22',
  step: '301579674',
  timestamp: 1507898370,
  totalDifficulty:
   { [String: '2.99448142608058926909306191165348594010739943e+44']
     s: 1,
     e: 44,
     c: [ 299, 44814260805892, 69093061911653, 48594010739943 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }

4. Second uncle:

> web3.eth.getBlock('0x654d93e4a7064ca92086788bb62890992acd7098b3d3e7d60124a9dac7cc0fbd')
{ author: '0x1f6bde6d907e25d907820b6180589537d077a6f9',
  difficulty: { [String: '3.40282366920938463463374607431768211453e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211453 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0x654d93e4a7064ca92086788bb62890992acd7098b3d3e7d60124a9dac7cc0fbd',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x1f6bde6d907e25d907820b6180589537d077a6f9',
  number: 879999,
  parentHash: '0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd9b',
     '0xb841722581c66b98e6cf39f66207a6bffcd95918014fae9dd3234797475f066ee7be5382fe0a73fe28ddbefc04e55c09d0d91fbe4ad5571e66fbd4399efa32a886fa00' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: '722581c66b98e6cf39f66207a6bffcd95918014fae9dd3234797475f066ee7be5382fe0a73fe28ddbefc04e55c09d0d91fbe4ad5571e66fbd4399efa32a886fa00',
  size: 579,
  stateRoot: '0x89c9c790d1858fa18e61bda8031d747e1174a255d92ec8ed2a5eaca9e2630dda',
  step: '301579675',
  timestamp: 1507898374,
  totalDifficulty:
   { [String: '2.99448142608058926909306191165348594010739942e+44']
     s: 1,
     e: 44,
     c: [ 299, 44814260805892, 69093061911653, 48594010739942 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }

5. Second uncle's parent:

> web3.eth.getBlock('0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0')
{ author: '0x58bad7cf29428c3d17cb859f5c580d3c25f134cd',
  difficulty: { [String: '3.40282366920938463463374607431768211454e+38'] s: 1, e: 38, c: [ 34028236692, 9384634633746, 7431768211454 ] },
  extraData: '0xd5830107008650617269747986312e31382e30826c69',
  gasLimit: 4700000,
  gasUsed: 0,
  hash: '0xb35497ba7ea789ed25764d8ea505bcb0fd064050fc193fc630f53f27423e6dc0',
  logsBloom: '0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
  miner: '0x58bad7cf29428c3d17cb859f5c580d3c25f134cd',
  number: 879998,
  parentHash: '0x651fc3b8ab4653c01c44165ebd71f36fdb2860352f16c855cbd864bf0a0a4180',
  receiptsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  sealFields:
   [ '0x8411f9bd99',
     '0xb841dece66ba152187b0446830e8a900e31fe12c502ec6d560cb3d5c7abacc3bfdb94a326aee69a0c18cd1d82566e86c5b5743ee624ef79bdcf27043620e1f13f73d00' ],
  sha3Uncles: '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347',
  signature: 'dece66ba152187b0446830e8a900e31fe12c502ec6d560cb3d5c7abacc3bfdb94a326aee69a0c18cd1d82566e86c5b5743ee624ef79bdcf27043620e1f13f73d00',
  size: 579,
  stateRoot: '0x37ad4474d2575dfbc3ab7b3e90e6a0198537da25fa2a7fd73019ca2f5e48c603',
  step: '301579673',
  timestamp: 1507898365,
  totalDifficulty:
   { [String: '2.99447802325692005970842727790741162242528489e+44']
     s: 1,
     e: 44,
     c: [ 299, 44780232569200, 59708427277907, 41162242528489 ] },
  transactions: [],
  transactionsRoot: '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421',
  uncles: [] }

[igorbarinov]

The consensus is broken. Picture of our disaster 😄 http://testnet.oracles.org:3000/
@5chdn Afri. Is there anything we can do besides recreating network from scratch?

[igorbarinov]

Nodes resynced by a magic way. The consensus is not broken. Thanks to @rstormsf to resolve issues on his node (no disk space).

[5chdn]

Cool!

[phahulin]

Please reopen this issue as the same error reproduces on v1.8.1.
Current block number is 168806, however one can't resync from scratch past 165001 block:

2017-11-03 13:25:17  Verifier #2 WARN client  Stage 3 block verification failed for #165002 (431b…6507)
Error: Block(UnknownUncleParent(10ffb5e23b62916c39880b31c72fa950120a97bd010057da65b82fd4149dd880))

I've attached a log with sync=trace,engine=trace sync_log.txt.gz

Could this be a possible scenario: one of validators starts a second node and sends transactions to it. If current block is his to seal but transaction didn't have time to propagate to the rest of the network, will this cause the network to split? Anyway please review the logs, this is just my guess.
Thanks.

[phahulin]

UPD: Now the error does not reproduce on 165001 block, but rather on 175001 block.

One more UPD on the following morning: Now it's 180001 block:

2017-11-04 11:35:36  Stage 3 block verification failed for #180001 (062f…fd70)
Error: Block(UnknownUncleParent(2faa7731683aefc251998f6d531e78278168fd3fbcb181c665b09f6f33093adb))

[igorbarinov]

@5chdn Afri, could we reopen the issue?
We have the problem on another 1.8 based network

[5chdn]

... 165001 block ... 175001 block

... 180001 block

I see a pattern here. Are you changing your chain config?

[phahulin]

No, each time I just completely delete parity data folder on my local machine and start parity --config node.toml to resync from scratch. I use the same spec.json, node.toml and have the same reserved_peers file.

If you'd like to try yourself, here's my node.toml, my peers.txt, our network's spec.json

This time it's 185002 block.

[igorbarinov]

The same problem happened on Kovan? #6947

[igorbarinov]

have you got this problem on Kovan guys?
@david-sg @General-Beck @retotrinkler @uzyn

[igorbarinov]

We are not using Uncles in AuRa, right?
Can we just comment out it as a temp solution?
https://github.com/paritytech/parity/blob/bc167a211bb5d01d6a7baa60ee7551733ddd1cc5/ethcore/src/verification/verification.rs#L129

[5chdn]

#7006 should address this

[igorbarinov]

@5chdn Afri, any estimates when it'll be merged?

[5chdn]

Should be fixed in 1.7.10 and 1.8.4