Private chain error: "Stage 1 block verification failed for...:

[sauliusgrigaitis]

The issue I reported in Parity was close as it could be an issue in Puppeth.

[karalabe]

How large is your chain? Any chance you could export it for us to inspect?

[karalabe]

Alternatively, could you send me the headers for the failing block and the one before it?

eth.getBlock('0xhash')

[sauliusgrigaitis]

@karalabe I'm afraid I already removed those servers with the data as it happened a long time ago. As I wrote in the original ticket one chain stopped at around 100K block, another one stopped at around 300K.

@noobcola maybe you still have the data needed for @karalabe ?

[noobcola]

I think I do - if I haven't deleted my chain, I'll send you guys the block headers. If not, I'll have to create a new network and wait a few days until my chain hits block 200k

[noobcola]

@karalabe @sauliusgrigaitis
Here's the failing block #82502 InvalidProofOfWork(OutOfBounds { min: Some(251185), max: None, found: 251184 }):

{"difficulty":"251185",
"extraData":"0xd783010703846765746887676f312e392e33856c696e7578",
"gasLimit":4712388,
"gasUsed":0,
"hash":"0xaecbcdc6ea4cd29bc9f54c330313b09dd8f918e72b55efbec116fd5a3b3bdc81",
"logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
"miner":"0x5181be40152caaba8e123a55b7762755d4e8e416",
"mixHash":"0xad73359687bb7003ec9a0de4b982b070282bcf177193d949f030ec3a7606f25d",
"nonce":"0x5a6c8dd73015481f",
"number":82502,
"parentHash":"0x5957b18139187c93791ad472e349219ffcfd2c621f9b0fb3d1bbdd4352764abc",
"receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
"size":538,"stateRoot":"0x491cfff69cf06e10b644e063811b3dcdabb61ed92e85f6feabb42e5a683c8545",
"timestamp":1526548514,
"totalDifficulty":"25001892373",
"transactions":[],
"transactionsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","uncles":[]}

Here's the block beforehand #82501:

{"difficulty":"251185",
"extraData":"0xd783010703846765746887676f312e392e33856c696e7578",
"gasLimit":4712388,
"gasUsed":0,
"hash":"0x5957b18139187c93791ad472e349219ffcfd2c621f9b0fb3d1bbdd4352764abc",
"logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
"miner":"0x5181be40152caaba8e123a55b7762755d4e8e416",
"mixHash":"0xf52145512fa65b27fb4e4dcb87030d1b724b921e4f78ba021d687b6f64a09076",
"nonce":"0x2bdadb5d396d74b1",
"number":82501,
"parentHash":"0x7dd5ffafeeafec69671e08908413e4b931c06a3122416dfbcffc30a7716296d9",
"receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
"sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
"size":538,
"stateRoot":"0xf8b47990e7a4d464ddb6360e2233e09b2cb7938e87e67a3e2c756c412413f2fd",
"timestamp":1526548501,
"totalDifficulty":"25001641188",
"transactions":[],
"transactionsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","uncles":[]}

If you need the entire chaindata, let me know. I'll try to zip it up and upload it somewhere.

[holiman]

So, it's a Parity error message, right? And block exported from Geth?

Weird that the error msg says found: 255184 when the printed block says it has difficulty 255185.

Would be interesting to see the genesis spec for both geth and parity.

@folsen, any ideas?

[sauliusgrigaitis]

@holiman it's a Parity error.

I don't have genesis files now (@noobcola maybe you have?), but mine were basic genesis files generated with Puppeth.

[holiman]

Ok. So I'd guess puppeth generates them wrongly, parity changed format a while ago. I had to update hive aswell.

[holiman]

They moved some params around

[noobcola]

geth genesis.json

{
    "config": {
        "chainId": 8675,
        "homesteadBlock": 0,
        "eip150Block": 0,
        "eip155Block": 0,
        "eip158Block": 0,
        "eip160Block": 0
    },
    "coinbase": "0x0000000000000000000000000000000000000000",
    "difficulty": "0x400",
    "extraData": "",
    "gasLimit": "0x2fefd8",
    "nonce": "0x0000000000000711",
    "mixhash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "timestamp": "0x00",
    "alloc": {}
}

parity genesis file

{
    "name": "Private TestNet",
    "engine": {
      "Ethash": {
        "params": {
          "gasLimitBoundDivisor": "0x400",
          "minimumDifficulty": "0x20000",
          "difficultyBoundDivisor": "0x800",
          "durationLimit": "0xd",
          "blockReward": "0x4563918244F40000",
          "registrar": "0x81a4b044831c4f12ba601adb9274516939e9b8a2",
          "homesteadTransition": 0,
          "eip150Transition": 0,
          "eip155Transition": 0,
          "eip160Transition": 0,
          "eip161abcTransition": 0,
          "eip161dTransition": 0
        }
      }
    },
    "params": {
      "accountStartNonce": "0x0",
      "maximumExtraDataSize": "0x20",
      "minGasLimit": "0x1388",
      "networkID": 8675,
      "chainID": 8675,
      "eip98Transition": 9223372036854775807,
      "gasLimitBoundDivisor": "0x0400"
    },
    "genesis": {
      "seal": {
        "ethereum": {
          "nonce": "0x0000000000000711",
          "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
        }
      },
      "difficulty": "0x400",
      "author": "0x0000000000000000000000000000000000000000",
      "timestamp": "0x00",
      "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
      "extraData": "",
      "gasLimit": "0x2fefd8"
    },
    "accounts": {}
  }

[noobcola]

The error originates from Parity. The block data listed for block num 82502 and 82501 are from geth

[holiman]

See ethereum/hive#99 , and openethereum/parity-ethereum#8067 . Also, I think all of these

 "eip155Transition": "0x7fffffffffffffff"

goes under params, not Ethash/params nowadays

[holiman]

Reference genesis: https://github.com/paritytech/parity-ethereum/blob/master/ethcore/res/ethereum/eip150_test.json

[karalabe]

Parity's PoW validation is wrong.

	/// Convert an Ethash boundary to its original difficulty. Basically just `f(x) = 2^256 / x`.
	pub fn boundary_to_difficulty(boundary: &H256) -> U256 {
		let d = U256::from(*boundary);
		if d <= U256::one() {
			U256::max_value()
		} else {
			((U256::one() << 255) / d) << 1
		}
	}

	/// Convert an Ethash difficulty to the target boundary. Basically just `f(x) = 2^256 / x`.
	pub fn difficulty_to_boundary(difficulty: &U256) -> H256 {
		if *difficulty <= U256::one() {
			U256::max_value().into()
		} else {
			(((U256::one() << 255) / *difficulty) << 1).into()
		}
	}

Both these calculations truncate the last bit. This means, that blocks with odd difficulty values get validated by Parity as if the difficulty was one less. This is the reason why the blocks are rejected.

Go repro:

package main

import (
    "fmt"
    "math/big"
)

func main() {
    difficulty := int64(251185)
    fmt.Printf("Source difficulty: %v\n", difficulty)

    // Geth
    two256 := new(big.Int).Exp(big.NewInt(2), big.NewInt(256), big.NewInt(0))
    fmt.Printf("Geth diff->boundary->diff: %v\n", new(big.Int).Div(two256, new(big.Int).Div(two256, big.NewInt(difficulty))))

    // Parity
    boundary := big.NewInt(1)
    boundary.Lsh(boundary, 255)
    boundary.Div(boundary, big.NewInt(difficulty))
    boundary.Lsh(boundary, 1)

    diff := big.NewInt(1)
    diff.Lsh(diff, 255)
    diff.Div(diff, boundary)
    diff.Lsh(diff, 1)

    fmt.Printf("Parity diff->boundary->diff: %v\n", diff)
}

Source difficulty: 251185
Geth diff->boundary->diff: 251185
Parity diff->boundary->diff: 251184

Ping @5chdn @folsen

[holiman]

Cpp: https://github.com/ethereum/cpp-ethereum/blob/7e4250bf194501eb39d34e9a93edf26d78de48f5/libethashseal/Ethash.h#L73
Python: https://github.com/ethereum/pyethereum/blob/0173c540e7837271985f8b9805bfb8480e3c04a1/ethereum/pow/ethpow.py#L77

(both does it like geth, from what I can tell)