Fix failing blockchain tests #338

holgerd77 · 2018-08-10T09:50:03Z

Just ran the blockchain tests after a long time (see also associated issue #337 on a revived BlockchainTests test run strategy).

Currently 66 tests are failing:

1..956
# tests 956
# pass  890
# fail  66

I will list all here so we have a basis to fix without the need (for the initial fixes) to re-run the tests again and again:

RPC_API_Test
GasLimitHigherThan2p63m1
ChainAtoChainB_BlockHash
randomStatetest218BC
randomStatetest224BC
randomStatetest234BC
randomStatetest240BC
randomStatetest255BC
randomStatetest331BC
randomStatetest34BC
randomStatetest35BC
randomStatetest403BC
randomStatetest431BC
randomStatetest432BC
randomStatetest529BC
randomStatetest598BC
randomStatetest65BC
BLOCKHASH_Bounds
OOGStateCopyContainingDeletedContract
blockhashNonConstArg
blockhashTests
suicideCoinbase
ExtraData32
RecallSuicidedContractInOneBlock
log1_correct
timeDiff0
timeDiff12
timeDiff13
timeDiff14
wallet2outOf3txs2
wallet2outOf3txsRevoke
wallet2outOf3txsRevokeAndConfirmAgain
walletReorganizeOwners

Test runs on the listed files can be triggered with node tests/tester.js -b --file='blockhashTests'.

Extra note: I think atm blockchain tests are not updated to take the HF setting into account and explicitly instantiate the VM with it (like on the state tests). Atm this shouldn't be a (the) problem, but this should be updated anyhow.

The text was updated successfully, but these errors were encountered:

holgerd77 · 2018-08-10T11:49:06Z

Blockchain tests are now running here: #341

holgerd77 · 2018-08-10T11:52:12Z

On-top-note: we are running low on CircleCI usage capacity (at 91%). If we want to keep the CircleCI switch (which I would assume regarding the advantages) we would have to upgrade rather sooner than later. Think spending an additional 50-100 $ here per month should be in the budget.

holgerd77 · 2018-08-10T11:58:36Z

Started with sending a mail to the CircleCI support if we qualify for the OSS package.

holgerd77 · 2018-08-13T10:36:29Z

Update on CircleCI usage capacity (cc @axic): had some very nice support exchange with Circle, OSS package is project/repo-wise. Seems that we are already on the OSS package with the VM, but it is not displayed on Circle (bug!), the capacity message seems to be another (sigh 😄) bug in the Circle UI and this should not apply for us and we should reach back if we have problems.

holgerd77 · 2018-08-27T12:36:44Z

First note on this:

Many failures preceed the following error stack output:

Invalid block error: Error: Error: Error: invalid receiptTrie invalid gasUsed invalid block stateRoot
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/dist/runBlock.js:159:19
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/dist/cache.js:106:7
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:473:16
    at replenish (/Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:1006:25)
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:1016:9
    at eachLimit$1 (/Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:3196:24)
    at Object.<anonymous> (/Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:1046:16)
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/dist/cache.js:102:11
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:969:16
    at next (/Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:5225:18)
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/dist/cache.js:99:7
    at next (/Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:5223:28)
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/dist/cache.js:99:7
    at next (/Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/async/dist/async.js:5223:28)
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/dist/cache.js:94:9
    at /Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/merkle-patricia-tree/util.js:51:36
not ok 571 last block hash
  ---
    operator: equal
    expected: |-
      '2bda3f5684b144a955a28a1ffdc2dae25e51603f685aae03d0fcc885184d568f'
    actual: |-
      'f7666415a06fe86f39a614be3e707683426ca0cec5a92ccce6664bebbe2ba55c'
    at: self.blockDb.get (/Users/hdrewes/Documents/DEV/EthereumJS/ethereumjs-vm/node_modules/ethereumjs-blockchain/index.js:331:7)

holgerd77 · 2018-08-29T15:59:05Z

Same level as before (66 failing) also with updated blockchain and block libraries.

danjm · 2018-09-28T15:09:07Z

Below is a script that can tell us the last commit at which as test was passing and the commit at which it started failing. The code is at bottom.

Example: the output for the script when running ruby commit-test-check.rb "tests/tester.js -b --file='RPC_API_Test'":

Once the failing blockchain tests are the top priority, this script can be expanded to iterate over all of the files listed above (#338 (comment)) and output a summary of the commits at which each test started failing. Which should be a good starting point for debugging.

commit-test-check.rb

# To get this script to work
#   - Save this file to the root of the `ethereumjs-vm` directory as
#     `commit-test-check.rb` (or whatever file name you like, but
#     note the name in the example below)
#   - Install ruby: https://www.ruby-lang.org/en/documentation/installation/
#   - Install the JSON gem for Ruby: `gem install json`
#   - Globally install tap-mocha-report: npm install -g tap-mocha-reporter
#   - `git checkout master`
#   - In your terminal, run `ruby commit-test-check.rb` followed by the command
#     you wish to check (which should be contained in double quotes). Example:
#     `ruby commit-test-check.rb "tests/tester.js -b --file='RPC_API_Test'"`
#
# Running the above example should produce the following output:
# ```
# The command `tests/tester.js -b --file='RPC_API_Test'` was passing at commit
# 941329898ad9a1dfce3dc9af0c800486ff71277a
# The stats for that test run were: {"suites"=>0, "tests"=>4, "passes"=>4,
# "pending"=>0, "failures"=>0, "start"=>"2018-09-28T14:46:09.830Z",
# "end"=>"2018-09-28T14:46:15.780Z", "duration"=>5950}
#
# It started failing at 914dfd610a68f6685f1179ee1de3c4bb389741e4
# The stats for that test run were: {"suites"=>0, "tests"=>4, "passes"=>2,
# "pending"=>0, "failures"=>2, "start"=>"2018-09-28T14:45:59.361Z",
# "end"=>"2018-09-28T14:46:05.386Z", "duration"=>6025}
# ```


require 'rubygems'
require 'json'

test_command = ARGV[0]

tests_pass = false
current_commit = `git rev-parse HEAD`
previous_commit = current_commit
test_stats = ''
prev_test_stats = ''

while !tests_pass
  previous_commit = current_commit
  checkout_result = `git checkout HEAD~1`
  # puts checkout_result[/(HEAD\sis\now.+)/]
  checkout_result[/(HEAD\sis\now.+)/]
  current_commit = `git rev-parse HEAD`
  `npm install`
  test_result = `node #{test_command} | tap-mocha-reporter json`
  parsed = JSON.parse(test_result)
  prev_test_stats = test_stats
  test_stats = parsed['stats']
  puts current_commit
  puts test_stats
  tests_pass = test_stats['failures'] === 0
  # puts tests_pass
  puts "-------------------------------------"
end

puts "===================================="
puts ""
puts "The command \`#{test_command}\` was passing at commit #{current_commit}"
puts "The stats for that test run were: #{test_stats}"
puts ""
puts "It started failing at #{previous_commit}"
puts "The stats for that test run were: #{prev_test_stats}"

holgerd77 · 2018-10-11T08:12:01Z

Hi @danjm, sorry I didn't have the time to have a look earlier. That is really great, I also already had some similar script idea but wasn't able to realize this with an appropriate amount of effort. Will be hopefully very helpful to get a grip on the blockchain tests.

danjm · 2018-10-11T16:57:39Z

@holgerd77 I ran the tests across all files you listed above and formatted the results a bit for readability.

The result is at the end of this comment. Importantly, searching for unique commit hashes shows that there are only five commits responsible for the failures:

"failing": "9d81b50505841664e2d4c31a94c1a0232fc2de42",
"failing": "c157a996a006a6fcdd58aeed5584b9cd5170afc1",
"failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc",
"failing": "51a6a0cfd43f4e659d733b6c31281ffc3cd10ff2",
"failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4",

We can then find the test names associated with each of these commits. For instance, 914dfd6 causes the following tests to fail: 'RPC_API_Test', 'randomStatetest224BC', 'randomStatetest234BC', 'randomStatetest529BC', 'ExtraData32', 'log1_correct', 'timeDiff0', 'timeDiff12', 'timeDiff13', 'timeDiff14', 'wallet2outOf3txs2', 'wallet2outOf3txsRevoke', 'wallet2outOf3txsRevokeAndConfirmAgain', 'walletReorganizeOwners',

Here is the full report

{
  "GasLimitHaigherThan2p63m1": {
    "passing": "6ceea165df097da914e71ae3c0c3cd4b1538579f\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8102 },
    "failing": "9d81b50505841664e2d4c31a94c1a0232fc2de42\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8051  }
  },
  "ChainAtoChainB_BlockHash": {
    "passing": "8dc282915eb180b98c94fa7e3b0a48e6c1110a64\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8335 },
    "failing": "c157a996a006a6fcdd58aeed5584b9cd5170afc1\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 7999  }
  },
  "randomStatetest218BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8044 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8610  }
  },
  "randomStatetest240BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8251 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8142  }
  },
  "randomStatetest255BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8079 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8003  }
  },
  "randomStatetest331BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 7972 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8058  }
  },
  "randomStatetest34BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8005 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 7965  }
  },
  "randomStatetest35BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8060 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8049  }
  },
  "randomStatetest403BC": {
    "passing": "8dc282915eb180b98c94fa7e3b0a48e6c1110a64\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8275 },
    "failing": "c157a996a006a6fcdd58aeed5584b9cd5170afc1\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8199  }
  },
  "randomStatetest431BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8102 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8049  }
  },
  "randomStatetest432BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8044 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8053  }
  },
  "randomStatetest598BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8135 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8023  }
  },
  "randomStatetest65BC": {
    "passing": "9812ee5d0e78adbf2b320f6b228ad4c6128fd38c\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8072 },
    "failing": "9bf9f1d5964026992af23cd101ab3b9959c771dc\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 7961  }
  },
  "BLOCKHASH_Bounds": {
    "passing": "8dc282915eb180b98c94fa7e3b0a48e6c1110a64\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8272 },
    "failing": "c157a996a006a6fcdd58aeed5584b9cd5170afc1\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8146  }
  },
  "OOGStateCopyContainingDeletedContract": {
    "passing": "4eae74f7df592eac9d1d5aff731ae8d4ab8061f4\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8212 },
    "failing": "51a6a0cfd43f4e659d733b6c31281ffc3cd10ff2\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8214  }
  },
  "blockhashNonConstArg": {
    "passing": "8dc282915eb180b98c94fa7e3b0a48e6c1110a64\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8204 },
    "failing": "c157a996a006a6fcdd58aeed5584b9cd5170afc1\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8274  }
  },
  "blockhashTests": {
    "passing": "8dc282915eb180b98c94fa7e3b0a48e6c1110a64\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8794 },
    "failing": "c157a996a006a6fcdd58aeed5584b9cd5170afc1\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8309  }
  },
  "suicideCoinbase": {
    "passing": "4eae74f7df592eac9d1d5aff731ae8d4ab8061f4\n",
    "passing_stats": { "tests": 8, "passes": 8, "failures": 0, "duration": 17096 },
    "failing": "51a6a0cfd43f4e659d733b6c31281ffc3cd10ff2\n",
    "failing_stats": { "tests": 8, "passes": 4, "failures": 4, "duration": 17090  }
  },
  "RecallSuicidedContractInOneBlock": {
    "passing": "4eae74f7df592eac9d1d5aff731ae8d4ab8061f4\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8382 },
    "failing": "51a6a0cfd43f4e659d733b6c31281ffc3cd10ff2\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8358  }
  },
  "RPC_API_Test": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 10392 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 9983  }
  },
  "randomStatetest224BC": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8256 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8294  }
  },
  "randomStatetest234BC": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8243 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8276  }
  },
  "randomStatetest529BC": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8365 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8236  }
  },
  "ExtraData32": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8285 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8314  }
  },
  "log1_correct": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8253 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8239  }
  },
  "timeDiff0": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8279 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8488  }
  },
  "timeDiff12": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8240 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8382  }
  },
  "timeDiff13": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8316 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8396  }
  },
  "timeDiff14": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8201 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8505  }
  },
  "wallet2outOf3txs2": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8814 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8719  }
  },
  "wallet2outOf3txsRevoke": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 8, "passes": 8, "failures": 0, "duration": 16256 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 8, "passes": 4, "failures": 4, "duration": 15857  }
  },
  "wallet2outOf3txsRevokeAndConfirmAgain": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 8775 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 8611  }
  },
  "walletReorganizeOwners": {
    "passing": "941329898ad9a1dfce3dc9af0c800486ff71277a\n",
    "passing_stats": { "tests": 4, "passes": 4, "failures": 0, "duration": 26959 },
    "failing": "914dfd610a68f6685f1179ee1de3c4bb389741e4\n",
    "failing_stats": { "tests": 4, "passes": 2, "failures": 2, "duration": 19155  }
  }
}

alextsg · 2018-10-11T17:27:44Z

28 out of the 72 test failures are from this PR:
#295
which seems like an intentional change to comply with other clients but differs from the yellow paper. Should we add these tests to the list of tests to skip until an EIP gets added and ethereum tests get updated?

holgerd77 · 2018-10-15T11:09:33Z

Hmm, yes, this probably make sense. Could you if you take on this make this context very clear in the comments on the skipped tests?

danjm · 2018-10-22T16:35:30Z

Here is a more useful summary, where each failing test has been grouped with all other tests that started failing on the same commit, along with the url for the PR where the respective commit was merged.

{
    "51a6a0cfd43f4e659d733b6c31281ffc3cd10ff2": {
        testsFailingWithCommit: [
            "OOGStateCopyContainingDeletedContract",
            "suicideCoinbase",
            "RecallSuicidedContractInOneBlock"
        ],
        introducedWithPR: "https://github.com/ethereumjs/ethereumjs-vm/pull/147"
    },
    "c157a996a006a6fcdd58aeed5584b9cd5170afc1": {
        testsFailingWithCommit: [
            "ChainAtoChainB_BlockHash",
            "randomStatetest403BC",
            "BLOCKHASH_Bounds",
            "blockhashNonConstArg",
            "blockhashTests"
        ],
        introducedWithPR: "https://github.com/ethereumjs/ethereumjs-vm/pull/168"
    },
    "9bf9f1d5964026992af23cd101ab3b9959c771dc": {
        testsFailingWithCommit: [
            "randomStatetest218BC",
            "randomStatetest240BC",
            "randomStatetest255BC",
            "randomStatetest331BC",
            "randomStatetest34BC",
            "randomStatetest35BC",
            "randomStatetest431BC",
            "randomStatetest432BC",
            "randomStatetest598BC",
            "randomStatetest65BC"
        ],
        introducedWithPR: "https://github.com/ethereumjs/ethereumjs-vm/pull/184"
    },
    "9d81b50505841664e2d4c31a94c1a0232fc2de42": {
        testsFailingWithCommit: [
            "GasLimitHaigherThan2p63m1"
        ],
        introducedWithPR: "https://github.com/ethereumjs/ethereumjs-vm/pull/197"
    ],
    "914dfd610a68f6685f1179ee1de3c4bb389741e4": {
        testsFailingWithCommit: [
            "RPC_API_Test",
            "randomStatetest224BC",
            "randomStatetest234BC",
            "randomStatetest529BC",
            "ExtraData32",
            "log1_correct",
            "timeDiff0",
            "timeDiff12",
            "timeDiff13",
            "timeDiff14",
            "wallet2outOf3txs2",
            "wallet2outOf3txsRevoke",
            "wallet2outOf3txsRevokeAndConfirmAgain",
            "walletReorganizeOwners"
        ],
        introducedWithPR: "https://github.com/ethereumjs/ethereumjs-vm/pull/295"
    }
}

danjm · 2018-10-22T16:39:46Z

I propose that we treat the resolution of these tests as five separate tasks:

Fix/skip tests that started failing with Spurious Dragon Working Branch #147 (assignee: @danjm) (PR: Stop adding account to cache when checking if it is empty. #375)
Fix/skip tests that started failing with Fix some broken opcodes due to JS number issues #168 (assignee: @danjm)
Fix/skip tests that started failing with BLOCKHASH reading past genesis is handled elsewhere, so handle actual errors) #184 (assignee: )
Fix/skip tests that started failing with Fix Error object creation resulting in failing blockchain tests #197 (assignee: )
Fix/skip tests that started failing with Align Bloom Filter Hashing with Mainnet Compatible Clients #295 (assignee: @alextsg )

I've started investigating the first group of these. Unless claimed by others, I will attempt to work through all of them this week.

cc @holgerd77

holgerd77 · 2018-10-23T11:04:12Z

Hi @danjm, thanks for such a deep analysis. Be careful a bit, the latest large amount of failing blockchain tests (see this PR #341, over 300) is largely induced by a new parameter sealEngine over on the tests repo side (see an associated docs PR here ethereum/tests#533). This had the consequence that one can't rely on PoW-specific block header values any more in tests for NoProof tagged tests which need changes to the test systematic over on our side, will submit a PR probably during the day.

Nevertheless the change points you identified above mostly look like things where stuff could have been broken (on a first look and just judging the nature of changes), so probably most of the stuff is worth being investigated.

danjm · 2018-10-29T10:36:12Z

@holgerd77 You were right to caution me about the sealEngine related failures. Even after the PRs on both repos were merged, I ran into trouble when investigating these test failure because I had old commits checked out (where #373 was not applied). Given that ethereumjs-testing is pinned to the latest version via the github url, but the tests were failing on commits where I had previously seen them passing. It took me a while to realizing that I was using the newest test.

Is there any reason we don't release ethereumjs-testing via a versioned npm package?

In other news, I am narrowing in on the cause of the failures of #147. It seems that the code required for the account deletion/cleanup of EIP158 (https://github.com/ethereumjs/ethereumjs-vm/blob/master/lib/runTx.js#L190) is inadvertently causing invalid state roots at the point of flushing the state managers cash (https://github.com/ethereumjs/ethereumjs-vm/blob/master/lib/runBlock.js#L172).

I will continue investigating tomorrow.

holgerd77 · 2018-10-30T20:50:00Z

Hi @danjm, npm list --depth=0 very clearly list the version of the testing library, e.g. ethereumjs-testing@1.2.4, so there shouldn't be a problem with that? Reason for not having this released on npm are size limitations, since the library includes the original tests as a submodule the distribution gets to big an is rejected by npm.

holgerd77 · 2018-10-30T20:57:42Z

Blockchain tests are now run by default, see #374. This should make it easier to proceed with this.

mattdean-digicatapult · 2018-11-06T10:00:51Z

I've submitted a PR (#381 ) that fixes the following tests:

randomStatetest218BC
randomStatetest240BC
randomStatetest255BC
randomStatetest331BC
randomStatetest34BC
randomStatetest35BC
randomStatetest403BC
randomStatetest431BC
randomStatetest432BC
randomStatetest598BC
randomStatetest65BC
BLOCKHASH_Bounds
blockhashNonConstArg
blockhashTests

This is mostly the set that started with #184 set (the ones I was actually looking at) but has also fixed some from the set starting at #168. The failing test from #197 is also fixed by #380

holgerd77 · 2018-11-06T23:16:08Z

Last 20 failing tests (state after merge of #329):

RPC_API_Test
randomStatetest224BC
randomStatetest234BC
randomStatetest529BC
ExtraData32
log1_correct
timeDiff0
timeDiff12
timeDiff13
timeDiff14

mattdean-digicatapult · 2018-11-07T08:48:26Z

I've opened a do-not-merge PR #384 which fixes the remaining blockchain tests but has some issues of it's own. Basically the change in bloom filter implementation from #295 has been hiding missing validation of the block. The following blockchain tests have therefore been passing when they shouldn't have been:

DifferentExtraData1025
ExtraData1024
ExtraData33
wrongDifficulty
wrongGasLimit
wrongTimestamp
wrongTransactionsTrie
wrongUncleHash

Not sure what the best approach is here. My current thought is that it involves leaving the bloom filter as per #295, skip the last couple of test failures caused by this and then write our own tests to ensure the block validation is happening correctly?

My hesitation is caused by the scope of the failing tests. Some of these really look like they should be running.

holgerd77 · 2018-11-28T09:07:29Z

Tada. Done. 😄 Thanks everyone! Special thanks also again to @danjm, we couldn't use everything of your work, but your write-up here was really outstanding! Should we integrate your ruby script into the scripts folder + add some note on the test doc section in README?

holgerd77 mentioned this issue Aug 17, 2018

Implement a realistic BlockchainTests test run strategy #337

Closed

holgerd77 added prio: P2 very important prio: P3 important effort: E3 weeks effort: E2 days and removed prio: P2 very important prio: P3 important labels Sep 20, 2018

holgerd77 mentioned this issue Oct 23, 2018

Run blockchain tests #341

Closed

holgerd77 mentioned this issue Oct 30, 2018

Run blockchain tests as default on CI #374

Merged

mattdean-digicatapult mentioned this issue Nov 5, 2018

Fix failing blockchain test GasLimitHigherThan2p63m1 #380

Merged

holgerd77 closed this as completed Nov 28, 2018

evertonfraga added the package: vm label Feb 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix failing blockchain tests #338

Fix failing blockchain tests #338

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 13, 2018

holgerd77 commented Aug 27, 2018

holgerd77 commented Aug 29, 2018

danjm commented Sep 28, 2018

holgerd77 commented Oct 11, 2018

danjm commented Oct 11, 2018

alextsg commented Oct 11, 2018

holgerd77 commented Oct 15, 2018

danjm commented Oct 22, 2018

danjm commented Oct 22, 2018 •

edited

Loading

holgerd77 commented Oct 23, 2018

danjm commented Oct 29, 2018

holgerd77 commented Oct 30, 2018

holgerd77 commented Oct 30, 2018

mattdean-digicatapult commented Nov 6, 2018

holgerd77 commented Nov 6, 2018

mattdean-digicatapult commented Nov 7, 2018 •

edited

Loading

holgerd77 commented Nov 28, 2018

Fix failing blockchain tests #338

Fix failing blockchain tests #338

Comments

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 10, 2018

holgerd77 commented Aug 13, 2018

holgerd77 commented Aug 27, 2018

holgerd77 commented Aug 29, 2018

danjm commented Sep 28, 2018

holgerd77 commented Oct 11, 2018

danjm commented Oct 11, 2018

alextsg commented Oct 11, 2018

holgerd77 commented Oct 15, 2018

danjm commented Oct 22, 2018

danjm commented Oct 22, 2018 • edited Loading

holgerd77 commented Oct 23, 2018

danjm commented Oct 29, 2018

holgerd77 commented Oct 30, 2018

holgerd77 commented Oct 30, 2018

mattdean-digicatapult commented Nov 6, 2018

holgerd77 commented Nov 6, 2018

mattdean-digicatapult commented Nov 7, 2018 • edited Loading

holgerd77 commented Nov 28, 2018

danjm commented Oct 22, 2018 •

edited

Loading

mattdean-digicatapult commented Nov 7, 2018 •

edited

Loading