RESET: All changes

docs/consuming_tests/blockchain_test.md

@@ -1,37 +1,37 @@
 # Blockchain Tests
 
-The Blockchain Test fixture format tests are included in subdirectory `blockchain_tests`.
+The Blockchain Test fixture format tests are included in the fixtures subdirectory `blockchain_tests`.
 
 These are produced by the `StateTest` and `BlockchainTest` test specs.
 
 ## Description
 
 The blockchain test fixture format is used to test block validation and the consensus rules of the Ethereum blockchain.
 
-It does so by defining a series of blocks, a pre-execution state, a post-execution state, and verifying that, after all the blocks have been processed, appended if valid or rejected if invalid, the result is the expected post-execution state.
+It does so by defining a pre-execution state, a series of blocks, and a post-execution state, verifying that, after all the blocks have been processed, appended if valid or rejected if invalid, the result is the expected post-execution state.
 
 A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`Fixture`](#fixture) test object, with the key string representing the test name.
 
 The JSON file path plus the test name are used as the unique test identifier.
 
-## Behavior
+## Consumption Procedure
 
 For each [`Fixture`](#fixture) test object in the JSON fixture file, perform the following steps:
 
 1. Use [`network`](#-network) to configure the execution fork schedule according to the [`Fork`](./common_types.md#fork) type definition
-2. Use [`pre`](#-pre-alloc) as the starting state of the execution environment for the test and calculate the state root
-3. Decode [`genesisRLP`](#-genesisrlp-bytes) to obtain the genesis block header
+2. Use [`pre`](#-pre-alloc) as the starting state allocation of the execution environment for the test and calculate the genesis state root
+3. Decode [`genesisRLP`](#-genesisrlp-bytes) to obtain the genesis block header, if the block cannot be decoded, fail the test
 4. Compare the genesis block header with [`genesisBlockHeader`](#-genesisblockheader-fixtureheader), if any field does not match, fail the test
-5. Compare the state root calculated from [`pre`](#-pre-alloc) with the state root in the genesis block header, if they do not match, fail the test
+5. Compare the state root calculated on step 2 with the state root in the genesis block header, if they do not match, fail the test
 6. Set the genesis block as the current head of the chain
 7. If [`blocks`](#-blocks-listfixtureblock--null) is not `null`, and contains at least one block, perform the following steps for each [`FixtureBlock`](#fixtureblock):
     1. Attempt to decode block from field [`rlp`](#-rlp-bytes), and if the block cannot be decoded:
-        1. If [`expectException`](#expectexception-str--null) is `null`, fail the test
+        1. If [`expectException`](#expectexception-str--null) field is missing or is `null`, fail the test
         2. Proceed to the next block
     2. Attempt to apply the decoded block on top of the current head of the chain, and if the block cannot be applied:
-        1. If [`expectException`](#expectexception-str--null) is `null`, fail the test
+        1. If [`expectException`](#expectexception-str--null) field is missing or is `null`, fail the test
         2. Proceed to the next block
-    3. If [`expectException`](#expectexception-str--null) is not `null`, fail the test
+    3. If [`expectException`](#expectexception-str--null) field is not missing and is not `null`, fail the test
     4. Set the decoded block as the current head of the chain
 8. Compare the hash of the current head of the chain against [`lastblockhash`](#-lastblockhash-hash), if they do not match, fail the test
 9. Compare all accounts and the fields described in [`post`](#-post-alloc) against the current state, if any do not match, fail the test
@@ -71,7 +71,7 @@ Root hash of the state trie.
 Root hash of the transactions trie.
 #### - `receiptTrie`: [`Hash`](./common_types.md#hash)
 Root hash of the receipts trie.
-#### - `bloom`: [`Bytes`](./common_types.md#bloom)
+#### - `bloom`: [`Bloom`](./common_types.md#bloom)
 Bloom filter composed of the logs of all the transactions in the block.
 #### - `difficulty`: [`ZeroPaddedHexNumber`](./common_types.md#zeropaddedhexnumber)
 Difficulty of the block.

docs/consuming_tests/blockchain_test_hive.md

@@ -1,19 +1,119 @@
 # Blockchain Hive Tests
 
-The Blockchain Hive Test fixture format tests are included in subdirectory `blockchain_tests_hive`, use Engine API directives instead of the usual BlockchainTest format.
+The Blockchain Hive Test fixture format tests are included in the fixtures subdirectory `blockchain_tests_hive`, and use Engine API directives instead of the usual BlockchainTest format.
 
 These are produced by the `StateTest` and `BlockchainTest` test specs.
 
-## Structure
+## Description
 
-A single JSON fixture file is composed of a JSON object where each key is a different test vector, with the key string representing the test name.
+The Blockchain Hive Test fixture format is used to test block validation and the consensus rules of the Ethereum blockchain, when a block is delivered through the Engine API as a `engine_newPayloadVX` directive.
 
+It does so by defining a pre-execution state, a series of blocks as `engine_newPayloadVX` directives, and a post-execution state, verifying that, after all the blocks have been processed, appended if valid or rejected if invalid, the result is the expected post-execution state.
 
-## How Test Fixtures are Consumed via Hive Simulators
+A single JSON fixture file is composed of a JSON object where each key-value pair is a different [`Fixture`](#fixture) test object, with the key string representing the test name.
 
-There are two Hive simulators that execute the test cases defined in the JSON test fixtures against fully instantiated client instances using the @ethereum/execution-apis:
+The JSON file path plus the test name are used as the unique test identifier.
 
-1. The `ethereum/pyspec` simulator executes the test cases against a fully instantiated execution client that executes blocks via the [`engine_newPayloadVX`](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#engine_newpayloadv1) method. Each block is verified by the client and tested against the `engine_newPayloadVX` response. The post-state section of the test is additionally explicitly verified via `eth_getStorageAt` and `eth_getBalance` alongside an internal nonce checking method. Note, this simulator only works for post-merge forks.
-2. The `ethereum/consensus` simulator executes the test cases by importing the RLP-encoded blocks to the execution client upon start-up on the command-line. The pre-state of `block[0]` and the post-state of `block[n]` are verified using the standard [Execution JSON-RPC API](https://ethereum.github.io/execution-apis/api-documentation), for example with `eth_getBlockByNumber`. This simulator works for all forks.
 
-Both of these methods only support test fixtures in the blockchain test format (but any state test can be expressed as blockchain test consisting of 1 block with 1 transaction).
+## Consumption Procedure
+
+For each [`HiveFixture`](#hivefixture) test object in the JSON fixture file, perform the following steps:
+
+1. Start a full node using:
+    - [`network`](#-network) to configure the execution fork schedule according to the [`Fork`](./common_types.md#fork) type definition
+    - [`pre`](#-pre-alloc) as the starting state allocation of the execution environment for the test and calculate the genesis state root
+    - [`genesisBlockHeader`](#-genesisblockheader-fixtureheader) as the genesis block header
+2. Verify the head of the chain is the genesis block, and the state root matches the one calculated on step 1, otherwise fail the test
+3. For each [`FixtureEngineNewPayload`](#fixtureenginenewpayload) in [`engineNewPayloads`](#-enginenewpayloads-listfixtureenginenewpayload):
+    1. Deliver the payload using the `engine_newPayloadVX` directive, using:
+        - [`version`](#-version-number) as the version of the directive
+        - [`executionPayload`](#-executionpayload-fixtureexecutionpayload) as the payload
+        - [`blob_versioned_hashes`](#-blob_versioned_hashes-listhash--null), if not `null`, as the list of hashes of the versioned blobs that are part of the execution payload
+        - [`parentBeaconBlockRoot`](#-parentbeaconblockroot-hash--null), if not `null`, as the hash of the parent beacon block root
+    2. If [`errorCode`](#-errorcode-number) is not `null`:
+        - Verify the directive returns an error, and the error code matches the one in [`errorCode`](#-errorcode-number), otherwise fail the test
+        - Proceed to the next payload
+    3. If [`valid`](#-valid) is `false`, verify that the directive returns `status` field of [PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1) as `INVALID`, otherwise fail the test
+    4. If [`valid`](#-valid) is `true`, verify that the directive returns `status` field of [PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1) as `VALID`, otherwise fail the test
+
+## Structures
+
+### `HiveFixture`: `Mapping`
+
+#### - `network`: [`Fork`](./common_types.md#fork)
+Fork configuration for the test.
+#### - `genesisBlockHeader`: [`FixtureHeader`](./blockchain_test.md#fixtureheader)
+Genesis block header.
+#### - `engineNewPayloads`: `List[`[`FixtureEngineNewPayload`](#fixtureenginenewpayload)`]`
+List of `engine_newPayloadVX` directives to be processed after the genesis block.
+#### - `engineFcuVersion`: [`Number`](./common_types.md#number)
+Version of the `engine_forkchoiceUpdatedVX` directive to use to set the head of the chain.
+#### - `pre`: [`Alloc`](./common_types.md#alloc)
+Starting account allocation for the test. State root calculated from this allocation must match the one in the genesis block.
+#### - `post`: [`Alloc`](./common_types.md#alloc)
+Account allocation for verification after all the blocks have been processed.
+
+### `FixtureEngineNewPayload`: `Mapping`
+
+#### - `executionPayload`: [`FixtureExecutionPayload`](#fixtureexecutionpayload)
+Execution payload.
+#### - `blob_versioned_hashes`: `List[`[`Hash`](./common_types.md#hash)`] | null` `(fork: Cancun)`
+List of hashes of the versioned blobs that are part of the execution payload.
+They can mismatch the hashes of the versioned blobs in the execution payload, for negative-testing reasons.
+#### - `parentBeaconBlockRoot`: [`Hash`](./common_types.md#hash)` | null` `(fork: Cancun)`
+Hash of the parent beacon block root.
+#### - `valid`: `bool` (To be deprecated)
+Whether the execution payload is valid or not. Expectation is `VALID` if `true`, `INVALID` if `false`, in the `status` field of [PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1).
+#### - `version`: [`Number`](./common_types.md#number)
+Version of the `engine_newPayloadVX` directive to use to deliver the payload.
+#### - `errorCode`: [`Number`](./common_types.md#number)` | null`
+Error code to be returned by the `engine_newPayloadVX` directive.
+
+
+### `FixtureExecutionPayload`
+
+#### - `parentHash`: [`Hash`](./common_types.md#hash)
+Hash of the parent block.
+#### - `feeRecipient`: [`Address`](./common_types.md#address)
+Address of the account that will receive the rewards for building the block.
+#### - `stateRoot`: [`Hash`](./common_types.md#hash)
+Root hash of the state trie.
+#### - `receiptsRoot`: [`Hash`](./common_types.md#hash)
+Root hash of the receipts trie.
+#### - `logsBloom`: [`Bloom`](./common_types.md#bloom)
+Bloom filter composed of the logs of all the transactions in the block.
+#### - `blockNumber`: [`HexNumber`](./common_types.md#hexnumber)
+Number of the block.
+#### - `gasLimit`: [`HexNumber`](./common_types.md#hexnumber)
+Total gas limit of the block.
+#### - `gasUsed`: [`HexNumber`](./common_types.md#hexnumber)
+Total gas used by all the transactions in the block.
+#### - `timestamp`: [`HexNumber`](./common_types.md#hexnumber)
+Timestamp of the block.
+#### - `extraData`: [`Bytes`](./common_types.md#bytes)
+Extra data of the block.
+#### - `prevRandao`: [`Hash`](./common_types.md#hash)
+PrevRandao of the block.
+#### - `blockHash`: [`Hash`](./common_types.md#hash)
+Hash of the block.
+#### - `transactions`: `List[`[`Bytes`](./common_types.md#bytes)`]`
+List of transactions in the block, in serialized format.
+#### - `withdrawals`: `List[`[`FixtureWithdrawal`](#fixturewithdrawal)`]`
+List of withdrawals in the block.
+
+#### - `baseFeePerGas`: [`HexNumber`](./common_types.md#hexnumber) `(fork: London)`
+Base fee per gas of the block.
+#### - `blobGasUsed`: [`HexNumber`](./common_types.md#hexnumber) `(fork: Cancun)`
+Total blob gas used by all the transactions in the block.
+#### - `excessBlobGas`: [`HexNumber`](./common_types.md#hexnumber) `(fork: Cancun)`
+Excess blob gas of the block used to calculate the blob fee per gas for this block.
+
+### `FixtureWithdrawal`
+#### - `index`: [`HexNumber`](./common_types.md#hexnumber)
+Index of the withdrawal
+#### - `validatorIndex`: [`HexNumber`](./common_types.md#hexnumber)
+Withdrawing validator index
+#### - `address`: [`Address`](./common_types.md#address)
+Address to withdraw to
+#### - `amount`: [`HexNumber`](./common_types.md#hexnumber)
+Amount of the withdrawal

docs/consuming_tests/common_types.md

@@ -1,34 +1,52 @@
-### Account
-Account description
+## Basic Types
 
-### Address
+### `Address`
 Common 20-byte Ethereum address in hexadecimal format encoded as a JSON string.
 
-### Alloc
-TODO: Alloc description.
-
-### Bloom
+### `Bloom`
 [Bytes](#bytes) of a 256-byte fixed length
 
-### Bytes
+### `Bytes`
 Hexadecimal representation of binary data of any length encoded as a JSON string,
 
-### Hash
+### `Hash`
 [Bytes](#bytes) of a 32-byte fixed length.
 
-### HeaderNonce
+### `HeaderNonce`
 [Bytes](#bytes) of a 8-byte fixed length.
 
-### Number
+### `Number`
 Decimal number encoded as a JSON string.
 
-### HexNumber
+### `HexNumber`
 Hexadecimal number with "0x" prefix encoded as a JSON string.
 
-### ZeroPaddedHexNumber
+### `ZeroPaddedHexNumber`
 Hexadecimal number with "0x" prefix encoded as a JSON string, with a single zero used to pad odd number of digits, and zero represented as "0x00".
 
-### Fork
+
+## Complex Types
+
+### `Account`: `object`
+Account represented as a JSON object with the following fields:
+
+#### - `balance`: [`ZeroPaddedHexNumber`](#zeropaddedhexnumber)
+Balance of the account.
+
+#### - `nonce`: [`ZeroPaddedHexNumber`](#zeropaddedhexnumber)
+Nonce of the account.
+
+#### - `code`: [`Bytes`](#bytes)
+Code of the account.
+
+#### - `storage`: `Mapping[`[`Hash`](#hash)`, [`[`Hash`](#hash)`]`
+Storage of the account.
+
+### `Alloc`: `Mapping[`[`Address`](#address)`, [`[`Account`](#account)`]`
+State allocation represented as a JSON object, where the keys are the addresses of the accounts, and the values are the accounts.
+
+
+## Fork
 Fork type is represented as a JSON string that can be set to one of the following values:
 
 - `"Frontier"`:

docs/consuming_tests/state_test.md

@@ -1,6 +1,6 @@
 # State Tests
 
-The State Test fixture format tests are included in subdirectory `state_tests`.
+The State Test fixture format tests are included in the fixtures subdirectory `state_tests`.
 
 These are produced by the `StateTest` and `StateTestOnly` test specs.
 
@@ -18,11 +18,11 @@ As opposed to other fixture formats, the state test fixture format could contain
 
 However tests generated by the `execution-spec-tests` repository do **not** use this feature, as every single test object contains only a single test vector.
 
-## Behavior
+## Consumption Procedure
 
 For each [`Fixture`](#fixture) test object in the JSON fixture file, perform the following steps:
 
-1. Use [`pre`](#-pre-alloc) as the starting state of the execution environment for the test
+1. Use [`pre`](#-pre-alloc) as the starting state allocation of the execution environment for the test
 2. Use [`env`](#-env-fixtureenvironment) to configure the current execution environment
 3. For each [`Fork`](./common_types.md#fork) key of [`post`](#-post-mappingfork-list-fixtureforkpost) in the test, and for each of the elements of the list of [`FixtureForkPost`](#fixtureforkpost) values:
     1. Configure the execution fork schedule according to the current [`Fork`](./common_types.md#fork) key
@@ -34,8 +34,8 @@ For each [`Fixture`](#fixture) test object in the JSON fixture file, perform the
             - If [`expectException`](#-expectexception-str) is not empty, revert the state to the pre-state
         2. If the transaction could be applied to the current execution context:
             - If [`expectException`](#-expectexception-str) is not empty, fail the test
-    5. Compare the resulting post-state with the expected post-state
-    6. Compare the resulting logs with the expected logs
+    5. Compare the resulting post-state root with the expected post-state root contained in the [`hash`](#-hash-hash) field of the current [`FixtureForkPost`](#fixtureforkpost), and fail the test if they do not match
+    6. Compare the resulting logs hash with the expected logs contained in the [`logs`](#-logs-hash) field of the current [`FixtureForkPost`](#fixtureforkpost), and fail the test if they do not match
 
 
 ## Structures

src/ethereum_test_tools/spec/blockchain/blockchain_test.py

@@ -370,7 +370,7 @@ def make_hive_fixture(
         """
         Create a hive fixture from the blocktest definition.
         """
-        fixture_payloads: List[Optional[FixtureEngineNewPayload]] = []
+        fixture_payloads: List[FixtureEngineNewPayload] = []
 
         pre, _, genesis = self.make_genesis(t8n, fork)
         alloc = to_json(pre)
@@ -395,6 +395,10 @@ def make_hive_fixture(
                     alloc = new_alloc
                     env = apply_new_parent(env, header)
         fcu_version = fork.engine_forkchoice_updated_version(header.number, header.timestamp)
+        assert (
+            fcu_version is not None
+        ), "A hive fixture was requested but no forkchoice update is defined. The framework should"
+        " never try to execute this test case."
 
         self.verify_post_state(t8n, alloc)
         return HiveFixture(

src/ethereum_test_tools/spec/blockchain/types.py

@@ -745,14 +745,13 @@ def from_fixture_header(
         withdrawals: Optional[List[Withdrawal]],
         valid: bool,
         error_code: Optional[EngineAPIError],
-    ) -> Optional["FixtureEngineNewPayload"]:
+    ) -> "FixtureEngineNewPayload":
         """
         Creates a `FixtureEngineNewPayload` from a `FixtureHeader`.
         """
         new_payload_version = fork.engine_new_payload_version(header.number, header.timestamp)
 
-        if new_payload_version is None:
-            return None
+        assert new_payload_version is not None, "Invalid header for engine_newPayload"
 
         new_payload = cls(
             payload=FixtureExecutionPayload.from_fixture_header(
@@ -1124,15 +1123,15 @@ class HiveFixture(FixtureCommon):
             to_json=True,
         ),
     )
-    payloads: Optional[List[Optional[FixtureEngineNewPayload]]] = field(
+    payloads: List[FixtureEngineNewPayload] = field(
         default=None,
         json_encoder=JSONEncoder.Field(
             name="engineNewPayloads",
             to_json=True,
         ),
     )
-    fcu_version: Optional[int] = field(
-        default=None,
+    fcu_version: int = field(
+        default=1,
         json_encoder=JSONEncoder.Field(
             name="engineFcuVersion",
         ),