Skip to content

Commit

Permalink
docs: Typed exceptions to consuming tests section
Browse files Browse the repository at this point in the history
  • Loading branch information
marioevz committed Jan 23, 2024
1 parent 78b2d31 commit ed5aa65
Show file tree
Hide file tree
Showing 6 changed files with 41 additions and 20 deletions.
4 changes: 2 additions & 2 deletions docs/consuming_tests/blockchain_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,9 +197,9 @@ Optional list of withdrawals included in the block RLP.

### `InvalidFixtureBlock`

#### - `expectException`: `str`
#### - `expectException`: [`TransactionException`](./exceptions.md#transactionexception)` | `[`BlockException`](./exceptions.md#blockexception)

Expected exception message that invalidates the block.
Expected exception that invalidates the block.

#### - `rlp`: [`Bytes`](./common_types.md#bytes)

Expand Down
12 changes: 10 additions & 2 deletions docs/consuming_tests/blockchain_test_hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,17 @@ They can mismatch the hashes of the versioned blobs in the execution payload, fo

Hash of the parent beacon block root.

#### - `valid`: `bool`
#### - `validationError`: [`TransactionException`](./exceptions.md#transactionexception)` | `[`BlockException`](./exceptions.md#blockexception)

To be deprecated: Whether the execution payload is valid or not. Expectation is `VALID` if `true`, `INVALID` if `false`, in the `status` field of [PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1).
Validation error expected when executing the payload.

When the payload is valid, this field is not present, and a `VALID` status is
expected in the `status` field of
[PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1).

If this field is present, the `status` field of
[PayloadStatusV1](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payloadstatusv1)
is expected to be `INVALID`.

#### - `version`: [`Number`](./common_types.md#number)

Expand Down
22 changes: 22 additions & 0 deletions docs/consuming_tests/exceptions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Exceptions

Exception types are represented as a JSON string in the test fixtures.

The exception converted into a string is composed of the exception type name,
followed by a period, followed by the specific exception name.

For example, the exception
[`TransactionException.INSUFFICIENT_ACCOUNT_FUNDS`](#ethereum_test_tools.TransactionException.INSUFFICIENT_ACCOUNT_FUNDS)

Check failure on line 9 in docs/consuming_tests/exceptions.md

View workflow job for this annotation

GitHub Actions / build (ubuntu-latest, 3.10, 0.8.20, main, tox)

Link fragments should be valid

docs/consuming_tests/exceptions.md:9:1 MD051/link-fragments Link fragments should be valid [Context: "[`TransactionException.INSUFFICIENT_ACCOUNT_FUNDS`](#ethereum_test_tools.TransactionException.INSUFFICIENT_ACCOUNT_FUNDS)"] https://github.com/DavidAnson/markdownlint/blob/v0.29.0/doc/md051.md
is represented as `"TransactionException.INSUFFICIENT_ACCOUNT_FUNDS"`.

The JSON string can contain multiple exception types, separated by the `|`
character, denoting that the transaction or block can throw either one of
the exceptions.

## `TransactionException`

::: ethereum_test_tools.TransactionException

## `BlockException`

::: ethereum_test_tools.BlockException
2 changes: 1 addition & 1 deletion docs/consuming_tests/state_test.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ Expected state root value that results of applying the transaction to the pre-st
Hash of the RLP representation of the state logs result of applying the transaction to the pre-state
(TODO: double-check this.)

#### - `expectException`: `str`
#### - `expectException`: [`TransactionException`](./exceptions.md#transactionexception)

Exception that is expected to be thrown by the transaction execution (Field is missing if the transaction is expected to succeed)

Expand Down
1 change: 1 addition & 0 deletions docs/navigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
* [Blockchain Tests](consuming_tests/blockchain_test.md)
* [Blockchain Hive Tests](consuming_tests/blockchain_test_hive.md)
* [Common Types](consuming_tests/common_types.md)
* [Exceptions](consuming_tests/exceptions.md)
* [Getting Help](getting_help/index.md)
* [Developer Doc](dev/index.md)
* [Documentation](dev/docs.md)
Expand Down
20 changes: 5 additions & 15 deletions docs/writing_tests/exception_tests.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,19 @@
# Exception Tests

Exception tests are a special type of test that verify that an invalid transaction or an invalid block are correctly rejected with the expected error.
Exception tests are a special type of test which verify that an invalid transaction or an invalid block are correctly rejected with the expected error.

## Creating an Exception Test

To test for an exception, the test can use either the following types from `ethereum_test_tools` library:

- [`TransactionException`](#transactionexception): To be added to the `error` field of the `Transaction` object, and to the `exception` field of the `Block` object that includes the transaction; this exception type is used when a transaction is expected to fail, and therefore when included in a block, the block is expected to be invalid too. This is different from transactions that can be included in blocks but where an exception during EVM execution is expected (e.g. a revert, or out-of-gas, which can be included in valid blocks).
- [`BlockException`](#blockexception): To be added to the `exception` field of the `Block` object; this exception type is used when a block is expected to be invalid, but the exception is related to a block property, e.g. an invalid value of the block header.
- [`TransactionException`](../consuming_tests/exceptions.md#transactionexception): To be added to the `error` field of the `Transaction` object, and to the `exception` field of the `Block` object that includes the transaction; this exception type is used when a transaction is invalid, and therefore when included in a block, the block is expected to be invalid too. This is different from valid transactions where an exception during EVM execution is expected (e.g. a revert, or out-of-gas), which can be included in valid blocks.
- [`BlockException`](../consuming_tests/exceptions.md#blockexception): To be added to the `exception` field of the `Block` object; this exception type is used when a block is expected to be invalid, but the exception is related to a block property, e.g. an invalid value of the block header.

Exceptions can be combined with the `|` operator, to indicate that a test vector can throw either exception.

Ideally, the tester should aim to use only one exception per test vector, and only use multiple exceptions on the rare instance when it is not possible to know which exception will be thrown because it depends on client implementation.
Although exceptions can be combined with the `|` operator to indicate that a test vector can throw either one of multiple exceptions, ideally the tester should aim to use only one exception per test vector, and only use multiple exceptions on the rare instance when it is not possible to know which exception will be thrown because it depends on client implementation.

## Adding a new exception

If a test requires a new exception, because none of the existing ones is suitable for the test, a new exception can be added to either [`TransactionException`](#transactionexception) or [`BlockException`](#blockexception) classes.
If a test requires a new exception, because none of the existing ones is suitable for the test, a new exception can be added to either [`TransactionException`](../consuming_tests/exceptions.md#transactionexception) or [`BlockException`](../consuming_tests/exceptions.md#blockexception) classes.

The new exception should be added as a new enum value, and the docstring of the attribute should be a string that describes the exception.

Expand All @@ -28,11 +26,3 @@ When an exception is added to a test vector, the test runner must check that the
The test runner must map the exception key to the corresponding error string that is expected to be returned by the client.

Exception mapping are particularly important in blockchain tests because the block can be invalid for multiple reasons, and the client returning a different error can mean that a verification in the client is faulty.

## `TransactionException`

::: ethereum_test_tools.TransactionException

## `BlockException`

::: ethereum_test_tools.BlockException

0 comments on commit ed5aa65

Please sign in to comment.