tests

Common tests for all clients to test against. See the documentation http://ethereum-tests.readthedocs.io/

Contents of this repository

Do not change test files in folders:

• StateTests
• BlockchainTests
• TransactionTests
• VMTests

It is being created by the testFillers which could be found at https://github.com/ethereum/cpp-ethereum/tree/develop/test/tools/jsontests at src folder. The filler specification and wiki are in development so please ask on gitter channel for more details.

If you want to modify a test filler or add a new test please contact @winsvega at https://gitter.im/ethereum/cpp-ethereum

All files should be of the form:

{
	"test1name":
	{
		"test1property1": ...,
		"test1property2": ...,
		...
	},
	"test2name":
	{
		"test2property1": ...,
		"test2property2": ...,
		...
	}
}

Arrays are allowed, but don't use them for sets of properties - only use them for data that is clearly a continuous contiguous sequence of values.

Test Set Sanitation

Checkers

Several basic checks against the test-set are performed to enusre that they have been filled and are formatted correctly. Currently, there are three types of checks that we can perform:

• make TEST_PREFIX.format: check that the JSON is formatted correctly.
• make TEST_PREFIX.valid: check that the JSON files are valid against the JSON schemas in ./JSONSchema.
• make TEST_PREFIX.filled: check that the JSON tests are filled with the correct source hashes against the fillers.

The constant TEST_PREFIX is a path prefix to the test-set you're interested in performing the checks on. For instance:

• make ./src/VMTestsFiller/vmArithmeticTest.format will check that all JSON files in ./src/VMTestsFiller/vmArithmeticTest are formatted correctly.
• make ./src.valid will check that all the JSON files in ./src are valid against the JSON schemas in ./JSONSchema.
• make ./BlockchainTests.filled will check that the source hashes in the JSON tests in ./BlockchainTests are the same as the hashes of the fillers in ./src/BlockchainTestsFiller.

These checks are all performed by the file ./test.py, which can be invoked on individual files as well. Run ./test.py with no arguments for help.

Sanitizers

The above checkers are packaged together into sanitizers for each test-suite, marking which testsuites are passing which testers. See the TODOs in the Makefile to see which checkers are enabled for which test-suites.

• make sani: will run all passing sanitizers on all passing testsuites.

• make sani-TESTNAME: will run just the passing sanitizers for the given testsuite. TESTNAME can be one of:

  • vm: VMTests and VMTestsFiller
  • gs: GeneralStateTests and GeneralStateTestsFiller
  • bc: BlockchainTests and BlockchainTestsFiller
  • tx: TransactionTests and TransactionTestsFiller

Runners/Fillers

The tests can also be run/filled with the same Makefile.

• make run: will use testeth to all the test-suites.
• make fill: will use testeth to fill all the test-suites.
• make TEST_PREFIX.run: runs a single testsuite.
• make TEST_PREFIX.fill: fills and formats a single testsuite.

FAQ

I want to test my client, which version shall I use?

The develop branch in ethereum/tests is the version to use.

How can I add a new test case?

testeth guide to generate test cases