Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to benchmarks in JSON format #511

Closed
2 tasks
chfast opened this issue Oct 3, 2022 · 2 comments · Fixed by #513
Closed
2 tasks

Migrate to benchmarks in JSON format #511

chfast opened this issue Oct 3, 2022 · 2 comments · Fixed by #513
Labels
good first issue Good for newcomers

Comments

@chfast
Copy link
Member

chfast commented Oct 3, 2022

The existing benchmark suite has been converted to JSON State Test format and is available in https://github.com/ipsilon/evm-benchmarks.

@chfast chfast added the good first issue Good for newcomers label Oct 3, 2022
@miles170
Copy link
Contributor

miles170 commented Oct 5, 2022

Could you mind helping clarify where the name and expected_output are in the evm-benchmarks JSON State Test format?

@chfast
Copy link
Member Author

chfast commented Oct 5, 2022

The names are in the labels section and they should match the elements of transaction.data. https://github.com/ipsilon/evm-benchmarks/blob/main/benchmarks/main/sha1_shifts.json#L8-L14
Our loader does not parse labels yet.

The output is simply not there. We proposed to include it but turned out this is substantial change in many tools. ethereum/tests#998

The good news is we have a tool evmone-statetest (not main branch yet) which can execute and verify these files. Later we can reuse its implementation to verify benchmark files before execution.

For now you should check if a benchmark returns EVMC_SUCCESS. You can also check the status and gas used (if the same) in every benchmark loop iteration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants