Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmarks should expose their options #31083

Closed
BridgeAR opened this issue Dec 24, 2019 · 0 comments
Closed

Benchmarks should expose their options #31083

BridgeAR opened this issue Dec 24, 2019 · 0 comments
Labels
benchmark Issues and PRs related to the benchmark subsystem. test Issues and PRs related to the tests.

Comments

@BridgeAR
Copy link
Member

Our benchmark files currently have no way to print the return the possible options to the user. If we'd implement such feature, we would not have to worry about our benchmark tests anymore that they might fail due to changes in any benchmark options. We could just go ahead and read the benchmarks specific options first and then run the test with the specific subset of options that are suitable.

I opened this issue as a kind of feature request in case someone else would like to open a PR to implement this and also to get feedback from others about it.

// cc @Trott I guess you might be interested in something like this.
@BridgeAR BridgeAR added test Issues and PRs related to the tests. benchmark Issues and PRs related to the benchmark subsystem. labels Dec 24, 2019
@BridgeAR BridgeAR mentioned this issue Jan 17, 2020
Closed
4 tasks
codebytere pushed a commit that referenced this issue Feb 17, 2020
@BridgeAR @codebytere
benchmark: add test and all options and improve errors
ca343ca 
This adds a new `test` option. Using it automatically uses a single
minimal option matrix to verify the benchmark works as expected.

Using the new `all` option makes sure all test suites are run.

On top of that the benchmarks will from now on report properly
what category might have a typo, if any.

The http duration was also refactored to use a option instead of
relying on a configuration setting.

The fixture folder is ignored as test suite from now on.

PR-URL: #31396
Fixes: #31083
Reviewed-By: Sam Roberts <vieuxtech@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
benchmark Issues and PRs related to the benchmark subsystem. test Issues and PRs related to the tests.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

benchmark: test refactoring BridgeAR/node
1 participant
@BridgeAR