-
Notifications
You must be signed in to change notification settings - Fork 150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add optional backends
parameter to benchmarking requests
#1992
Conversation
|
I'm not sure it's necessary, but it seemed safer to not parse for example Probably more ready for review, but I still can't really test this easily. So still draft :3. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great, thanks! Left one nit. Since we don't really have a way to tell the users if they choose an invalid backend, I would make the backend parsing a bit more lenient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, thank you!
Opening as draft as I don't know how to test this: how do I simulate locally a request to benchmark a try build @Kobzol? Do I have to manually add rows to the database like in the 18th century?! :p.
This should 🤞 allow choosing which codegen backends we want to use for benchmarks, from the GH benchmarking requests. Hopefully. Maybe. I don't know how to test that it works, I don't know where is the documentation describing how to run these realistic workflows locally.
(Note that I chose to specifically not document the parameter in the help page, until we can test it out a bit in the real world, after this PR is completed)