Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CDF plot to benchmark reports #18

Open
bheisler opened this issue Jan 14, 2021 · 2 comments
Open

Add CDF plot to benchmark reports #18

bheisler opened this issue Jan 14, 2021 · 2 comments

Comments

@bheisler
Copy link
Owner

CDF plots are less commonly understood, but once one learns to read them they're actually much better at conveying information than the PDF plots cargo-criterion generates. We should generate a CDF plot as well as the PDF plot for the more advanced users.

Could maybe generate a normal-quantile plot for the really advanced users as well?

@link2xt
Copy link

link2xt commented Jan 14, 2021

Related issue in criterion.rs: bheisler/criterion.rs#443

@link2xt
Copy link

link2xt commented Apr 25, 2021

See top plot at deltachat/deltachat-core-rust#2382 (comment) for an example of how misleading PDF plots can be. Red curve suggests some values fall below 300ms, but in fact there are no runs at all which completed in under 300ms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants