Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plot CDF instead of PDF #443

Closed
link2xt opened this issue Sep 19, 2020 · 2 comments
Closed

Plot CDF instead of PDF #443

link2xt opened this issue Sep 19, 2020 · 2 comments

Comments

@link2xt
Copy link

link2xt commented Sep 19, 2020

criterion.rs plots PDF using undocumented kernel density estimation procedure. Plotting empirical CDF is much easier and there is only one way to do it, so it is always clear how the plot was obtained. Correctly interpreting PDF may be hard when resulting distributions are close to each other. Probably that is why median is plotted for each benchmark, but with CDF it can be obtained automatically, it is in the center of the plot.

PDF is probably plotted because http://www.serpentine.com/criterion/ used it, but consider replacing default plots with CDF + regression plot, they are actually more useful.

@bheisler
Copy link
Owner

Hey, thanks for the suggestion!

While it's true that decision was partially inherited from Haskell's criterion, it's also easier for people without experience in statistics to read a PDF than a CDF. One of Criterion.rs' explicit goals is to be usable for people without experience in statistics, so I would be reluctant to replace it. I would be open to adding a CDF (or perhaps even Normal-Quantile) plot in addition to the PDF plot, however.

Criterion.rs' plotting features are... well, not deprecated (yet), but they're being moved to cargo-criterion so to save myself the trouble of implementing this twice I'll probably just add that feature to cargo-criterion instead.

@Eh2406
Copy link
Contributor

Eh2406 commented Jan 14, 2021

One of the things I am excited about with cargo-criterion is that it (or other tools that read the same data) can provide more and different analysis of the run. cc #324.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants