You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
criterion.rs plots PDF using undocumented kernel density estimation procedure. Plotting empirical CDF is much easier and there is only one way to do it, so it is always clear how the plot was obtained. Correctly interpreting PDF may be hard when resulting distributions are close to each other. Probably that is why median is plotted for each benchmark, but with CDF it can be obtained automatically, it is in the center of the plot.
PDF is probably plotted because http://www.serpentine.com/criterion/ used it, but consider replacing default plots with CDF + regression plot, they are actually more useful.
The text was updated successfully, but these errors were encountered:
While it's true that decision was partially inherited from Haskell's criterion, it's also easier for people without experience in statistics to read a PDF than a CDF. One of Criterion.rs' explicit goals is to be usable for people without experience in statistics, so I would be reluctant to replace it. I would be open to adding a CDF (or perhaps even Normal-Quantile) plot in addition to the PDF plot, however.
Criterion.rs' plotting features are... well, not deprecated (yet), but they're being moved to cargo-criterion so to save myself the trouble of implementing this twice I'll probably just add that feature to cargo-criterion instead.
One of the things I am excited about with cargo-criterion is that it (or other tools that read the same data) can provide more and different analysis of the run. cc #324.
criterion.rs plots PDF using undocumented kernel density estimation procedure. Plotting empirical CDF is much easier and there is only one way to do it, so it is always clear how the plot was obtained. Correctly interpreting PDF may be hard when resulting distributions are close to each other. Probably that is why median is plotted for each benchmark, but with CDF it can be obtained automatically, it is in the center of the plot.
PDF is probably plotted because http://www.serpentine.com/criterion/ used it, but consider replacing default plots with CDF + regression plot, they are actually more useful.
The text was updated successfully, but these errors were encountered: