Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: ignore significance when using --runs 1 #8299

Closed
wants to merge 1 commit into from
Closed

benchmark: ignore significance when using --runs 1 #8299

wants to merge 1 commit into from

Conversation

AndreasMadsen
Copy link
Member

@AndreasMadsen AndreasMadsen commented Aug 27, 2016

Checklist
  • commit message follows commit guidelines
Affected core subsystem(s)

benchmark

Description of change

Because the standard deviation can't be calculated when there is only
one observation the R scripts raises an error. However it may still be
useful to run them for non-statistical purposes.

This changes the behaviour such when there is only one observation, the
values that depends on the standard deviation becomes Not Applicable (NA).

Fixes: #8288

/cc @mscdex


examples:

                                                               improvement significant   p.value
 http/check_is_http_token.js n=50000000 key="Content-Encoding"      2.42 %          NA        NA
 http/check_is_http_token.js n=50000000 key="Content-encoding"      1.79 %             0.8908776
 chunks length     rate confidence.interval
      0      4 12834.16            5850.699
      1      4 11944.62                  NA

@nodejs-github-bot nodejs-github-bot added the benchmark Issues and PRs related to the benchmark subsystem. label Aug 27, 2016
significant = '*';
p.value = NA;
significant = 'NA';
# Check if there is enogth data to calulate the calculate the p-value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: enogth

@addaleax
Copy link
Member

Can’t really review R code, so rubber-stamp LGTM from me.

@mscdex
Copy link
Contributor

mscdex commented Aug 27, 2016

I can't really review the code either, but one minor nit is the last line of the commit message is a tad too long (>72).

Because the standard deviation can't be calculated when there is only
one observation the R scripts raises an error. However it may still be
useful to run them for non-statistical purposes.

This changes the behaviour such when there is only one observation, the
values that depends on the standard deviation becomes Not Applicable
(NA).
@AndreasMadsen
Copy link
Member Author

Yeah, I'm not amazing at R either :)

@addaleax
Copy link
Member

Hmm… we could give pinging the rest of @nodejs/math aka @fhinkel a try? 😄

@AndreasMadsen
Copy link
Member Author

ping

@addaleax
Copy link
Member

If the count remains at 1 rubber-stamp approval (mine), I’d say it’s okay to land this.

I know you’re busy these days, so please don’t feel pressured, but if you’re ever up for porting some of the R code to JS — and if you think that’s feasible – that might be kind of nice for attracting more reviewers. :)

@AndreasMadsen
Copy link
Member Author

@addaleax Great, I will land it then.

I know you’re busy these days, so please don’t feel pressured, but if you’re ever up for porting some of the R code to JS — and if you think that’s feasible – that might be kind of nice for attracting more reviewers. :)

I have already written ttest, so it is feasible. The problem is that it's a big dependency and it's actually super hard to write a correct t-test implementation. When I wrote ttest I checked all other js implementations and found at least on error in each, so I have every reason to expect that ttest is buggy too.

@AndreasMadsen
Copy link
Member Author

@addaleax
Copy link
Member

The problem is that it's a big dependency

Yeah, that’s a general problem we have in core. Maybe we should just get a package.json for our top level directory or something…

@AndreasMadsen
Copy link
Member Author

Landed in d3834a1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark Issues and PRs related to the benchmark subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants