Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools, benchmark: test util benchmark #16050

Closed
wants to merge 3 commits into from

Conversation

sarahmeyer
Copy link
Contributor

coverage for benchmarking util

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines
Affected core subsystem(s)

doc, benchmark

@nodejs-github-bot nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. test Issues and PRs related to the tests. util Issues and PRs related to the built-in util module. labels Oct 6, 2017
@@ -38,7 +38,7 @@ function main(conf) {
const util = process.binding('util');
const types = require('internal/util/types');

const n = (+conf.millions * 1e6) | 0;
const n = (+conf.n * 1e6) | 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would increase n from 5 * 1e6 to 1e7 * 1e6, which is going to cost more time if the benchmark is run normally. Can you change the default n to 5e6 and use that value as-is(without multiplying) instead?

Copy link
Contributor

@Fishrock123 Fishrock123 Oct 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, I agree that is a better solution. (Please keep the number coercion bits though.)


const runBenchmark = require('../common/benchmark');

runBenchmark('util', ['n=1', 'arguments=0']);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting arguments=0 would turn the arg in the test undefined. I believe we are good with n=1 only.

@Trott Trott added the code-and-learn Issues related to the Code-and-Learn events and PRs submitted during the events. label Oct 6, 2017
@Trott Trott assigned Trott and unassigned Trott Oct 10, 2017
@sarahmeyer
Copy link
Contributor Author

@joyeecheung thank you so much for your thoughtful feedback! ❤️

Fishrock123
Fishrock123 previously approved these changes Oct 11, 2017
Copy link
Contributor

@Fishrock123 Fishrock123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, CI: https://ci.nodejs.org/job/node-test-pull-request/10632/

(which probably won't fully pass regardless but hey)

@Fishrock123
Copy link
Contributor

Fishrock123 commented Oct 11, 2017

Hmm, timed out on SmartOS. [1]

not ok 296 parallel/test-benchmark-util
  ---
  duration_ms: 120.492
  severity: fail
  stack: |-
    timeout
  ...

Edit: looks like we are taking too long still for some reason, the longest other benchmark test is test-benchmark-path:

ok 198 parallel/test-benchmark-path
  ---
  duration_ms: 59.499
  ...

@Fishrock123
Copy link
Contributor

Fishrock123 commented Oct 11, 2017

My guess is that it just spawns too many child processes for slow machines. It still takes 10s on my i7 2.6ghz.

So, this just got a bit more complex, but to get this to reliably pass on all systems there are two options (I think):

  1. Add test-benchmark-util: PASS,FLAKY to [$system==solaris] in parallel.status. (Simple)
  2. Modify benchmark/_cli.jsto be able to take a more specific "category" format, such as util/inspect and then separate this test into multiple files. (More work)

Maybe @nodejs/testing can advise on whether option 1 is reasonable.

Trott
Trott previously requested changes Oct 12, 2017
Copy link
Member

@Trott Trott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on! Before this lands, we need some more options added to that array that is the second argument to runBenchmark(). Basically, anything that has more than one option in any of the util benchmark files will need an option passed so that we aren't running every combination of options for each file. While that would be a thorough test, it will time out in CI for some hosts. The goal of these benchmark tests are to flag benchmark changes that totally break everything, rather than thoroughly test the files, so it's OK if there is code in the benchmarks that end up being skipped because of this.

So, you'll probably want to make the array ['argument=false', 'input=', 'method=Array', 'n=1', 'option=none'] plus a few more options in there.

To get an idea of how the options affect the way the benchmark runs, you can try adding elements to the array one at a time and see how it affects the output of ./node test/parallel/test-benchmark-util.js.

There may be other pitfalls that pop up--we'll see. Be persistent. These benchmark tests aren't difficult to write, but they do require a bit of fussing and patience to get right.

@Trott
Copy link
Member

Trott commented Oct 12, 2017

Maybe @nodejs/testing can advise on whether option 1 is reasonable.

@Fishrock123 I'd be a pretty hard "no" on option 1. Option 2 could work, but the usual way we handle this is to set more items in the settings array so that the benchmark has fewer combinations of settings to run. See my comment above and/or look at existing test-benchmark-* files for examples. I'd encourage @sarahmeyer to try that first. I'm pretty certain that will work and will probably be easier than digging into _cli.js.

(This was a "good second commit" task from Code + Learn. It's totally do-able but likely to require more back-and-forth and experimentation than the first tasks given at the event.)

@joyeecheung
Copy link
Member

@Trott @Fishrock123 If we are talking about running one file at a time, run.js --filter inspect util would do the job already, we only need to modify runBenchmark so it takes --filter arguments, which should be pretty easy.

Also this reminds me of the child_process.spawn in other tests that have caused hanging problems before. I am not terribly familiar with tools/test.py but I think it should be possible for the javascript test script to talk to it's parent so we can coordinate the scheduling of these sub-child processes? I will try to take a look later.

@joyeecheung
Copy link
Member

@Trott @Fishrock123 OK I think this is not related to "too many processes at the same time" because run.js would lauch the benchmark one by one. I think we can still try to run all the combinations on faster machines while only run a sample of them on slower ones.

(We don't have a common.isSmartOS right now....should we implement one?)

@Trott
Copy link
Member

Trott commented Oct 12, 2017

I think we can still try to run all the combinations on faster machines while only run a sample of them on slower ones.

With the right options, this test will run plenty fast. Just given the options I put in my comment above (which is a partial list--still more to add, but I left that as an exercise for the contributor), the run time on my laptop for the test is reduced from 19 seconds to less than 3 seconds. With the right arguments this test will be no problem at all.

(We don't have a common.isSmartOS right now....should we implement one?)

If we wanted to do that (which we don't in this case!), we could use common.isSunOS. SmartOS started life as Solaris, I believe.

@Fishrock123 Fishrock123 dismissed their stale review October 14, 2017 22:49

Prefer @Trott's suggestions

@Trott Trott force-pushed the test_util_benchmark branch from c20929b to 78ebc37 Compare October 28, 2017 01:17
@Trott
Copy link
Member

Trott commented Oct 28, 2017

I went ahead and made the changes involving the arguments/settings. Hope that's OK. This should be good to land, I think. @joyeecheung @Fishrock123 PTAL

CI: https://ci.nodejs.org/job/node-test-pull-request/11032/

@Trott
Copy link
Member

Trott commented Oct 28, 2017

(Oh, run time is now down to less than 1.5 seconds locally for me. 🎉)

@Trott Trott dismissed their stale review October 28, 2017 01:23

settings are now implemented here

Trott pushed a commit to Trott/io.js that referenced this pull request Oct 28, 2017
Create a minimal test for the util benchmark files.

PR-URL: nodejs#16050
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
@Trott
Copy link
Member

Trott commented Oct 28, 2017

Landed in 50fe1a8.

Thanks for the contribution! 🎉

@Trott Trott closed this Oct 28, 2017
gibfahn pushed a commit that referenced this pull request Oct 30, 2017
Create a minimal test for the util benchmark files.

PR-URL: #16050
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
@gibfahn
Copy link
Member

gibfahn commented Oct 30, 2017

Should this be backported to v8.x-staging? If yes please follow the guide and raise a backport PR, if no let me know or add the dont-land-on label.

cc/ @joyeecheung @Trott regarding suitability for backporting.

The commit cherry-picks fine, but we get this test failure:

=== release test-benchmark-util ===                                            
Path: parallel/test-benchmark-util
util/format.js
util/format.js type="" n=1: 2,428.8410299257503

util/inspect-array.js
util/inspect-array.js type="" len=100000 n=1: 54.0944640807349

util/inspect-proxy.js
util/inspect-proxy.js n=1: 456.76616860883644

util/inspect.js
util/inspect.js option="none" method="Array" n=1: 316.97499352578575

util/normalize-encoding.js
util/normalize-encoding.js n=1 input="": 6,362.861251447551

util/type-check.js
/home/gib/node/benchmark/util/type-check.js:40
  const arg = args[conf.type][conf.argument];
                             ^

TypeError: Cannot read property 'false' of undefined
    at main (/home/gib/node/benchmark/util/type-check.js:40:30)
    at Benchmark.process.nextTick (/home/gib/node/benchmark/common.js:34:28)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
    at Function.Module.runMain (module.js:676:11)
    at startup (bootstrap_node.js:187:16)
    at bootstrap_node.js:608:3
assert.js:42
  throw new errors.AssertionError({
  ^

AssertionError [ERR_ASSERTION]: 1 === 0
    at ChildProcess.child.on (/home/gib/node/test/common/benchmark.js:25:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
Command: out/Release/node /home/gib/node/test/parallel/test-benchmark-util.js

@Trott
Copy link
Member

Trott commented Oct 30, 2017

I wouldn't bother backporting this unless someone wanted to do it as an exercise. It's great for code changes going forward, but we don't need it on old release lines. We've gotten by just fine thus far without it. :-D

Qard pushed a commit to ayojs/ayo that referenced this pull request Nov 2, 2017
Create a minimal test for the util benchmark files.

PR-URL: nodejs/node#16050
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Qard pushed a commit to ayojs/ayo that referenced this pull request Nov 2, 2017
Create a minimal test for the util benchmark files.

PR-URL: nodejs/node#16050
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
addaleax pushed a commit to ayojs/ayo that referenced this pull request Dec 7, 2017
Create a minimal test for the util benchmark files.

PR-URL: nodejs/node#16050
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark Issues and PRs related to the benchmark subsystem. code-and-learn Issues related to the Code-and-Learn events and PRs submitted during the events. test Issues and PRs related to the tests. util Issues and PRs related to the built-in util module.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants