doc: reorganize benchmarking guide/readme, update test/README.md #11237

joyeecheung · 2017-02-08T11:19:44Z

The current benchmark/README.md is in fact a guide on how to run and write benchmarks, this PR moves it to doc/guides (where writing-tests.md is) and create another README.md that explains the directory layout/common API (like what test/README.md does). This should make it easier for new contributors to find out what to do when they need to provide a benchmark.

Also updates the test/README.md to add a link to the testing guide.

Fixes: #11190

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

doc

joyeecheung · 2017-02-08T11:21:31Z

Would like some feedback from @AndreasMadsen on the new benchmark/README.md

AndreasMadsen · 2017-02-08T11:38:58Z

I'm not sure I agree with the hole guide concept as I think it is too hidden, but looking besides that:

please compress the Benchmark Directories section, perhaps use a <dl> list or a table (I would say the same for test/README.md, but that may be out of scope).
Do we need Other Top-level files, the test/README.md doc doesn't have this.

joyeecheung · 2017-02-08T11:53:55Z

@AndreasMadsen Thanks for the feedback. I agree the guides are a little bit hidden at the moment, but as we put more documents there hopefully they can get more traction.

About the directories, if we make the same changes to test/README.md then a table would be better since they have Run on CI info. That can also make it easier to read. cc @Trott considering the history of that file.

The Other Top-level files could give info to people who want to improve the tooling, that probably doesn't apply to test/README.md since the testing tools are in tools, not test.

vsemozhetbyt · 2017-02-08T15:41:57Z

benchmark/README.md

+* `_cli.R`: parses the command line arguments passed to `compare.R`
+* `_http-benchmarkers.js`: selects and runs external tools for benchmarking
+  the `http` subsystem.
+* `common.js`: see [Common API]((#common-api)).


Should it be only one pair of parentheses?

Oh, good catch!

jasnell

Almost there

jasnell · 2017-02-08T16:10:14Z

benchmark/README.md

+writing JavaScript run by the built-in JavaScript engine.
+
+For a detailed guide on how to write and run benchmarks in this
+directory, see [the guide on benchmarks](../doc/guides/writing-and-running-benchmarks.md).

 ## Table of Content


I know this was already there but s/Content/Contents

jasnell · 2017-02-08T16:10:49Z

benchmark/README.md

+
+## Benchmark Directories
+
+### arrays


Having these as a table may be more concise. Separate headers doesn't seem necessary

jasnell · 2017-02-08T16:12:56Z

benchmark/README.md

-```
+Force V8 to optimize the function with the native function
+`%OptimizeFunctionOnNextCall()` and return the optimization status
+after that.


A mention about the (common.v8ForceOptimization()](https://github.com/nodejs/node/blob/master/benchmark/common.js#L233) utility method would be good here.

Uh, well, this is actually the documentation of that method :S

jasnell · 2017-02-08T16:14:23Z

doc/guides/writing-and-running-benchmarks.md

+
+## Prerequisites
+
+Most of the HTTP benchmarks require a benchmarker to be installed, this can be


I would recommend putting these under sub-headings.. e.g. ### HTTP Benchmark Requirements, etc

jasnell · 2017-02-08T16:15:24Z

doc/guides/writing-and-running-benchmarks.md

+
+### Comparing node versions
+
+To compare the effect of a new node version use the `compare.js` tool. This


In general throughout this doc, since edits are being made anyway, please s/node/Node.js/g

jasnell · 2017-02-08T16:20:47Z

test/README.md

+On how to run tests in this direcotry, see
+[the contributing guide](../CONTRIBUTING.md#step-5-test).
+
+## Table of Content


s/Content/Contents

TimothyGu · 2017-02-12T08:46:02Z

benchmark/README.md

+
+It is used to prevent the benchmark from getting disrupted
+by the optimizer kicking in halfway through. However, this could result in a
+less effective optimization, so it should be used with caution.


The actual signature is v8ForceOptimization(method, ...args), where args are passed to method.apply. Also of note is that method is executed with this set to null. You might also want to explain why it can "result in a less effective optimization."

I think the reasons for a less effective optimization are probably too many to explain here(no proper inlining because the profiler doesn't know the functions calls inside are hot, no proper type feedback, possible deopt, etc)..probably better left for another PR to update the guide instead of a readme.

A general guide would be to grep deps/v8/test with OptimizeFunctionOnNextCall to see how it's used though.

Hmmm..also this doesn't actually optimize the monomorphic pattern method(...args) if the method has been called with other context/args during the startup, it could be polymorphic or even megamorphic.

..so documenting the null and ..args part could be misleading, I am inclined to stay a little bit vague here, if we need to be precise probably it would be best understood by reading its source code.

The signature should be corrected though :D

joyeecheung · 2017-02-12T10:27:17Z

@jasnell @TimothyGu @vsemozhetbyt Thanks for the reviews, updated, PTAL.

vsemozhetbyt · 2017-02-12T10:46:14Z

@joyeecheung I am not sure if this is a subject for this PR, but could the parameters of the compare.js be documented? I've recently added a benchmark to the dgram suite and I could not find a way to run just one benchmark file with compare.js in the guide. Currently, the "Comparing Node.js versions" chapter has only an example how to run a whole suite (and the whole dgram suite lasts in my machine ~ 6 hours). So maybe the --filter and other params from this fragment are worth to be documented for benchmark beginners.

AndreasMadsen · 2017-02-12T11:20:35Z

@vsemozhetbyt My opinion is that this is a bit out of scope for this pull request. But you are right that it is missing, at the very least we should document how to print the help text. I would also like to consistently have a --help option. Do you want to make a pull request?

vsemozhetbyt · 2017-02-12T11:26:03Z

@AndreasMadsen I am not so good in English, so I usually abstain from creating a non-js text in docs. Sorry for not to be helpful here.

joyeecheung · 2017-02-12T11:41:11Z

@vsemozhetbyt @AndreasMadsen The current logic in _cli.js is to print the usage when process.argv.length < 3 i.e. when those scripts are run without command line arguments. I'll put this information in the guide for now, another PR describing the arguments in detail should be useful.

AndreasMadsen · 2017-02-12T11:53:27Z

The current logic in _cli.js is to print the usage when process.argv.length < 3 i.e. when those scripts are run without command line arguments

I'm aware, but it is not the most intuitive way of getting the help text, it was mostly intended as an input validation. There is also the R scripts where one of them only prints if --help is provided (because it only takes optional arguments) and the other behaves likes the js scripts.

joyeecheung · 2017-02-12T11:58:58Z

@AndreasMadsen Should we document the current behavior in this PR? I think it's probably better than nothing. If another PR implements --help it can update the guide(or I can rebase, whichever comes the first)

AndreasMadsen · 2017-02-12T12:02:08Z

Should we document the current behavior in this PR? I think it's probably better than nothing. If another PR implements --help it can update the guide(or I can rebase, whichever comes the first)

I like the simple documentation as you have done it now. But as a principal one shouldn't need to look in the documentation in order to learn how to print the help text.

joyeecheung · 2017-02-13T13:21:49Z

@jasnell @TimothyGu @vsemozhetbyt @AndreasMadsen Can I have a few LGTM please?

AndreasMadsen · 2017-02-13T15:01:30Z

Do we prefer HTML tables or Markdown tables? Personally I find HTML tables easier to read than Markdown tables when there is a lot of text (markdown lines exceed 80 char).

joyeecheung · 2017-02-13T15:05:30Z

I tried the markdown tables first but couldn't find a way to use linebreaks in them, some of the lines are a bit long so I just switched to HTML. I think markdown tables are easier to read when your cells tend to be small and just contain a few words, but not when you want to write actual sentences in them?

Since benchmark/README.md is in fact a guide on how to write and run benchmarks, move it to doc/guides. PR-URL: #11237 Fixes: #11190 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Andreas Madsen <amwebdk@gmail.com>

* Write a new benchmark/README.md describing the benchmark directory layout and common API. * Fix the moved benchmarking guide accordingly, add tips about how to get the help text from the benchmarking tools. PR-URL: #11237 Fixes: #11190 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Andreas Madsen <amwebdk@gmail.com>

* Add description about the test directory * Add link to the test writing guide and the contributing guide * Use table to describe the directory layout and CI info PR-URL: #11237 Fixes: #11190 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Andreas Madsen <amwebdk@gmail.com>

joyeecheung · 2017-02-17T15:54:12Z

Landed in f8ee197...736814e, thanks!

Since benchmark/README.md is in fact a guide on how to write and run benchmarks, move it to doc/guides. PR-URL: #11237 Fixes: #11190 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Andreas Madsen <amwebdk@gmail.com>

* Write a new benchmark/README.md describing the benchmark directory layout and common API. * Fix the moved benchmarking guide accordingly, add tips about how to get the help text from the benchmarking tools. PR-URL: #11237 Fixes: #11190 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Andreas Madsen <amwebdk@gmail.com>

* Add description about the test directory * Add link to the test writing guide and the contributing guide * Use table to describe the directory layout and CI info PR-URL: #11237 Fixes: #11190 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Andreas Madsen <amwebdk@gmail.com>

jasnell · 2017-03-07T07:18:23Z

requires a backport PR to land on v4

joyeecheung · 2017-03-07T10:06:12Z

Got some weird merge conflicts..looks like there are ancestor commits not yet backported.

AndreasMadsen · 2017-03-07T11:36:27Z

The benchmark suite runner was rewritten in v7 and because of some issue reports it was decleared a major change. This should not be backported to v4 or v6.

* Add description about the test directory * Add link to the test writing guide and the contributing guide * Use table to describe the directory layout and CI info PR-URL: #11237 Fixes: #11190 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Andreas Madsen <amwebdk@gmail.com>

joyeecheung changed the title ~~doc:~~ doc: move benchmark/README.md to doc/guides, update test/README.md Feb 8, 2017

joyeecheung changed the title ~~doc: move benchmark/README.md to doc/guides, update test/README.md~~ doc: reorganize benchmarking guide/readme, update test/README.md Feb 8, 2017

joyeecheung added doc Issues and PRs related to the documentations. benchmark Issues and PRs related to the benchmark subsystem. test Issues and PRs related to the tests. labels Feb 8, 2017

vsemozhetbyt reviewed Feb 8, 2017

View reviewed changes

jasnell requested changes Feb 8, 2017

View reviewed changes

joyeecheung force-pushed the guide-and-readme branch from 2397287 to e18eeb1 Compare February 12, 2017 07:52

doc: move benchmark/README.md to doc/guides

ded5d20

joyeecheung force-pushed the guide-and-readme branch from e18eeb1 to 09f96f7 Compare February 12, 2017 08:27

joyeecheung added 2 commits February 12, 2017 16:29

doc: add benchmark/README.md and fix guide

9ae1f0d

doc: add descriptions and links to test/README.md

09f96f7

joyeecheung force-pushed the guide-and-readme branch from 4f98007 to fd331c1 Compare February 12, 2017 08:44

TimothyGu reviewed Feb 12, 2017

View reviewed changes

joyeecheung added 2 commits February 12, 2017 16:46

fix benchmark guide links

fd331c1

use tables for test/README.md

409e2fc

joyeecheung force-pushed the guide-and-readme branch from d11bb78 to b88d113 Compare February 12, 2017 10:30

update v8ForceOptimization docs

b88d113

hint about how to use benchmark tools

d5302a1

jasnell approved these changes Feb 17, 2017

View reviewed changes

AndreasMadsen approved these changes Feb 17, 2017

View reviewed changes

joyeecheung closed this Feb 17, 2017

joyeecheung deleted the guide-and-readme branch February 19, 2017 17:43

italoacasas mentioned this pull request Feb 25, 2017

7.7.0 Proposal #11553

Merged

jasnell added the dont-land-on-v4.x label Mar 7, 2017

AndreasMadsen added the dont-land-on-v6.x label Mar 7, 2017

MylesBorins mentioned this pull request Mar 9, 2017

v6.10.1 proposal #11759

Merged

joyeecheung mentioned this pull request Apr 19, 2017

doc: edit the benchmark guide #12041

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: reorganize benchmarking guide/readme, update test/README.md #11237

doc: reorganize benchmarking guide/readme, update test/README.md #11237

joyeecheung commented Feb 8, 2017 •

edited

Loading

joyeecheung commented Feb 8, 2017

AndreasMadsen commented Feb 8, 2017 •

edited

Loading

joyeecheung commented Feb 8, 2017

vsemozhetbyt Feb 8, 2017

joyeecheung Feb 8, 2017

jasnell left a comment

jasnell Feb 8, 2017

jasnell Feb 8, 2017

jasnell Feb 8, 2017

joyeecheung Feb 12, 2017

jasnell Feb 8, 2017

jasnell Feb 8, 2017

jasnell Feb 8, 2017

TimothyGu Feb 12, 2017 •

edited

Loading

joyeecheung Feb 12, 2017

joyeecheung Feb 12, 2017

joyeecheung Feb 12, 2017

joyeecheung Feb 12, 2017 •

edited

Loading

joyeecheung Feb 12, 2017

joyeecheung commented Feb 12, 2017

vsemozhetbyt commented Feb 12, 2017 •

edited

Loading

AndreasMadsen commented Feb 12, 2017

vsemozhetbyt commented Feb 12, 2017

joyeecheung commented Feb 12, 2017

AndreasMadsen commented Feb 12, 2017

joyeecheung commented Feb 12, 2017

AndreasMadsen commented Feb 12, 2017

joyeecheung commented Feb 13, 2017

AndreasMadsen commented Feb 13, 2017 •

edited

Loading

joyeecheung commented Feb 13, 2017

joyeecheung commented Feb 17, 2017 •

edited

Loading

jasnell commented Mar 7, 2017

joyeecheung commented Mar 7, 2017

AndreasMadsen commented Mar 7, 2017


		## Prerequisites

		Most of the HTTP benchmarks require a benchmarker to be installed, this can be


		### Comparing node versions

		To compare the effect of a new node version use the `compare.js` tool. This

doc: reorganize benchmarking guide/readme, update test/README.md #11237

doc: reorganize benchmarking guide/readme, update test/README.md #11237

Conversation

joyeecheung commented Feb 8, 2017 • edited Loading

Checklist

Affected core subsystem(s)

joyeecheung commented Feb 8, 2017

AndreasMadsen commented Feb 8, 2017 • edited Loading

joyeecheung commented Feb 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasnell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TimothyGu Feb 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung Feb 12, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joyeecheung commented Feb 12, 2017

vsemozhetbyt commented Feb 12, 2017 • edited Loading

AndreasMadsen commented Feb 12, 2017

vsemozhetbyt commented Feb 12, 2017

joyeecheung commented Feb 12, 2017

AndreasMadsen commented Feb 12, 2017

joyeecheung commented Feb 12, 2017

AndreasMadsen commented Feb 12, 2017

joyeecheung commented Feb 13, 2017

AndreasMadsen commented Feb 13, 2017 • edited Loading

joyeecheung commented Feb 13, 2017

joyeecheung commented Feb 17, 2017 • edited Loading

jasnell commented Mar 7, 2017

joyeecheung commented Mar 7, 2017

AndreasMadsen commented Mar 7, 2017

joyeecheung commented Feb 8, 2017 •

edited

Loading

AndreasMadsen commented Feb 8, 2017 •

edited

Loading

TimothyGu Feb 12, 2017 •

edited

Loading

joyeecheung Feb 12, 2017 •

edited

Loading

vsemozhetbyt commented Feb 12, 2017 •

edited

Loading

AndreasMadsen commented Feb 13, 2017 •

edited

Loading

joyeecheung commented Feb 17, 2017 •

edited

Loading