fs: improve performance for sync stat() functions #11522

mscdex · 2017-02-23T18:07:15Z

Results from benchmarks:

                                                 improvement confidence      p.value
 fs/bench-statSync.js kind="fstatSync" n=1000000     13.51 %        *** 4.033815e-18
 fs/bench-statSync.js kind="statSync" n=1000000       7.09 %        *** 6.502451e-15
 fs/bench-statSync.js kind="lstatSync" n=1000000      7.43 %        *** 9.979362e-16

CI: https://ci.nodejs.org/job/node-test-pull-request/6568/

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

fs

bnoordhuis

LGTM. Is there value in keeping the sync non-float64array code paths in the binding methods?

mscdex · 2017-02-23T18:44:30Z

@bnoordhuis I left them there for backwards compatibility for now, even though third-party use of process.binding() is frowned upon/not supported. *shrug*

bnoordhuis · 2017-02-23T19:51:55Z

I left them there for backwards compatibility for now, even though third-party use of process.binding() is frowned upon/not supported.

I figured that was the reason but I'd worry that people construe that as a nod of approval that it's okay to use the bindings layer directly.

mscdex · 2017-02-23T21:06:22Z

@bnoordhuis I don't mind either way. If most @nodejs/collaborators agree to remove support for the old method, then I will remove it.

cjihrig · 2017-02-23T23:56:59Z

src/node_file.cc

@@ -649,7 +696,14 @@ static void FStat(const FunctionCallbackInfo<Value>& args) {

  int fd = args[0]->Int32Value();

-  if (args[1]->IsObject()) {
+    if (args[1]->IsFloat64Array()) {


Indentation looks off here.

jasnell · 2017-02-24T00:15:16Z

The recently added deprecation policy makes it fairly clear that process.binding() is not covered under our API guarantees. I know that it's a recent change but I believe it gives sufficient coverage on this.

mscdex · 2017-02-24T02:27:31Z

I've removed the non-Float64Array sync() branches.

mscdex · 2017-02-24T02:28:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/6573/

mcollina

CI does not seem green.
LGTM when it does.

bnoordhuis · 2017-02-24T08:43:30Z

fs.exists() and fs.existsSync() use binding.stat() directly, they'll need to be updated.

mscdex · 2017-02-24T09:10:15Z

Fixed.

CI again: https://ci.nodejs.org/job/node-test-pull-request/6576/

PR-URL: nodejs#11522 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Node 7.7 changed the behavior of the `binding.{stat,lstat,fstat}` functions (see nodejs/node#11522). This updates the binding functions used in mock-fs to match the new Node binding behavior, while still maintaining compatibility with old Node versions. The new behavior is detected when the second argument to `binding.{stat,lstat,fstat}` is a `Float64Array`, which would be an invalid argument for previous versions of the binding.

…haub#197) Node 7.7 changed the behavior of the `binding.{stat,lstat,fstat}` functions (see nodejs/node#11522). This updates the binding functions used in mock-fs to match the new Node binding behavior, while still maintaining compatibility with old Node versions. The new behavior is detected when the second argument to `binding.{stat,lstat,fstat}` is a `Float64Array`, which would be an invalid argument for previous versions of the binding.

jasnell · 2017-03-07T04:57:05Z

@nodejs/lts @mscdex ... should we pull this into v6.x-staging?

mscdex · 2017-03-07T05:06:33Z

@jasnell Fine by me, it should still provide a speedup even though I haven't explicitly benchmarked v6.x.

Why the dont-land-on-v4.x?

jasnell · 2017-03-07T05:09:07Z

like the other one, the don't land is largely precautionary

PR-URL: #11522 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Notable changes * performance: The performance of several APIs has been improved. - `Buffer.compare()` is up to 35% faster on average. (Brian White) #10927 - `buffer.toJSON()` is up to 2859% faster on average. (Brian White) #10895 - `fs.*statSync()` functions are now up to 9.3% faster on average. (Brian White) #11522 - `os.loadavg` is up to 151% faster. (Brian White) #11516 - `process.memoryUsage()` is up to 34% faster. (Brian White) #11497 - `querystring.unescape()` for `Buffer`s is 15% faster on average. (Brian White) #10837 - `querystring.stringify()` is up to 7.8% faster on average. (Brian White) #10852 - `querystring.parse()` is up to 21% faster on average. (Brian White) #10874 * IPC: - Batched writes have been enabled for process IPC on platforms that support Unix Domain Sockets. (Alexey Orlenko) #10677 - Performance gains may be up to 40% for some workloads. * child_process: - `spawnSync` now returns a null `status` when child is terminated by a signal. (cjihrig) #11288 - This fixes the behavior to act like `spawn()` does. * http: - Control characters are now always rejected when using `http.request()`. (Ben Noordhuis) #8923 - Debug messages have been added for cases when headers contain invalid values. (Evan Lucas) #9195 * node: - Heap statistics now support values larger than 4GB. (Ben Noordhuis) #10186 * timers: - Timer callbacks now always maintain order when interacting with domain error handling. (John Barboza) #10522 PR-URL: #11759

Notable changes * performance: The performance of several APIs has been improved. - `Buffer.compare()` is up to 35% faster on average. (Brian White) nodejs/node#10927 - `buffer.toJSON()` is up to 2859% faster on average. (Brian White) nodejs/node#10895 - `fs.*statSync()` functions are now up to 9.3% faster on average. (Brian White) nodejs/node#11522 - `os.loadavg` is up to 151% faster. (Brian White) nodejs/node#11516 - `process.memoryUsage()` is up to 34% faster. (Brian White) nodejs/node#11497 - `querystring.unescape()` for `Buffer`s is 15% faster on average. (Brian White) nodejs/node#10837 - `querystring.stringify()` is up to 7.8% faster on average. (Brian White) nodejs/node#10852 - `querystring.parse()` is up to 21% faster on average. (Brian White) nodejs/node#10874 * IPC: - Batched writes have been enabled for process IPC on platforms that support Unix Domain Sockets. (Alexey Orlenko) nodejs/node#10677 - Performance gains may be up to 40% for some workloads. * child_process: - `spawnSync` now returns a null `status` when child is terminated by a signal. (cjihrig) nodejs/node#11288 - This fixes the behavior to act like `spawn()` does. * http: - Control characters are now always rejected when using `http.request()`. (Ben Noordhuis) nodejs/node#8923 - Debug messages have been added for cases when headers contain invalid values. (Evan Lucas) nodejs/node#9195 * node: - Heap statistics now support values larger than 4GB. (Ben Noordhuis) nodejs/node#10186 * timers: - Timer callbacks now always maintain order when interacting with domain error handling. (John Barboza) nodejs/node#10522 PR-URL: nodejs/node#11759 Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

…haub#197) Node 7.7 changed the behavior of the `binding.{stat,lstat,fstat}` functions (see nodejs/node#11522). This updates the binding functions used in mock-fs to match the new Node binding behavior, while still maintaining compatibility with old Node versions. The new behavior is detected when the second argument to `binding.{stat,lstat,fstat}` is a `Float64Array`, which would be an invalid argument for previous versions of the binding.

mscdex added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. performance Issues and PRs related to the performance of Node.js. labels Feb 23, 2017

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. labels Feb 23, 2017

mscdex force-pushed the fs-statsync-perf branch from 92a8350 to 99f37aa Compare February 23, 2017 18:11

bnoordhuis approved these changes Feb 23, 2017

View reviewed changes

jasnell approved these changes Feb 23, 2017

View reviewed changes

addaleax approved these changes Feb 23, 2017

View reviewed changes

cjihrig reviewed Feb 23, 2017

View reviewed changes

mscdex force-pushed the fs-statsync-perf branch 2 times, most recently from ef8da93 to 0f3da51 Compare February 24, 2017 02:26

mcollina approved these changes Feb 24, 2017

View reviewed changes

mscdex force-pushed the fs-statsync-perf branch from 0f3da51 to 08db001 Compare February 24, 2017 09:09

bnoordhuis approved these changes Feb 24, 2017

View reviewed changes

cjihrig approved these changes Feb 24, 2017

View reviewed changes

mscdex force-pushed the fs-statsync-perf branch from 08db001 to 22abb39 Compare February 26, 2017 02:01

mscdex merged commit 22abb39 into nodejs:master Feb 26, 2017

mscdex deleted the fs-statsync-perf branch February 26, 2017 02:05

italoacasas mentioned this pull request Feb 27, 2017

7.7.0 Proposal #11553

Merged

not-an-aardvark mentioned this pull request Mar 2, 2017

Calling fs.statSync with an active mock causes an error in Node 7.7.1 tschaub/mock-fs#197

Closed

not-an-aardvark mentioned this pull request Mar 2, 2017

Update fs.stat(), fs.lstat(), and fs.fstat() for Node 7.7+ (fixes #197) tschaub/mock-fs#198

Merged

mscdex mentioned this pull request Mar 3, 2017

fs: include more fs.stat*() optimizations #11665

Closed

3 tasks

jasnell added the dont-land-on-v4.x label Mar 7, 2017

MylesBorins mentioned this pull request Mar 9, 2017

v6.10.1 proposal #11759

Merged

mscdex mentioned this pull request Apr 15, 2017

fs.stat and fs.statSync may not return identical struct #12419

Closed

JLHwung mentioned this pull request Nov 3, 2017

backport: 11665 to v6 #16667

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fs: improve performance for sync stat() functions #11522

fs: improve performance for sync stat() functions #11522

mscdex commented Feb 23, 2017 •

edited

Loading

bnoordhuis left a comment

mscdex commented Feb 23, 2017 •

edited

Loading

bnoordhuis commented Feb 23, 2017

mscdex commented Feb 23, 2017 •

edited

Loading

cjihrig Feb 23, 2017

mscdex Feb 24, 2017

jasnell commented Feb 24, 2017

mscdex commented Feb 24, 2017

mscdex commented Feb 24, 2017

mcollina left a comment

bnoordhuis commented Feb 24, 2017 •

edited

Loading

mscdex commented Feb 24, 2017

jasnell commented Mar 7, 2017

mscdex commented Mar 7, 2017 •

edited

Loading

jasnell commented Mar 7, 2017

fs: improve performance for sync stat() functions #11522

fs: improve performance for sync stat() functions #11522

Conversation

mscdex commented Feb 23, 2017 • edited Loading

Checklist

Affected core subsystem(s)

bnoordhuis left a comment

Choose a reason for hiding this comment

mscdex commented Feb 23, 2017 • edited Loading

bnoordhuis commented Feb 23, 2017

mscdex commented Feb 23, 2017 • edited Loading

cjihrig Feb 23, 2017

Choose a reason for hiding this comment

mscdex Feb 24, 2017

Choose a reason for hiding this comment

jasnell commented Feb 24, 2017

mscdex commented Feb 24, 2017

mscdex commented Feb 24, 2017

mcollina left a comment

Choose a reason for hiding this comment

bnoordhuis commented Feb 24, 2017 • edited Loading

mscdex commented Feb 24, 2017

jasnell commented Mar 7, 2017

mscdex commented Mar 7, 2017 • edited Loading

jasnell commented Mar 7, 2017

mscdex commented Feb 23, 2017 •

edited

Loading

mscdex commented Feb 23, 2017 •

edited

Loading

mscdex commented Feb 23, 2017 •

edited

Loading

bnoordhuis commented Feb 24, 2017 •

edited

Loading

mscdex commented Mar 7, 2017 •

edited

Loading