src: use libuv to get env vars #29188

addaleax · 2019-08-17T21:04:48Z

benchmark: improve process.env benchmarks

Benchmark different types of operations and make results comparable
by normalizing process.env for enumeartion.

src: use libuv to get env vars

This allows us to remove OS-dependent code.

                                                       confidence improvement accuracy (*)    (**)   (***)
 process/bench-env.js operation='delete' n=1000000                    3.57 %      ±10.86% ±14.46% ±18.85%
 process/bench-env.js operation='enumerate' n=1000000        ***    -14.06 %       ±7.46%  ±9.94% ±12.96%
 process/bench-env.js operation='get' n=1000000                      -7.97 %      ±11.80% ±15.70% ±20.45%
 process/bench-env.js operation='query' n=1000000                    -1.32 %       ±8.38% ±11.17% ±14.58%
 process/bench-env.js operation='set' n=1000000                      -0.98 %       ±9.63% ±12.81% ±16.68%

The drop in enumeration performance is likely due to the large
number of extra allocations that libuv performs. However, enumerating
process.env should generally not be a hot path in most applications.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

addaleax · 2019-08-17T21:05:53Z

If the drop in enumeration performance is considered an issue for this to happen, I’d probably try to add a callback-based enumeration API to libuv.

nodejs-github-bot · 2019-08-17T21:11:06Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25057/ ❌ (Windows test failures seem relevant?)

addaleax · 2019-08-18T10:27:41Z

I think this is blocked on libuv/libuv#2419 (which, ironically, solves a bug that Node.js also had in the past). I’ve pushed changes for the linter failure.

joyeecheung · 2019-08-20T13:47:33Z

This fixes #27211 right?

addaleax · 2019-08-20T13:52:14Z

@joyeecheung Yeah, it does … I had kind of forgotten that #27310 existed. Should we rebase that one first? It doesn’t take care of everything that this PR does, but most of it.

joyeecheung · 2019-08-20T13:55:46Z

Should we rebase that one first? It doesn’t take care of everything that this PR does, but most of it.

I think either rebasing or keeping this PR works considering #27310 has been inactive for 2 months, but I'll leave it to you to decide :)

nodejs-github-bot · 2019-08-22T05:23:54Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25178/ ❌ (Windows test failures are relevant)

Trott · 2019-08-25T23:26:32Z

benchmark/process/bench-env.js

@@ -3,15 +3,55 @@
 const common = require('../common');

 const bench = common.createBenchmark(main, {
-  n: [1e5],
+  n: [1e6],
+  operation: ['get', 'set', 'enumerate', 'query', 'delete']


I think this change means we need to add a line for operation in test/benchmark/test-benchmark-process.js.

@Trott Thanks for catching, done!

nodejs-github-bot · 2019-09-12T18:00:06Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25452/

nodejs-github-bot · 2019-09-12T19:51:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25455/

nodejs-github-bot · 2019-09-13T13:41:10Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25473/

nodejs-github-bot · 2019-09-13T17:06:36Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25477/

Trott

Windows failures in CI seem relevant. Feel free to dismiss this when tests are fixed or if it is determined that I'm wrong and they're not relevant--no need to ask me or anything. Just putting this here so no one over-ambitious tries to land the PR without that getting resolved.

nodejs-github-bot · 2019-09-13T20:52:10Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25481/

addaleax · 2019-09-13T20:52:31Z

Windows CI should be better now, see libuv/libuv#2473 for the bug behind it (which, luckily, is easy to work around here).

nodejs-github-bot · 2019-09-16T04:47:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25493/

Trott · 2019-09-16T18:41:19Z

Wondering if the failures are due to using "Rebuild" for some reason. Will start a fresh CI from "Build With Parameters" instead and we'll see what happens....

nodejs-github-bot · 2019-09-16T18:41:48Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25503/

nodejs-github-bot · 2019-09-16T22:13:03Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25506/

richardlau · 2019-09-16T22:20:45Z

Wondering if the failures are due to using "Rebuild" for some reason. Will start a fresh CI from "Build With Parameters" instead and we'll see what happens....

Quite likely. "Rebuild" only reruns jobs that have failed. In the case of Windows (and ARM) the compile and test jobs are separate so "rebuild" will not recompile code if the compile job in the original CI passed (even if the code was changed in between the original CI and the rebuild).

addaleax · 2019-09-17T22:15:30Z

Ok, looks like the Windows failures are related again. I’ll take a look.

nodejs-github-bot · 2019-09-17T23:26:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25525/

nodejs-github-bot · 2019-09-17T23:42:06Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25530/

Benchmark different types of operations and make results comparable by normalizing process.env for enumeartion.

This allows us to remove OS-dependent code. confidence improvement accuracy (*) (**) (***) process/bench-env.js operation='delete' n=1000000 3.57 % ±10.86% ±14.46% ±18.85% process/bench-env.js operation='enumerate' n=1000000 *** -14.06 % ±7.46% ±9.94% ±12.96% process/bench-env.js operation='get' n=1000000 -7.97 % ±11.80% ±15.70% ±20.45% process/bench-env.js operation='query' n=1000000 -1.32 % ±8.38% ±11.17% ±14.58% process/bench-env.js operation='set' n=1000000 -0.98 % ±9.63% ±12.81% ±16.68% The drop in enumeration performance is likely due to the large number of extra allocations that libuv performs. However, enumerating process.env should generally not be a hot path in most applications.

nodejs-github-bot · 2019-09-18T17:46:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25551/

addaleax · 2019-09-18T17:47:16Z

Rebased & should have fixed the issue that lead to CI failing, which was that the skipped, unenumerable env vars on Windows were counted towards the length of the resulting array, even though there were no corresponding entries in the array.

nodejs-github-bot · 2019-09-18T20:44:37Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25553/

Benchmark different types of operations and make results comparable by normalizing process.env for enumeartion. PR-URL: #29188 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This allows us to remove OS-dependent code. confidence improvement accuracy (*) (**) (***) process/bench-env.js operation='delete' n=1000000 3.57 % ±10.86% ±14.46% ±18.85% process/bench-env.js operation='enumerate' n=1000000 *** -14.06 % ±7.46% ±9.94% ±12.96% process/bench-env.js operation='get' n=1000000 -7.97 % ±11.80% ±15.70% ±20.45% process/bench-env.js operation='query' n=1000000 -1.32 % ±8.38% ±11.17% ±14.58% process/bench-env.js operation='set' n=1000000 -0.98 % ±9.63% ±12.81% ±16.68% The drop in enumeration performance is likely due to the large number of extra allocations that libuv performs. However, enumerating process.env should generally not be a hot path in most applications. PR-URL: #29188 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

addaleax · 2019-09-18T21:25:15Z

Landed in 82ebcb3 5664791

Benchmark different types of operations and make results comparable by normalizing process.env for enumeartion. PR-URL: #29188 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This allows us to remove OS-dependent code. confidence improvement accuracy (*) (**) (***) process/bench-env.js operation='delete' n=1000000 3.57 % ±10.86% ±14.46% ±18.85% process/bench-env.js operation='enumerate' n=1000000 *** -14.06 % ±7.46% ±9.94% ±12.96% process/bench-env.js operation='get' n=1000000 -7.97 % ±11.80% ±15.70% ±20.45% process/bench-env.js operation='query' n=1000000 -1.32 % ±8.38% ±11.17% ±14.58% process/bench-env.js operation='set' n=1000000 -0.98 % ±9.63% ±12.81% ±16.68% The drop in enumeration performance is likely due to the large number of extra allocations that libuv performs. However, enumerating process.env should generally not be a hot path in most applications. PR-URL: #29188 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Benchmark different types of operations and make results comparable by normalizing process.env for enumeartion. PR-URL: #29188 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

This allows us to remove OS-dependent code. confidence improvement accuracy (*) (**) (***) process/bench-env.js operation='delete' n=1000000 3.57 % ±10.86% ±14.46% ±18.85% process/bench-env.js operation='enumerate' n=1000000 *** -14.06 % ±7.46% ±9.94% ±12.96% process/bench-env.js operation='get' n=1000000 -7.97 % ±11.80% ±15.70% ±20.45% process/bench-env.js operation='query' n=1000000 -1.32 % ±8.38% ±11.17% ±14.58% process/bench-env.js operation='set' n=1000000 -0.98 % ±9.63% ±12.81% ±16.68% The drop in enumeration performance is likely due to the large number of extra allocations that libuv performs. However, enumerating process.env should generally not be a hot path in most applications. PR-URL: #29188 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

anonrig · 2024-06-17T18:18:56Z

A small investigation took me to this pull-request. I have a quick question: Any reason to why we are not using std::getenv which is available on both unix and windows machines?

nodejs-github-bot added the c++ Issues and PRs that require attention from people who are familiar with C++. label Aug 17, 2019

addaleax added the process Issues and PRs related to the process subsystem. label Aug 17, 2019

jasnell approved these changes Aug 18, 2019

View reviewed changes

addaleax added the blocked PRs that are blocked by other issues or PRs. label Aug 18, 2019

cjihrig approved these changes Aug 19, 2019

View reviewed changes

addaleax mentioned this pull request Aug 20, 2019

Fix to #27211 - Refactor RealEnvStore methods to use libuv methods for env operations #27310

Closed

2 tasks

Trott reviewed Aug 25, 2019

View reviewed changes

addaleax force-pushed the env-var-os branch from feb6d8f to ce47850 Compare September 12, 2019 17:57

addaleax added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. and removed blocked PRs that are blocked by other issues or PRs. labels Sep 12, 2019

Trott previously requested changes Sep 13, 2019

View reviewed changes

Trott force-pushed the master branch from 1ecc406 to 49cf67e Compare September 17, 2019 16:51

addaleax added 4 commits September 18, 2019 19:24

benchmark: improve process.env benchmarks

ef68e58

Benchmark different types of operations and make results comparable by normalizing process.env for enumeartion.

fixup! src: use libuv to get env vars

e75c72e

fixup! fixup! src: use libuv to get env vars

eed2884

addaleax force-pushed the env-var-os branch from 347f340 to eed2884 Compare September 18, 2019 17:46

addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Sep 18, 2019

addaleax closed this Sep 18, 2019

addaleax deleted the env-var-os branch September 18, 2019 21:25

BridgeAR mentioned this pull request Sep 24, 2019

v12.11.0 proposal #29695

Merged

cjihrig mentioned this pull request Dec 27, 2019

Assertion failed: len < MAX_ENV_VAR_LENGTH, file c:\ws\deps\uv\src\win\util.c, line 1476 libuv/libuv#2587

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

src: use libuv to get env vars #29188

src: use libuv to get env vars #29188

addaleax commented Aug 17, 2019

addaleax commented Aug 17, 2019

nodejs-github-bot commented Aug 17, 2019 •

edited by Trott

Loading

addaleax commented Aug 18, 2019

joyeecheung commented Aug 20, 2019

addaleax commented Aug 20, 2019

joyeecheung commented Aug 20, 2019

nodejs-github-bot commented Aug 22, 2019 •

edited by Trott

Loading

Trott Aug 25, 2019

addaleax Sep 12, 2019

nodejs-github-bot commented Sep 12, 2019

nodejs-github-bot commented Sep 12, 2019

nodejs-github-bot commented Sep 13, 2019

nodejs-github-bot commented Sep 13, 2019

Trott left a comment •

edited

Loading

nodejs-github-bot commented Sep 13, 2019

addaleax commented Sep 13, 2019

nodejs-github-bot commented Sep 16, 2019

Trott commented Sep 16, 2019

nodejs-github-bot commented Sep 16, 2019

nodejs-github-bot commented Sep 16, 2019

richardlau commented Sep 16, 2019

addaleax commented Sep 17, 2019

nodejs-github-bot commented Sep 17, 2019

nodejs-github-bot commented Sep 17, 2019

nodejs-github-bot commented Sep 18, 2019

addaleax commented Sep 18, 2019 •

edited

Loading

nodejs-github-bot commented Sep 18, 2019

addaleax commented Sep 18, 2019

anonrig commented Jun 17, 2024

src: use libuv to get env vars #29188

src: use libuv to get env vars #29188

Conversation

addaleax commented Aug 17, 2019

Checklist

addaleax commented Aug 17, 2019

nodejs-github-bot commented Aug 17, 2019 • edited by Trott Loading

addaleax commented Aug 18, 2019

joyeecheung commented Aug 20, 2019

addaleax commented Aug 20, 2019

joyeecheung commented Aug 20, 2019

nodejs-github-bot commented Aug 22, 2019 • edited by Trott Loading

Trott Aug 25, 2019

Choose a reason for hiding this comment

addaleax Sep 12, 2019

Choose a reason for hiding this comment

nodejs-github-bot commented Sep 12, 2019

nodejs-github-bot commented Sep 12, 2019

nodejs-github-bot commented Sep 13, 2019

nodejs-github-bot commented Sep 13, 2019

Trott left a comment • edited Loading

Choose a reason for hiding this comment

nodejs-github-bot commented Sep 13, 2019

addaleax commented Sep 13, 2019

nodejs-github-bot commented Sep 16, 2019

Trott commented Sep 16, 2019

nodejs-github-bot commented Sep 16, 2019

nodejs-github-bot commented Sep 16, 2019

richardlau commented Sep 16, 2019

addaleax commented Sep 17, 2019

nodejs-github-bot commented Sep 17, 2019

nodejs-github-bot commented Sep 17, 2019

nodejs-github-bot commented Sep 18, 2019

addaleax commented Sep 18, 2019 • edited Loading

nodejs-github-bot commented Sep 18, 2019

addaleax commented Sep 18, 2019

anonrig commented Jun 17, 2024

nodejs-github-bot commented Aug 17, 2019 •

edited by Trott

Loading

nodejs-github-bot commented Aug 22, 2019 •

edited by Trott

Loading

Trott left a comment •

edited

Loading

addaleax commented Sep 18, 2019 •

edited

Loading