Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: perf regression on hot string munging path #286

Merged
merged 2 commits into from
Aug 26, 2021
Merged

Conversation

isaacs
Copy link
Owner

@isaacs isaacs commented Aug 25, 2021

This doesn't change any functionality, but it optimizes a few extremely
hot code paths for the input typically encountered during a large npm
project installation.

String.normalize() is cached, and trailing-slash removal is done with
a single String.slice(), rather than multiple slices and
String.length comparisons.

It is extremely rare that any code path is ever hot enough for this kind
of thing to be relevant enough to justify this sort of
microoptimization, but these two issues resulted in a 40-50% install
time increase in some cases, which is fairly dramatic.

Fix: npm/cli#3676

References

@isaacs isaacs requested a review from nlf August 25, 2021 06:08
@isaacs isaacs requested a review from a team as a code owner August 25, 2021 06:08
lib/path-reservations.js Outdated Show resolved Hide resolved
lib/unpack.js Outdated Show resolved Hide resolved
@isaacs
Copy link
Owner Author

isaacs commented Aug 25, 2021

Running a --offline build using arborist, with the same dependency set provided in npm/cli#3676 with before/after versions of tar:

$ bash test.sh
tar-24b8bdadf37118182496ecf81fa7a872196fb38b

real	0m43.881s
user	0m44.143s
sys	0m30.960s

$ bash test.sh
tar-7ea55c00bd4867d14e25f5be1f7793185f738137

real	0m33.197s
user	0m32.661s
sys	0m31.076s

With a fresh cache, the difference is sometimes much greater, but sometimes smaller, because it's affected by network variations, but with --offline and a full cache, the effect is extremely reliable.

lib/normalize-unicode.js Outdated Show resolved Hide resolved
Copy link
Contributor

@wraithgar wraithgar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with a suggestion you can take or leave.

This doesn't change any functionality, but it optimizes a few extremely
hot code paths for the input typically encountered during a large npm
project installation.

`String.normalize()` is cached, and trailing-slash removal is done with
a single `String.slice()`, rather than multiple slices and
`String.length` comparisons.

It is extremely rare that any code path is ever hot enough for this kind
of thing to be relevant enough to justify this sort of
microoptimization, but these two issues resulted in a 25-50% install
time increase in some cases, which is fairly dramatic.

Fix: npm/cli#3676

PR-URL: #286
Credit: @isaacs
Close: #286
Reviewed-by: @wraithgar
@isaacs isaacs closed this in edb8e9a Aug 26, 2021
@isaacs isaacs merged commit edb8e9a into main Aug 26, 2021
lukekarrys pushed a commit that referenced this pull request Oct 30, 2021
This doesn't change any functionality, but it optimizes a few extremely
hot code paths for the input typically encountered during a large npm
project installation.

`String.normalize()` is cached, and trailing-slash removal is done with
a single `String.slice()`, rather than multiple slices and
`String.length` comparisons.

It is extremely rare that any code path is ever hot enough for this kind
of thing to be relevant enough to justify this sort of
microoptimization, but these two issues resulted in a 25-50% install
time increase in some cases, which is fairly dramatic.

Fix: npm/cli#3676

PR-URL: #286
Credit: @isaacs
Close: #286
Reviewed-by: @wraithgar
lukekarrys pushed a commit that referenced this pull request Oct 30, 2021
This doesn't change any functionality, but it optimizes a few extremely
hot code paths for the input typically encountered during a large npm
project installation.

`String.normalize()` is cached, and trailing-slash removal is done with
a single `String.slice()`, rather than multiple slices and
`String.length` comparisons.

It is extremely rare that any code path is ever hot enough for this kind
of thing to be relevant enough to justify this sort of
microoptimization, but these two issues resulted in a 25-50% install
time increase in some cases, which is fairly dramatic.

Fix: npm/cli#3676

PR-URL: #286
Credit: @isaacs
Close: #286
Reviewed-by: @wraithgar
@lukekarrys lukekarrys deleted the isaacs/fix-npm-cli-3676 branch February 28, 2022 22:21
const {hasOwnProperty} = Object.prototype
module.exports = s => {
if (!hasOwnProperty.call(normalizeCache, s))
normalizeCache[s] = s.normalize('NFKD')
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In long-running server-side application scenarios, this can lead to memory leaks here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] serious npm ci performance regression in 7.21.0
4 participants