[BUG] npm 10.4.0+ running out of memory with no `package-lock.json` #7276

JanStureNielsen · 2024-03-10T17:08:30Z

Is there an existing issue for this?

I have searched the existing issues

This issue exists in the latest npm version

I am using the latest npm

Current Behavior

It appears the latest dependency resolving algorithm is taking more memory than before. The following fails:

npm install -g npm@10.4.0; rm -rf node_modules/ package-lock.json; npm install

as does

npm install -g npm@10.5.0; rm -rf node_modules/ package-lock.json; npm install

with

jan@jan-lite:~/src/my-ui$ npm install -g npm@10.4.0; rm -rf node_modules/ package-lock.json; npm install

removed 13 packages, and changed 28 packages in 2s

24 packages are looking for funding
  run `npm fund` for details
npm WARN using --force Recommended protections disabled.
(⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠼ idealTree:@angular/core: sill placeDep ROOT zone.js@0.14.4 OK for: my-ui@3.4.0 want: ^0.14.4
<--- Last few GCs --->

[11434:0x7323c70]    33445 ms: Scavenge (reduce) 2045.6 (2082.1) -> 2045.6 (2083.1) MB, 7.5 / 0.0 ms  (average mu = 0.634, current mu = 0.776) allocation failure; 
[11434:0x7323c70]    33785 ms: Mark-sweep (reduce) 2046.6 (2083.1) -> 2046.1 (2083.6) MB, 167.1 / 0.0 ms  (+ 70.6 ms in 16 steps since start of marking, biggest step 7.7 ms, walltime since start of marking 255 ms) (average mu = 0.507, current mu = 0.339) 

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb95be0 node::Abort() [npm install]
 2: 0xa9a7f8  [npm install]
 3: 0xd6f5b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [npm install]
 4: 0xd6f957 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [npm install]
 5: 0xf4ceb5  [npm install]
 6: 0xf5f38d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [npm install]
 7: 0xf39a7e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm install]
 8: 0xf3ae47 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm install]
 9: 0xf1b3c0 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [npm install]
10: 0xf12e34 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [npm install]
11: 0xf150e8 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [npm install]
12: 0x1055349 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [npm install]
13: 0x1057066 v8::internal::JsonParser<unsigned short>::ParseJsonValue() [npm install]
14: 0x1057b4f v8::internal::JsonParser<unsigned short>::ParseJson() [npm install]
15: 0xdf5213 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [npm install]
16: 0x170e179  [npm install]
Aborted (core dumped)

I have described this issue on SO as well.

Expected Behavior

The following succeeds:

npm install -g npm@10.3.0; rm -rf node_modules/ package-lock.json; npm install

as does:

npm install -g npm@10.2.5; rm -rf node_modules/ package-lock.json; npm install

Steps To Reproduce

npm install -g npm@10.4.0; rm -rf node_modules/ package-lock.json; npm install --force

or

npm install -g npm@10.5.0; rm -rf node_modules/ package-lock.json; npm install --force

with

{
  "name": "npm-oom",
  "version": "0.0.1",
  "dependencies": {
    "@angular/animations": "^16.2.12",
    "@angular/cdk": "^16.2.14",
    "@angular/cli": "^16.2.12",
    "@angular/common": "^16.2.12",
    "@angular/compiler": "^16.2.12",
    "@angular/compiler-cli": "^16.2.12",
    "@angular/core": "^16.2.12",
    "@angular/flex-layout": "^15.0.0-beta.42",
    "@angular/forms": "^16.2.12",
    "@angular/language-service": "^16.2.12",
    "@angular/material": "^16.2.14",
    "@angular/platform-browser": "^16.2.12",
    "@angular/platform-browser-dynamic": "^16.2.12",
    "@angular/router": "^16.2.12",
    "@sendgrid/mail": "^8.1.1",
    "@stomp/stompjs": "^7.0.0",
    "@types/file-saver-es": "^2.0.3",
    "cross-var": "^1.1.0",
    "enhanced-resolve": "^5.15.1",
    "file-saver-es": "^2.0.5",
    "jwt-decode": "^4.0.0",
    "ng-storages": "^1.1.5",
    "npm": "^10.5.0",
    "reflect-metadata": "^0.2.1",
    "reinstall": "^2.0.0",
    "rxjs": "^7.4.0",
    "tslib": "^2.0.0",
    "typescript": "^5.1.6",
    "zone.js": "^0.14.4"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^16.2.12",
    "@angular-eslint/builder": "^16.3.1",
    "@angular-eslint/eslint-plugin": "^16.3.1",
    "@angular-eslint/eslint-plugin-template": "^16.3.1",
    "@angular-eslint/schematics": "^16.3.1",
    "@angular-eslint/template-parser": "^16.3.1",
    "@types/jasmine": "^5.1.4",
    "@types/node": "^20",
    "@typescript-eslint/eslint-plugin": "^7.1.1",
    "@typescript-eslint/parser": "^7.1.1",
    "jasmine-core": "^5.1.2",
    "jasmine-spec-reporter": "^7.0.0",
    "karma": "^6.3.14",
    "karma-chrome-launcher": "^3.1.0",
    "karma-coverage-istanbul-reporter": "^3.0.2",
    "karma-jasmine": "^5.1.0",
    "karma-jasmine-html-reporter": "^2.1.0",
    "protractor": "^7.0.0",
    "ts-node": "^10.0.0",
    "webdriver-manager": "^12.1.9"
  }
}

Environment

npm: 10.5.0 or 10.4.0
Node.js: 18.19.1
OS Name: Ubuntu 23.10
System Model Name: Dell XPS 13 9310 (0991) 64-bit w/ 8Gb RAM 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
npm config:

npm WARN using --force Recommended protections disabled.
; "project" config from /home/jan/src/qm/fm-ui/.npmrc

force = true 

; node bin location = /home/jan/.nvm/versions/node/v18.19.1/bin/node
; node version = v18.19.1
; npm local prefix = /home/jan/src/my-ui
; npm version = 10.4.0
; cwd = /home/jan/src/my-ui
; HOME = /home/jan
; Run `npm config ls -l` to show all defaults.

The text was updated successfully, but these errors were encountered:

ljharb · 2024-03-10T17:14:31Z

Pretty sure your dependence on “18” is a typo (not sure if it’s relevant) also, your engines.npm isn’t using ^ and doesn’t match the npm version you’re reporting.

JanStureNielsen · 2024-03-10T17:23:39Z

Thanks, @ljharb -- report fixed; same result.

XhmikosR · 2024-03-29T07:32:11Z

I can reproduce this myself under WSL with half the host's RAM available (8 GB) and it's quite a serious regression especially since these npm versions ship with Node.js LTS.

npm v10.2.4 works fine that I could try with the Node.js 20.11.1 docker image. v20.12.0 which has npm 10.5.0 has the memory leak issue which brings down my machine since it starts swapping.

Could someone have a look at it please?

melroy89 · 2024-04-08T09:15:16Z

I got the same problem I believe with: #7351

Somewhat related to #7276 and #7463. I don't think there is a reason to cache the promise here. And if we ever did choose to replace this with an LRUCache we would need to know the size of what we are caching which will be easier if we only cache the resulting manifest. I also added a comment about why I think we are removing the license from manifests here. License was removed in #7126 which looks to be a purposeful change but I could not find a reason. Adding the license back in causes many snapshots to fail because the license is now present in lockfiles, so that's how I came up with the comment.

wraithgar · 2024-05-05T04:14:10Z

We have identified the primary culprit here, it's an unbound cache that saves all packuments in Arborist. Because we now fetch full packuments for all situations, this is causing issues on what would otherwise have been a working system configuration.

This was always technically a problem. We are doing some metrics so we know exactly the right way to fix this, and fix it in a way that hopefully even benefits those running on more memory limited environments.

You can see some initial proof of concept testing at #7463.

Ultimately we plan on putting in an lru-cache instead of an unbound Map, and we are doing analytics to be able to let npm make a decision about how large to let that cache get.

There is also an existing bug that has up until now not meaningfully impacted npm, and that is when the cache has stale data the packument._contentLength attribute is 0. This bug will also have to be fixed so that in those situations npm still knows the size of the packument.

JanStureNielsen · 2024-05-05T14:10:12Z

Thank you for the update, @wraithgar -- have you identified a root-cause commit for this regression?

wraithgar · 2024-05-05T15:00:21Z

Yes, as I said.

Because we now fetch full packuments for all situations

#7126

However it must be reiterated this was always a problem, it just didn't manifest as commonly on systems with plenty of memory. This commit pushed the line much lower, is all.

Somewhat related to #7276 and #7463. I don't think there is a reason to cache the promise here. And if we ever did choose to replace this with an LRUCache we would need to know the size of what we are caching which will be easier if we only cache the resulting manifest. I also added a comment about why I think we are removing the license from manifests here. License was removed in #7126 which looks to be a purposeful change but I could not find a reason. Adding the license back in causes many snapshots to fail because the license is now present in lockfiles, so that's how I came up with the comment.

This adds a new packument cache that is an instance of `lru-cache`. It uses that package's ability to limit content based on size, and has some multipliers based on research to mostly correctly approximate the correlation between packument size and its memory usage. It also limits the total size of the cache based on the actual heap available. Closes: #7276 Related: npm/pacote#369

JanStureNielsen added Bug thing that needs fixing Needs Triage needs review for next steps Release 10.x labels Mar 10, 2024

wraithgar mentioned this issue Apr 14, 2024

[BUG] FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory #7351

Closed

2 tasks

milaninfy added Priority 1 high priority issue and removed Needs Triage needs review for next steps labels May 2, 2024

wraithgar mentioned this issue May 2, 2024

fix: use lru-cache for packuments #7463

Merged

lukekarrys mentioned this issue May 3, 2024

fix: avoid caching manifests as promises #7468

Closed

wraithgar closed this as completed in #7463 May 7, 2024

ZissisT mentioned this issue May 30, 2024

Out of memory after upgrade to latest version angular/angular-cli#27633

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] npm 10.4.0+ running out of memory with no `package-lock.json` #7276

[BUG] npm 10.4.0+ running out of memory with no `package-lock.json` #7276

JanStureNielsen commented Mar 10, 2024 •

edited

Loading

ljharb commented Mar 10, 2024

JanStureNielsen commented Mar 10, 2024 •

edited

Loading

XhmikosR commented Mar 29, 2024 •

edited

Loading

melroy89 commented Apr 8, 2024

wraithgar commented May 5, 2024

JanStureNielsen commented May 5, 2024

wraithgar commented May 5, 2024

[BUG] npm 10.4.0+ running out of memory with no package-lock.json #7276

[BUG] npm 10.4.0+ running out of memory with no package-lock.json #7276

Comments

JanStureNielsen commented Mar 10, 2024 • edited Loading

Is there an existing issue for this?

This issue exists in the latest npm version

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

ljharb commented Mar 10, 2024

JanStureNielsen commented Mar 10, 2024 • edited Loading

XhmikosR commented Mar 29, 2024 • edited Loading

melroy89 commented Apr 8, 2024

wraithgar commented May 5, 2024

JanStureNielsen commented May 5, 2024

wraithgar commented May 5, 2024

[BUG] npm 10.4.0+ running out of memory with no `package-lock.json` #7276

[BUG] npm 10.4.0+ running out of memory with no `package-lock.json` #7276

JanStureNielsen commented Mar 10, 2024 •

edited

Loading

JanStureNielsen commented Mar 10, 2024 •

edited

Loading

XhmikosR commented Mar 29, 2024 •

edited

Loading