[Memory Leak] on module loading

[siluri]

🐛 Bug Report

Jest no longer releases memory for specific modules - e.g. fs-extra | graceful-fs | archiver

To Reproduce

make a testcase which loads one of the moduls and repeat that. so u can see the heapusage increase or u use my repo for that

Expected behavior

memory decrease after testcase

Link to repo

Testcases to reproduce Memory Leak easily include documented test results.

System:
    OS: Windows 10
    CPU: x64 Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz
  Binaries:
    Yarn: 1.7.0 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 5.6.0 - C:\Program Files\nodejs\npm.CMD

Test Result of my repo

✔️result without one of "leaky" moduls.

$ node --expose-gc ./node_modules/jest/bin/jest --runInBand --no-cache --logHeapUsage
 PASS  __tests__/leak.test.js (1).js (40 MB heap size)
 PASS  __tests__/leak.test.js (11).js (40 MB heap size)
 PASS  __tests__/leak.test.js (2).js (40 MB heap size)
 PASS  __tests__/leak.test.js (4).js (40 MB heap size)
 PASS  __tests__/leak.test.js (10).js (40 MB heap size)
 PASS  __tests__/leak.test.js (5).js (40 MB heap size)
 PASS  __tests__/leak.test.js (6).js (40 MB heap size)
 PASS  __tests__/leak.test.js (8).js (40 MB heap size)
 PASS  __tests__/leak.test.js (7).js (40 MB heap size)
 PASS  __tests__/leak.test.js (9).js (40 MB heap size)
 PASS  __tests__/leak.test.js (3).js (40 MB heap size)

Test Suites: 11 passed, 11 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        4.134s
Ran all test suites.
Done in 5.05s.

❌result with fs-extra

$ node --expose-gc ./node_modules/jest/bin/jest --runInBand --no-cache --logHeapUsage
 PASS  __tests__/leak.test.js (1).js (40 MB heap size)
 PASS  __tests__/leak.test.js (11).js (43 MB heap size)
 PASS  __tests__/leak.test.js (2).js (46 MB heap size)
 PASS  __tests__/leak.test.js (3).js (50 MB heap size)
 PASS  __tests__/leak.test.js (10).js (53 MB heap size)
 PASS  __tests__/leak.test.js (5).js (56 MB heap size)
 PASS  __tests__/leak.test.js (6).js (59 MB heap size)
 PASS  __tests__/leak.test.js (7).js (62 MB heap size)
 PASS  __tests__/leak.test.js (9).js (65 MB heap size)
 PASS  __tests__/leak.test.js (8).js (68 MB heap size)
 PASS  __tests__/leak.test.js (4).js (71 MB heap size)

Test Suites: 11 passed, 11 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        4.788s
Ran all test suites.
Done in 5.71s.

❌result with graceful-fs

$ node --expose-gc ./node_modules/jest/bin/jest --runInBand --no-cache --logHeapUsage
 PASS  __tests__/leak.test.js (1).js (40 MB heap size)
 PASS  __tests__/leak.test.js (11).js (43 MB heap size)
 PASS  __tests__/leak.test.js (2).js (46 MB heap size)
 PASS  __tests__/leak.test.js (3).js (49 MB heap size)
 PASS  __tests__/leak.test.js (10).js (52 MB heap size)
 PASS  __tests__/leak.test.js (5).js (55 MB heap size)
 PASS  __tests__/leak.test.js (6).js (58 MB heap size)
 PASS  __tests__/leak.test.js (7).js (61 MB heap size)
 PASS  __tests__/leak.test.js (8).js (64 MB heap size)
 PASS  __tests__/leak.test.js (9).js (66 MB heap size)
 PASS  __tests__/leak.test.js (4).js (69 MB heap size)

Test Suites: 11 passed, 11 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        4.682s
Ran all test suites.
Done in 5.57s.

❌result with node fs & archiver

$ node --expose-gc ./node_modules/jest/bin/jest --runInBand --no-cache --logHeapUsage
 PASS  __tests__/leak.test.js (1).js (43 MB heap size)
 PASS  __tests__/leak.test.js (2).js (47 MB heap size)
 PASS  __tests__/leak.test.js (3).js (51 MB heap size)
 PASS  __tests__/leak.test.js (4).js (54 MB heap size)
 PASS  __tests__/leak.test.js (11).js (58 MB heap size)
 PASS  __tests__/leak.test.js (6).js (62 MB heap size)
 PASS  __tests__/leak.test.js (7).js (65 MB heap size)
 PASS  __tests__/leak.test.js (8).js (69 MB heap size)
 PASS  __tests__/leak.test.js (9).js (72 MB heap size)
 PASS  __tests__/leak.test.js (10).js (76 MB heap size)
 PASS  __tests__/leak.test.js (5).js (80 MB heap size)

Test Suites: 11 passed, 11 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        5.335s
Ran all test suites.
Done in 6.26s.

[siluri]

test with module fast-glob

✔️result with node fs & fast-glob

$ node --expose-gc ./node_modules/jest/bin/jest --runInBand --no-cache --logHeapUsage
 PASS  __tests__/leak.test.js (1).js (43 MB heap size)
 PASS  __tests__/leak.test.js (11).js (43 MB heap size)
 PASS  __tests__/leak.test.js (2).js (43 MB heap size)
 PASS  __tests__/leak.test.js (3).js (43 MB heap size)
 PASS  __tests__/leak.test.js (10).js (46 MB heap size)
 PASS  __tests__/leak.test.js (5).js (43 MB heap size)
 PASS  __tests__/leak.test.js (7).js (43 MB heap size)
 PASS  __tests__/leak.test.js (6).js (43 MB heap size)
 PASS  __tests__/leak.test.js (9).js (43 MB heap size)
 PASS  __tests__/leak.test.js (8).js (43 MB heap size)
 PASS  __tests__/leak.test.js (4).js (46 MB heap size)

Test Suites: 11 passed, 11 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        5.47s
Ran all test suites.
Done in 6.42s.

[kirillgroshkov]

Wow, sounds bad! Any update/progress on that? We noticed that in our project where we have ~50 test files, it started to fail in CircleCI because of memory usage. We use fs-extra in our code.

[siluri]

@kirillgroshkov workaround for your issue is: change fs-extra to fs-jetpack

[kirillgroshkov]

Well, I'm not that easy on just changing dependencies that we use in many projects extensively (fs-extra) to dependency I've never seen before. I'll check it out, of course, thanks! But more likely I'll change to native Node's fs in some problematic cases.

Interesting that we use fs-extra in production and we don't have a memory leak for days and days. It only appears in Jest.

[siluri]

Yes, fs-extra is not leaky. It declare global stuff and every test in jest will declare it again and that leaks. Fs-jetpack don't do that. If u find a solution to cleanup up the global stuff, let me know.

[siluri]

@kirillgroshkov here u see the result of node fs native. But we need fs async - so we use fs-jetpack.

Checkout #6814

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[SimenB]

From what I can tell, this is an issue with those modules, not Jest. Or, Jest probably exposes/excarebates/creates the condition for them to leak, but still. graceful-fs was fixed a couple of years ago, the others still have the issue. But I'd say this is a duplicate of #6814 (which has some details about why this happens).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.