How to clear used memory between test suites? (possible memory leak)

[racs4]

Versions

• NodeJS: 18.16.1
• mongodb-memory-server-*: 6.3.3
• mongodb(the binary version): 4.4.13
• mongoose: 5.9.4
• system: Linux, Ubuntu 23.04

package: mongo-memory-server

What is your question?

Hello, I'm using jest to perform integration tests and I have a lot of suites that were running smoothly. But as the number of suites increased, a considerable increase in memory usage was noticed, to the point that it is impossible to run the tests. (The image below shows the heap usage after some time testing)

Doing a bit of debugging, using node --inspect-brk --expose-gc ./node_modules/.bin/jest --runInBand --logHeapUsage and taking a snapshot of the heap with the chromium inspection tool, you can see that there are several "lost" BSON objects in memory, which for some reason are referenced when moving from one test suite to another.

My jest configuration follows a single file setup present in this template:

let mongoServer

beforeAll(async () => {
  mongoServer = new MongodbMemoryServer()
  const mongoUri = await mongoServer.getUri()
  mongoose.connect(mongoUri, undefined, (err) => {
    if (err) console.error(err)
  })
})

afterAll(async () => {
  await mongoose.disconnect()
  await mongoServer.stop()
})

afterEach(async () => {
  const { collections } = mongoose.connection
  const promises = []
  Object.keys(collections).forEach((collection) => {
    promises.push(collections[collection].deleteMany({}))
  })
  await Promise.all(promises)
})

I already saw the recommended settings for jest present in the documentation of this library, but there a single instance of mongo-memory-server is created for all suites, while in the code of this template a new one is created for each test suite and I would like to keep it that way.

My question is if you or someone who has had this problem could help me to find out why these objects are getting referenced between one test suite and another, preventing them from being cleared from memory, and how to solve it.

Thanks in advance for any help

[hasezoey]

mongodb-memory-server-*: 6.3.3

please update to a newer version and test again, currently latest version is 8.13.0, migrations guides

also if you still have the issue after the upgrade, please make a reproduction repository so that it can be tested

[racs4]

Thanks for the quick response.

I updated and still got the same problem.

But I discovered that when I create a project with the template I used, it presents the same problem, only on a much smaller scale because it has far fewer tests than my project (The "template" is actually a yeoman generator called rest).

I made a repository with it in its initial settings to make it easier to test. The default version it uses of your library is the outdated one, but I made a branch called new-version with the update.

With that, it could be that the error is in some configuration of this template, so I will make an issue there as well. But I would be very grateful if you could give this repository a test to see what is happening (I don't think I have enough knowledge)

[hasezoey]

i did some inspecting myself

you can see that there are several "lost" BSON objects in memory, which for some reason are referenced when moving from one test suite to another.

at least the onces shown in your screenshot are expected to be there, because they are (module)global buffers (i dont know why they make it (module)global though), see https://github.com/mongodb/js-bson/blob/1dcca92b24e609a069ca7f4187e5b9521380b2f5/src/bson.ts#L68

---

i found that when disabling the following get-port lines there is less memory leakage, but i dont understand why (the interval never actually gets run because the tests are short, and the values that are assigned would become "lost" and so should be collected and the interval itself is unrefd) https://github.com/sindresorhus/get-port/blob/a9b445ea0afbd75b81b7b9011c896e111e2d4aee/index.js#L50-L60

aside from those, i am not proficient in debugging these issues and i cannot figure out where the problem is, the only common things across runs i have found is that jest sometimes just does not clean-up the test-suite and somehow still references it (and so has duplicate modules in string and compiled form)

note: i have never understood how to properly use the chrome memory debugger or how to read its output, not being much different in this case

[hasezoey]

i still dont understand how to read the output of the chrome debugger fully, but my assumption for now is that jest somehow keeps all the test contexts around (for as long as jest is running), which includes all module definitions (by extension mongodb-memory-server, mongodb, bson) and i have not found a way to manually force it to be removed

for now i have created this stackoverflow question in hopes someone more experienced has a better answer

[racs4]

Thank you very much for your time and dedication in responding.

I stayed last night debugging, and I also couldn't solve the problem. I found some questions and problems related to the 3.x version of the mongodb package like this one here. The template I used is using mongoose in version 5.9.4 which uses version 3.5.x of this lib (as can be seen in this package.json). I thought I'd try upgrading to a newer version of mongoose to see if this is the issue.

I also ran into mongoose-related issues (mostly on old versions), like this Automattic/mongoose#2874. But I tried the solutions mentioned there without success.

On my chrome://inspect debugs i found out that the "persistent" objets were placed in memory between the end of a suite and the start of another (I did this putting breakpoints on the start of beforeAll and the end of afterAll and then taking snapshots in these places), and in my case it seems to be always placing two system / JSArrayBufferData objects between these times, but the console below always change what is shown (the BSON object seems to appears in most of the times, but the other references changes regularly), so i am not being able to get anything more consistent.

my assumption for now is that jest somehow keeps all the test contexts around

Yes, I got this feeling too. I was thinking of testing with another test lib just to see what the result would be

I think you can close this issue if you want, if I have any updates I'll post it here in case it helps someone else. thank you so much again for your time and for asking the question on stack overflow

[hasezoey]

The template I used is using mongoose in version 5.9.4 which uses version 3.5.x of this lib (as can be seen in this package.json). I thought I'd try upgrading to a newer version of mongoose to see if this is the issue.

i had been able to repro this issue without much else dependencies, see https://github.com/hasezoey/jest-mem-leak/tree/advanced

and in my case it seems to be always placing two system / JSArrayBufferData objects between these times, but the console below always change what is shown (the BSON object seems to appears in most of the times, but the other references changes regularly), so i am not being able to get anything more consistent.

in the chain there is likely something like <symbol JEST_STATE_SYMBOL> right? that is also what i ran into

and most BSON related thing (at least those i found) were related to the module buffer of BSON which i mentioned earlier

---

as i outlined in my last comments (and on stackoverflow), it seems the setInterval from get-port is a major part, and so i will try to make a own version that is more suited to MMS for the next major version (i will keep this issue open until then OR the stackoverflow gets a decisive / fixable answer)

[racs4]

i had been able to repro this issue without much else dependencies, see https://github.com/hasezoey/jest-mem-leak/tree/advanced

Yes, I saw this. I tested this repo with mocha just to see what the result would be in
racs4/jest-mem-leak:mocha. If I did everything right, this repo shows that mocha doesn't leak memory like jest does in this case.

in the chain there is likely something like right?

Yes, this image shows the debugger attached to your test repo:

There seem to be known issues regarding this behavior of jest like here jestjs/jest#7874 that jest leaks memory without any additional dependencies and here jestjs/jest#14042 several people mentioning a problem similar to the one I had at the beginning of this thread. Some people in these issues said they solved it by downgrading the node version to version 16.10, but I did that and tested it in your repo and still got the same memory leak.

[hasezoey]

for now i have replaced the get-port package with a internal implementation in feature/9.0, i will likely not backport this change

also see #579 (comment)

@racs4 could you re-test with the current feature/9.0 branch and see if that at least lowers the leaking?
how-to:

• have MMS repository downloaded and the mentioned branch checked-out
• run yarn install (anywhere except in website/)
• cd into packages/mongodb-memory-server-core
• run yarn run build
• run yarn link
• go to your project and run yarn link mongodb-memory-server-core (if you are using anything else than -core you will need to replace that for this testing)

Notes:

• when using the linked package in https://github.com/hasezoey/jest-mem-leak/tree/advanced with --expose-gc the heap stays consistent
• when using the linked package in https://github.com/hasezoey/jest-mem-leak/tree/advanced without --expose-gc it grows somewhat, but not out of range
• when using the linked package in https://github.com/racs4/yo-memory-test/tree/new-version with or without --expose-gc it still grows, but inconsistently
• when using the linked package in https://github.com/racs4/yo-memory-test/tree/new-version with a inspector attached, the heap actually stays consistent (i have no clue why)

Edit:
it seems like updating mongoose to 6.x also fixes some issues, while being run with --expose-gc (and having global.gc && global.gc() in afterAll) the memory is lower than with mongoose 5.x, and without --expose-gc it is still getting high, but also going lower again

[racs4]

it seems to have worked, it's much better than it was before, even when inconsistent.

I did some tests and got these interesting results:

• Running raw jest (just jest --runInBand --logHeapUsage) it looks like it grows up to an arbitrary limit and then falls off and stays in this endless loop. The first time I ran it it went up to ~500mb and then it went back to ~300mb and stayed in that cycle. The maximum it reached (in another run) was ~1500mb, later going back to ~300mb as well.

• Running with node --expose-gc and doing global.gc() in afterAll didn't bring much difference. But adding --no-compilation-cache, as mentioned in the jest issues, made the memory consumption much more consistent (memory still seemed to be growing, but at a much slower rate).

• Lastly I compared two node versions, 16.10 and 18+, using the last mentioned command. In both versions, the use of heap memory behaved in the same way as described in the previous paragraph. The difference is that on node 18+ the test already started with usage close to 390mb and ended at 550mb+ and on node 16.10 it started at 315mb and ended at 400mb.

Remembering that the repository where I tested has 390+ test suites and almost all of them use the mongo-memory-server. Before this change you made, it was impossible to run these tests normally, since memory usage increased at a very high rate (in addition to causing a lack of memory, it also caused slowdowns if you filtered a folder to test), so thank you very much for this finding and for this change you made.

[marco-bertelli]

i know that is a jest problem, but if it can help someone i have mitigated this with 2 steps (reduce mem from 1.4gb to 650/700 mb)

• first change the default jest runner to the @side/jest-runtime this is possible putting into the jest config the following line: "runtime": "@side/jest-runtime",
• force clear the global cache object with following command when you close the mongo instance:

 export const closeInMongodConnection = async () => {
  global.gc && global.gc();

  if (mongod) await mongod.stop();
};

this not resolve the problem but mitigate it.

EDIT:
using the beta feature jest flag --processWorkers=true will mitigate a lot more (from 1.4gb to max 400mb of heap use), hope it can help

[hasezoey]

beta 9.0.0-beta.1 has been released, which has the replacement for get-port, which hopefully has less leakage

[hasezoey]

this was kinda wrongly marked stale, i will re-open this at least until 9.0.0 is properly released

[hasezoey]

🎉 This issue has been resolved in version 9.0.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• npm package (@latest dist-tag)
• npm package (@latest dist-tag)
• npm package (@latest dist-tag)
• npm package (@latest dist-tag)
• npm package (@latest dist-tag)
• GitHub release

Manual message because semantic-release hit the limit

---

i will close this now, because i cant reproduce more memory-leaks, please open a new issue (maybe reference this old one) if another problem occurs with the latest version