The `ephemeralForTest` storage engine is reaching its end of life #742

LucVidal360 · 2023-02-09T14:58:07Z

Versions

NodeJS: 16.19.0
mongodb-memory-server-*: 8.4.2
mongodb(the binary version): 5.0.14
mongodb(the js package): 4.5.0
system: Linux Kubuntu 22.04

package: mongo-memory-server-global

What is your question?

The ephemeralForTest storage engine is reaching its end of life.

Context

In the project I'm working on, each test cleans its test collections by issuing collection.deleteMany({}) on every modified collection during the test. All these deleteMany are executed in parallel.

When executing many tests, it happens that the mongod server crashes with a segfault.

After opening an issue in MongoDB's Jira, it appears the "ephemeralForTest" engine has become unstable, starting with MongoDB 5.
They even intend to suppress entirely this engine with MongoDB 7. Cf. this comment:

In 5.0, we replaced the ephemeralForTest storage engine [...] The new implementation [...] had many bugs, like the one you discovered, and in the 7.0 release, we will remove the ephemeralForTest engine entirely

Workaround

As suggested, I used the wiredTiger engine (thanks to mongo-memory-server's option instance.storageEngine) and everything works like a charm.
I made a few benchmarks, and the performances are comparable.

Question

Knowing this, I have a few interrogations/suggestions:

should this information be documented?
would it be possible to have a different default engine depending on the mongodb version used?

The text was updated successfully, but these errors were encountered:

hasezoey · 2023-02-09T15:03:39Z

would it be possible to have a different default engine depending on the mongodb version used?

not currently

should this information be documented?

probably should be documented yes, though personally i have not run into any issues with that yet

i also already read that they want to remove this storage-engine (sadly). also there seems to be a separate in-memory engine for enterprise.

when the change occurs and MMS changes its default version (to that version), then the default will also be changed (because otherwise it literally wouldnt work anymore).

if it really becomes required, the default may be changed earlier (though this will only be in a major MMS version).

for now use the mentioned workaround of manually setting the storage engine

LucVidal360 · 2023-02-10T07:32:34Z

Thank you for your answer.

The current workaround is fine by me. I mostly intended to inform you about this.

Feel free to close this issue, or change its status, whatever suits you best.

hasezoey · 2023-02-10T11:24:33Z

Feel free to close this issue, or change its status, whatever suits you best.

for now i will keep it open until at least a documentation note is made about that it may be unstable and that the default will be changed in the future

github-actions · 2023-08-15T12:43:27Z

🎉 This issue has been resolved in version 9.0.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

hasezoey · 2023-08-15T12:45:56Z

a note has been added in beta documnetation at https://nodkz.github.io/mongodb-memory-server/versions/beta/docs/api/interfaces/mongo-memory-instance-opts#helper-type-storageengine

alexbjorlig · 2023-08-22T09:29:10Z

Wait a moment?

Does this means that if I would like to run MongoDB memory server, with version 7, I have to use wiredTiger?

But if I understand wiredTiger correctly, this means that it will not be in-memory then?

hasezoey · 2023-08-22T10:22:29Z

But if I understand wiredTiger correctly, this means that it will not be in-memory then?

yes that is correct, because with 7.0.0 wiredTiger is the only engine supported (at least for the community server version)

i know that the name mongodb-memory-server might not be entirely correct anymore, but i think this package will not rebrand

…s 7.0.0 or higher fixes #742

github-actions · 2023-08-22T10:36:06Z

🎉 This issue has been resolved in version 9.0.0-beta.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

sibelius · 2023-10-04T11:23:23Z

can we release v9 outside beta? mongo 7 is out

hasezoey · 2023-10-04T11:29:30Z

mms 9 i currently in beta to get some more feedback, but yes, i think it is kinda final (it works in both typegoose and mongoose, my main real-world testing use-cases)

i will try to release it this week

sibelius · 2023-10-06T18:33:12Z

writeTiger is slower than ephemeral and breaks some tests

hasezoey · 2023-10-06T18:37:47Z

🎉 This issue has been resolved in version 9.0.0 🎉

The release is available on:

Manual message because semantic-release hit the limit

hasezoey · 2023-10-06T18:39:10Z

well wiredTiger is definitely slower than the in-memory engine, but it should not cause different behavior where the in-memory engine worked.

it is recommended to set the db-path to somewhere which is using tmpfs(or equivalent)

clemp6r · 2024-07-17T15:36:59Z

But if I understand wiredTiger correctly, this means that it will not be in-memory then?

yes that is correct, because with 7.0.0 wiredTiger is the only engine supported (at least for the community server version)

i know that the name mongodb-memory-server might not be entirely correct anymore, but i think this package will not rebrand

The project description and documentation should at least be updated because it is currently misleading.

hasezoey · 2024-07-17T15:57:10Z

The project description and documentation should at least be updated because it is currently misleading.

Thanks for pointing this out, completely forgot about that, for now i have updated the github description, other places (like documentation landing page) may still need to be updated.

I am not a good writer, so for now the more concise and less miss-leading text is:

Manage & spin up mongodb server binaries with zero(or slight) configuration for tests.

EDIT:
i now also updated the documentation (though only for 10.0+ and currently only available in beta)

required because mongodb 7.0 does not have the public in-memory engine anymore. re #742 (comment)

Climax777 · 2024-12-06T15:07:05Z

I know the issue is closed, but wanted to discuss the option of using Percona's Memory Engine? It is a drop-in replacement for MongoDB.

https://docs.percona.com/percona-server-for-mongodb/7.0/inmemory.html
https://www.percona.com/downloads provides a few links. Not sure if it will follow the same download granularity though.

But I believe it will address the concern relating to performance.

hasezoey · 2024-12-06T15:28:19Z

Thanks for the hint @Climax777, but from my quick skim of the site, it looks like percona does not offer a archive overview like official mongodb does, so finding all the links and configurations wouldnt be as easy.
Additionally it looks like they use a different url directory structure all-together, meaning a full custom implementation would be needed.

https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-debian11-7.0.15.tgz
vs
https://downloads.percona.com/downloads/percona-server-mongodb-7.0/percona-server-mongodb-7.0.15-9/binary/tarball/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal.tar.gz

If the tar structure is the same and the commands also work the same, i think it is possible to use percona if you set config option DOWNLOAD_URL (or manually download it and use SYSTEM_BINARY) in mongodb-memory-server today.

sibelius · 2024-12-06T17:41:12Z

does percona works with mongo 8 ?

is it faster ?

is it an in-place replacement ?

sibelius · 2024-12-06T18:02:56Z

https://www.mongodb.com/docs/manual/core/inmemory/

what about using mongodb in memory engine ?

sibelius · 2024-12-06T18:10:37Z

should we open an issue to support percona ?

hasezoey · 2024-12-07T11:32:05Z

@sibelius

what about using mongodb in memory engine ?

An In-Memory storage engine is available in MongoDB Enterprise. Rather than storing documents on-disk, it retains them in-memory for more predictable data latencies.

source
TL;DR: The inMemory engine in official mongodb is not available in the community edition.

is it an in-place replacement ?

Percona Server for MongoDB is an enhanced, fully compatible, source available, drop-in replacement for MongoDB 7.0 Community Edition with enterprise-grade features.

source

does percona works with mongo 8 ?

From what i have seen on their website, the highest version is currently 7.0.x.

should we open an issue to support percona ?

I currently do not plan to add this myself, but PRs are welcome.
(Also #701 had been closed as it was a question of "can we use"(note not "support") and i had written that it should work with SYSTEM_BINARY and then had become stale)
Suggestion: open a "Idea" discussion instead of a issue.

sibelius · 2024-12-07T13:00:12Z

I'd tried with download dir and also system binary neither worked

hasezoey · 2024-12-07T14:27:35Z

I'd tried with download dir

config option DOWNLOAD_DIR changes the directory the binary is downloaded into, it does not set the download URL.

also system binary neither worked

what is the Debug output if you run it with config option SYSTEM_BINARY?

sibelius · 2024-12-09T15:10:56Z

MongoMS:DryMongoBinary getSystemPath: found system binary path at "percona/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal/bin/mongod" +0ms
  MongoMS:DryMongoBinary locateBinary: env "SYSTEM_BINARY" was provided with value: "percona/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal/bin/mongod" +0ms
  MongoMS:DryMongoBinary getSystempath +0ms
  MongoMS:DryMongoBinary getSystemPath: found system binary path at "percona/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal/bin/mongod" +0ms
  MongoMS:MongoBinary getPath: Spawning binaryPath "percona/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal/bin/mongod" to get version +1ms
  MongoMS:MongoMemoryServer Mongo[unknown]: _startUpInstance threw a Error:  TypeError: Cannot read properties of undefined (reading 'toString')
    at MongoBinary.getPath (woovi/node_modules/mongodb-memory-server-core/lib/util/MongoBinary.js:79:28)
    at async MongoInstance.start (woovi/node_modules/mongodb-memory-server-core/lib/util/MongoInstance.js:135:26)
    at async MongoInstance.create (woovi/node_modules/mongodb-memory-server-core/lib/util/MongoInstance.js:84:9)
    at async MongoMemoryServer._startUpInstance (woovi/node_modules/mongodb-memory-server-core/lib/MongoMemoryServer.js:223:26)
    at async MongoMemoryServer.start (woovi/node_modules/mongodb-memory-server-core/lib/MongoMemoryServer.js:84:9)
    at async MongoMemoryServer.create (woovi/node_modules/mongodb-memory-server-core/lib/MongoMemoryServer.js:63:9)
    at async getMMSMongoD (woovi/packages/testutils/src/config/getMMSMongoD.js:16:20)
    at async module.exports (woovi/packages/testutils/src/config/mongo/setup.js:20:20)
    at async woovi/node_modules/@jest/core/build/runGlobalHook.js:109:13
    at async waitForPromiseWithCleanup (woovi/node_modules/@jest/transform/build/ScriptTransformer.js:160:5) +3ms
  MongoMS:MongoMemoryServer Mongo[unknown]: stop: Called .stop() method +0ms
  MongoMS:MongoMemoryServer Mongo[unknown]: stop: "instanceInfo" is not defined (never ran?) +0ms
TypeError: Cannot read properties of undefined (reading 'toString')
    at MongoBinary.getPath (woovi/node_modules/mongodb-memory-server-core/lib/util/MongoBinary.js:79:28)
    at async MongoInstance.start (woovi/node_modules/mongodb-memory-server-core/lib/util/MongoInstance.js:135:26)
    at async MongoInstance.create (woovi/node_modules/mongodb-memory-server-core/lib/util/MongoInstance.js:84:9)
    at async MongoMemoryServer._startUpInstance (woovi/node_modules/mongodb-memory-server-core/lib/MongoMemoryServer.js:223:26)
    at async MongoMemoryServer.start (woovi/node_modules/mongodb-memory-server-core/lib/MongoMemoryServer.js:84:9)
    at async MongoMemoryServer.create (woovi/node_modules/mongodb-memory-server-core/lib/MongoMemoryServer.js:63:9)
    at async getMMSMongoD (woovi/packages/testutils/src/config/getMMSMongoD.js:16:20)
    at async module.exports (woovi/packages/testutils/src/config/mongo/setup.js:20:20)
    at async woovi/node_modules/@jest/core/build/runGlobalHook.js:109:13
    at async waitForPromiseWithCleanup (woovi/node_modules/@jest/transform/build/ScriptTransformer.js:160:5)

hasezoey · 2024-12-09T15:19:23Z

Thanks for the output, i think i should do more checking there, though from what i know this should never happen in nodejs (according to types):

mongodb-memory-server/packages/mongodb-memory-server-core/src/util/MongoBinary.ts

Lines 91 to 92 in a9d8986

    
           const spawnOutput = spawnSync(binaryPath, ['--version']) 
        
             .stdout.toString()

Does anything change if you provide a absolute path to the binary? (ie /path/to/somewhere/mongod)

sibelius · 2024-12-09T15:26:46Z

this is the absolute path, I only removed in the logs

hasezoey · 2024-12-09T15:28:59Z

this is the absolute path, I only removed in the logs

just to confirm the path you had set for SYSTEM_BINARY was starting with a /(absolute)? (ie /percona/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal/bin/mongod instead of percona/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal/bin/mongod)

sibelius · 2024-12-09T15:33:53Z

it was absolute

/Users/sibeliusseraphini/Dev/entria/woovi-projects/percona/percona-server-mongodb-7.0.15-9-x86_64.bookworm-minimal/bin/mongod

because it seemingly can be undefined, see #742#issuecomment-2528284865

hasezoey · 2024-12-09T15:47:14Z

Thanks, then it is very weird that nodejs seemingly did not throw a error and gave back what it shouldnt have.
In any case, i have now implemented a path absolutization for the SYSTEM_BINARY (in beta) and also a extra check that stdout is defined.

Could you manually try running the percona binary with argument --version? (if it works, please provide the output)
Additionally, you can try 10.1.3-beta.1 with config option SYSTEM_BINARY_VERSION_CHECK set to false to see if skipping the version check makes it work.

sibelius · 2024-12-09T16:10:16Z

You can download percona and give it a try

I will test more

LucVidal360 added the question label Feb 9, 2023

hasezoey added enhancement docs This issue/pr is about documentation and removed question labels Feb 10, 2023

github-actions bot added the released on @beta label Aug 15, 2023

hasezoey added a commit that referenced this issue Aug 22, 2023

feat(MongoMemoryServer): use storage engine "wiredTiger" if version i…

952609b

…s 7.0.0 or higher fixes #742

hasezoey closed this as completed in 9d94112 Oct 6, 2023

hasezoey added the released Pull Request released | Issue is fixed label Oct 6, 2023

hasezoey added a commit that referenced this issue Jul 17, 2024

docs: update landing page to be less misleading regarding "in-memory"

1d0f18b

required because mongodb 7.0 does not have the public in-memory engine anymore. re #742 (comment)

hasezoey added a commit that referenced this issue Nov 1, 2024

docs: update landing page to be less misleading regarding "in-memory"

bfb36bd

required because mongodb 7.0 does not have the public in-memory engine anymore. re #742 (comment)

sibelius mentioned this issue Dec 6, 2024

Can we use custom MongoDB variant like Percona Server for MongoDB? #701

Closed

hasezoey added a commit that referenced this issue Dec 9, 2024

fix(MongoBinary::getPath): check that "stdout" actually exists

613e670

because it seemingly can be undefined, see #742#issuecomment-2528284865

delfuego mentioned this issue Dec 10, 2024

MongoDB 7+ no longer support ephemeralForTest storage engine mikhail-angelov/mongo-unit#79

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The `ephemeralForTest` storage engine is reaching its end of life #742

The `ephemeralForTest` storage engine is reaching its end of life #742

LucVidal360 commented Feb 9, 2023 •

edited

Loading

hasezoey commented Feb 9, 2023

LucVidal360 commented Feb 10, 2023

hasezoey commented Feb 10, 2023

github-actions bot commented Aug 15, 2023

hasezoey commented Aug 15, 2023

alexbjorlig commented Aug 22, 2023

hasezoey commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

sibelius commented Oct 4, 2023

hasezoey commented Oct 4, 2023

sibelius commented Oct 6, 2023

hasezoey commented Oct 6, 2023

hasezoey commented Oct 6, 2023

clemp6r commented Jul 17, 2024 •

edited

Loading

hasezoey commented Jul 17, 2024 •

edited

Loading

Climax777 commented Dec 6, 2024

hasezoey commented Dec 6, 2024 •

edited

Loading

sibelius commented Dec 6, 2024

sibelius commented Dec 6, 2024

sibelius commented Dec 6, 2024

hasezoey commented Dec 7, 2024

sibelius commented Dec 7, 2024

hasezoey commented Dec 7, 2024 •

edited

Loading

sibelius commented Dec 9, 2024

hasezoey commented Dec 9, 2024

sibelius commented Dec 9, 2024

hasezoey commented Dec 9, 2024

sibelius commented Dec 9, 2024

hasezoey commented Dec 9, 2024

sibelius commented Dec 9, 2024

The ephemeralForTest storage engine is reaching its end of life #742

The ephemeralForTest storage engine is reaching its end of life #742

Comments

LucVidal360 commented Feb 9, 2023 • edited Loading

Versions

What is your question?

Context

Workaround

Question

hasezoey commented Feb 9, 2023

LucVidal360 commented Feb 10, 2023

hasezoey commented Feb 10, 2023

github-actions bot commented Aug 15, 2023

hasezoey commented Aug 15, 2023

alexbjorlig commented Aug 22, 2023

hasezoey commented Aug 22, 2023

github-actions bot commented Aug 22, 2023

sibelius commented Oct 4, 2023

hasezoey commented Oct 4, 2023

sibelius commented Oct 6, 2023

hasezoey commented Oct 6, 2023

hasezoey commented Oct 6, 2023

clemp6r commented Jul 17, 2024 • edited Loading

hasezoey commented Jul 17, 2024 • edited Loading

Climax777 commented Dec 6, 2024

hasezoey commented Dec 6, 2024 • edited Loading

sibelius commented Dec 6, 2024

sibelius commented Dec 6, 2024

sibelius commented Dec 6, 2024

hasezoey commented Dec 7, 2024

sibelius commented Dec 7, 2024

hasezoey commented Dec 7, 2024 • edited Loading

sibelius commented Dec 9, 2024

hasezoey commented Dec 9, 2024

sibelius commented Dec 9, 2024

hasezoey commented Dec 9, 2024

sibelius commented Dec 9, 2024

hasezoey commented Dec 9, 2024

sibelius commented Dec 9, 2024

The `ephemeralForTest` storage engine is reaching its end of life #742

The `ephemeralForTest` storage engine is reaching its end of life #742

LucVidal360 commented Feb 9, 2023 •

edited

Loading

clemp6r commented Jul 17, 2024 •

edited

Loading

hasezoey commented Jul 17, 2024 •

edited

Loading

hasezoey commented Dec 6, 2024 •

edited

Loading

hasezoey commented Dec 7, 2024 •

edited

Loading