Add MongoDB Atlas Local Testcontainer #8760

luketn · 2024-06-09T01:52:16Z

This adds a new Testcontainer class for MongoDB's Atlas Local container to the existing databases/mongodb module.

The benefits of having this as a Testcontainer (rather than an example for a GenericContainer):

solves the startup time issue using a custom wait strategy based on the runner healthcheck command (see below)
provides the connection string for MongoDB

I've tried to follow all the conventions and style for contributions. I've added unit tests and documentation.
* I might have been a bit verbose in my documentation contribution - happy to trim that down / cut it down

Background

MongoDB Atlas Local combines the MongoDB database engine with MongoT, a sidecar process for advanced searching capabilities built by MongoDB and powered by Apache Lucene.

It allows you to use the following features:

MongoDB Atlas Search: Atlas Search gives MongoDB queries access to the incredible search toolbox that is Lucene. The main use-case is advanced lexical text querying capabilities similar to those found in many search engines. In addition, Atlas Search supports queries with faceting and parallel index search. These can extend MongoDB's aggregation capabilities and performance for uses like statistics and complex filters.
https://www.mongodb.com/docs/atlas/atlas-search/
MongoDB Atlas Vector Search: Supports artificial intelligence (AI) based searches for semantically similar items in your data. Vector indexes store embeddings (high-dimensional vectors encoding semantic meaning) used in large language models (LLMs). This feature makes use of Lucene's vector search capabilities to find the nearness between the values of each vector. This can be a powerful alternative or compliment to lexical text search.
https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/
Pairing these Lucene backed technologies with your MongoDB database allows you to build powerful search capabilities into your applications without the need to manage a separate search engine. You can also extend your search capabilities to include AI based vector searches, which can be useful for recommendation engines, image search, and other applications that require similarity searches.

The container (mongodb/mongodb-atlas-local) documentation can be found here: https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/

Container Healthcheck

You cannot start calling Atlas Search commands, such as creating Atlas Search indexes, until the container is ready. The container takes some seconds to attain readiness, whilst:

MongoDB database starts
MongoDB initialises itself as a replica set
MongoT starts and connects to the MongoDB database ready to follow Change Streams for indexing
MongoDB connects to MongoT ready to perform $search and $vectorSearch queries

The MongoDBAtlasLocalContainer uses the container's runner healthcheck command to check for readiness.

…te that it works correctly

…w MongoDBAtlasLocalContainer and how to use it in tests with Testcontainers.

…required but appears no longer)

…roject

eddumelendez

Thanks for your contribution, @luketn ! Glad to see the PR with less more complexity than the initial implementation. I've left some comments. Also, related to the docs I'd say we should focus on what the Testcontainers implementation offers. Of course, if there is more we can do with it and the mongodb atlas container then we should document it.

eddumelendez · 2024-06-25T22:28:47Z

modules/mongodb/src/test/java/org/testcontainers/containers/MongoDBAtlasLocalContainerTest.java

+
+import java.time.Instant;
+
+import static org.junit.Assert.*;


Let's use assertj instead.

modules/mongodb/src/main/java/org/testcontainers/containers/MongoDBAtlasLocalContainer.java

eddumelendez · 2024-06-25T22:29:29Z

modules/mongodb/src/main/java/org/testcontainers/containers/MongoDBAtlasLocalContainer.java

@@ -0,0 +1,43 @@
+package org.testcontainers.containers;


Suggested change

package org.testcontainers.containers;

package org.testcontainers.mongodb;

@eddumelendez I'm concerned that this would be a breaking change for existing users of the MongoDBContainer (assuming we would move both classes to this new package). Perhaps we should leave both in the current package unless making a breaking change to move both to the new package?

Let's only move MongoDBAtlasLocalContainer, since it is a new implementation there is no issues.

I tried doing that, but the container uses a custom ContainerDef extension to allow it to override the wait strategy. This class is package local, so requires that we remain in the org.testcontainers.containers package.

The use of a custom container def will be important for future changes too, since we will want to add custom with... statements for atlas search indexes and seed data once the container supports that (assuming MongoDB agree and add the necessary dependencies to the container).

eddumelendez · 2024-06-25T22:31:02Z

modules/mongodb/src/main/java/org/testcontainers/containers/MongoDBAtlasLocalContainer.java

+import org.testcontainers.containers.wait.strategy.Wait;
+import org.testcontainers.utility.DockerImageName;
+
+public class MongoDBAtlasLocalContainer extends GenericContainer<MongoDBAtlasLocalContainer> {


wonder if can be renamed to MongoDbAtlasContainer instead. WDYT?

My feeling is Atlas Local is the more consistent name with MongoDB's own documentation and naming conventions:
https://hub.docker.com/r/mongodb/mongodb-atlas-local
https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/

eddumelendez · 2024-06-25T22:32:08Z

docs/modules/databases/mongodb.md

+### General info
+MongoDB Atlas Local combines the MongoDB database engine with MongoT, a sidecar process for advanced searching capabilities built by MongoDB and powered by [Apache Lucene](https://lucene.apache.org/). 
+
+It allows you to use the following features:
+
+* MongoDB Atlas Search: Atlas Search gives MongoDB queries access to the incredible search toolbox that is Lucene. The main use-case is advanced lexical text querying capabilities similar to those found in many search engines. In addition, Atlas Search supports queries with faceting and parallel index search. These can extend MongoDB's aggregation capabilities and performance for uses like statistics and complex filters.  
+  [https://www.mongodb.com/docs/atlas/atlas-search/](https://www.mongodb.com/docs/atlas/atlas-search/)
+* MongoDB Atlas Vector Search: Supports artificial intelligence (AI) based searches for semantically similar items in your data. Vector indexes store embeddings (high-dimensional vectors encoding semantic meaning) used in large language models (LLMs). This feature makes use of Lucene's [vector search](https://lucene.apache.org/core/9_10_0/core/org/apache/lucene/search/KnnVectorQuery.html) capabilities to find the nearness between the values of each vector. This can be a powerful alternative or compliment to lexical text search.  
+  [https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/)
+
+Pairing these Lucene backed technologies with your MongoDB database allows you to build powerful search capabilities into your applications without the need to manage a separate search engine. You can also extend your search capabilities to include AI based vector searches, which can be useful for recommendation engines, image search, and other applications that require similarity searches.
+
+The container (mongodb/mongodb-atlas-local) documentation can be found here:
+[https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/](https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/)


I think we can add a link to MongoDB Atlas documentation instead of adding product related information in our docs.

@eddumelendez should we remove the 'incubating' statement from the top of this docs page? It's been there a long while!

Is it still incubating?

let's remove it :)

eddumelendez · 2024-06-25T22:34:59Z

modules/mongodb/src/test/java/org/testcontainers/containers/AtlasLocalDataAccess.java

+                    }
+                }
+            }
+            sleep(50);


We can use awaitility here.

I can not see the awaitility usage here :)

eddumelendez · 2024-06-25T22:39:54Z

docs/modules/databases/mongodb.md

+
+

Suggested change

eddumelendez · 2024-06-25T22:46:16Z

modules/mongodb/src/test/java/org/testcontainers/containers/AtlasLocalDataAccess.java

+    }
+
+    // initAtlasSearchIndex {
+    public void initAtlasSearchIndex() throws URISyntaxException, IOException, InterruptedException {


Currently, databases provide some default values for username, password and databases. What if we do the same for MongoDB Atlas implementation providing default values for database, collection name that can be customized? Also, a withIndex(Transferable json) copies the mapping json file from classpath to the container and execute the right commands in the container. No need to do so right away but just thinking out loud.

would be also nice to see how we can configure credentials support. I see langchain4j using it https://github.com/langchain4j/langchain4j/blob/main/langchain4j-mongodb-atlas/src/test/java/dev/langchain4j/store/embedding/mongodb/MongoDbEmbeddingStoreLocalIT.java#L55-L61

I've spoken to MongoDB and requested that they add the MongoDB shell mongosh, as well as the database tools mongoimport, mongoexport, mongodump... which would allow us to create these nice extra features. If they add these to the image, I will add these features as a further PR.

ok, nice! any public place where we can follow the request?

I'm very interested on this because would like to improve the time to execute test in langchain4j and spring-ai integrations.

I shared this request with the product team directly, but it's a great idea to have a ticket - I created one and will share with the product team too:
https://feedback.mongodb.com/forums/924868-atlas-search/suggestions/48857015-include-mongodb-shell-mongosh-and-mongodb-tools

Co-authored-by: Eddú Meléndez Gonzales <eddu.melendez@gmail.com>

…rs require it)

eddumelendez · 2024-09-18T18:15:54Z

docs/modules/databases/mongodb.md

-## Usage example
+The MongoDB module provides two Testcontainers for MongoDB unit testing:
+
+* [MongoDBContainer](#mongodbcontainer) - the core MongoDB database


Suggested change

* [MongoDBContainer](#mongodbcontainer) - the core MongoDB database

* [MongoDBContainer](#mongodbcontainer) - the core MongoDB database with ReplicaSet enabled

Can we discuss this one? (booked a session next week)
I actually think the current MongoDB test container could use a lot of work (including some updates to its docs).
I thought about including some of these changes, but perhaps we could do that in a separate PR?
In this case, I think we should really make replica set configuration optional.

Can we discuss this one?

Sure

I thought about including some of these changes, but perhaps we could do that in a separate PR?

Yes, let's discuss it first and then a separate PR is the way to go

eddumelendez · 2024-09-18T18:18:43Z

modules/mongodb/src/test/java/org/testcontainers/containers/MongoDBAtlasLocalContainerTest.java

+        try (
+            // creatingAtlasLocalContainer {
+            MongoDBAtlasLocalContainer atlasLocalContainer = new MongoDBAtlasLocalContainer(
+                MongoDBAtlasLocalContainer.DEFAULT_IMAGE_NAME.withTag("7.0.9")


for documentations, let's use the string raw image

eddumelendez · 2024-09-18T18:19:31Z

modules/mongodb/src/test/java/org/testcontainers/containers/MongoDBAtlasLocalContainerTest.java

+                // writeAndReadBack {
+                atlasLocalDataAccess.insertData(new AtlasLocalDataAccess.TestData("tests", 123, true));
+
+                //Wait for Atlas Search to index the data (Atlas Search is eventually consistent)
+                await()
+                    .atMost(5, TimeUnit.SECONDS)
+                    .pollInterval(10, TimeUnit.MILLISECONDS)
+                    .pollInSameThread()
+                    .until(
+                        () -> atlasLocalDataAccess.findAtlasSearch("test"),
+                        Objects::nonNull);
+                // }


AtlasLocalDataAccess.TestData is an class used only for testing. So, let's avoid showing it in the docs

done - actually I already removed from the docs, I've now also removed the docs comments here too

eddumelendez · 2024-09-18T18:21:00Z

docs/modules/databases/mongodb.md

+The container (mongodb/mongodb-atlas-local) documentation can be found here:
+[https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/](https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/)


Suggested change

The container (mongodb/mongodb-atlas-local) documentation can be found here:

[https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/](https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/)

The container (mongodb/mongodb-atlas-local) documentation can be found [here](https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/)

eddumelendez · 2024-09-18T18:21:35Z

docs/modules/databases/mongodb.md

+General information about Atlas Search can be found here:  
+[https://www.mongodb.com/docs/atlas/atlas-search/](https://www.mongodb.com/docs/atlas/atlas-search/)


Suggested change

General information about Atlas Search can be found here:

[https://www.mongodb.com/docs/atlas/atlas-search/](https://www.mongodb.com/docs/atlas/atlas-search/)

General information about Atlas Search can be found [here](https://www.mongodb.com/docs/atlas/atlas-search/).

…consistent with approach used in other containers

luketn added 7 commits June 7, 2024 21:29

Implement a MongoDB Atlas Local test container + unit tests to valida…

0404cce

…te that it works correctly

Add to the mongodb module documentation page information about the ne…

fe7fed8

…w MongoDBAtlasLocalContainer and how to use it in tests with Testcontainers.

Remove redundant close override

4ec8945

update the connection string not to use directConnection (used to be …

37b918a

…required but appears no longer)

update the connection string not to use directConnection (used to be …

39aaf60

…required but appears no longer)

run ./gradlew spotlessApply to apply code style conventions for the p…

cd83f51

…roject

follow project guidelines for container names and tags

48f9bc0

luketn requested a review from a team as a code owner June 9, 2024 01:52

github-actions bot added type/docs modules/mongodb labels Jun 9, 2024

eddumelendez requested changes Jun 25, 2024

View reviewed changes

luketn and others added 8 commits September 15, 2024 16:20

Merge branch 'testcontainers:main' into main

f9b58d5

switch to assertJ to align with project best practice (PR 8760 feedback)

b8354b7

Accept change suggestion to make public static internal to package

e424012

Co-authored-by: Eddú Meléndez Gonzales <eddu.melendez@gmail.com>

simplify documentation for Atlas Local container (PR 8760 feedback)

85b304f

add in the directConnection flag to the connection string (some drive…

90cbb96

…rs require it)

Merge remote-tracking branch 'origin/main'

9ffe93b

use awaitility instead of custom loop (PR 8760 feedback)

c04d176

fix the link to atlas docs (was not being shown as a hyperlink)

19b6dc0

eddumelendez reviewed Sep 18, 2024

View reviewed changes

luketn added 5 commits September 19, 2024 12:26

remove overly detailed explanation of health check

760c2d4

make default image name private, and use string constructor in docs, …

887ae6d

…consistent with approach used in other containers

remove documentation comments from unused docs example

f39ccbc

make links inline with text

ca7bc31

use awaitility in test example data access class

a1daab8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MongoDB Atlas Local Testcontainer #8760

Add MongoDB Atlas Local Testcontainer #8760

luketn commented Jun 9, 2024

eddumelendez left a comment

eddumelendez Jun 25, 2024

luketn Sep 15, 2024

eddumelendez Jun 25, 2024

luketn Sep 15, 2024

eddumelendez Sep 18, 2024

luketn Sep 19, 2024

eddumelendez Jun 25, 2024

luketn Sep 15, 2024

eddumelendez Jun 25, 2024

luketn Sep 15, 2024

luketn Sep 15, 2024

eddumelendez Sep 18, 2024

eddumelendez Jun 25, 2024

luketn Sep 15, 2024

eddumelendez Sep 18, 2024

eddumelendez Jun 25, 2024

luketn Sep 15, 2024

eddumelendez Jun 25, 2024

eddumelendez Jun 25, 2024

luketn Sep 15, 2024

eddumelendez Sep 18, 2024

luketn Sep 19, 2024

eddumelendez Sep 18, 2024

luketn Sep 19, 2024 •

edited

Loading

eddumelendez Sep 19, 2024

eddumelendez Sep 18, 2024

eddumelendez Sep 18, 2024

luketn Sep 19, 2024

eddumelendez Sep 18, 2024

luketn Sep 19, 2024

eddumelendez Sep 18, 2024

luketn Sep 19, 2024

	package org.testcontainers.containers;
	package org.testcontainers.mongodb;

	* [MongoDBContainer](#mongodbcontainer) - the core MongoDB database
	* [MongoDBContainer](#mongodbcontainer) - the core MongoDB database with ReplicaSet enabled

		The container (mongodb/mongodb-atlas-local) documentation can be found here:
		[https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/](https://www.mongodb.com/docs/atlas/cli/current/atlas-cli-deploy-docker/)

		General information about Atlas Search can be found here:
		[https://www.mongodb.com/docs/atlas/atlas-search/](https://www.mongodb.com/docs/atlas/atlas-search/)

Add MongoDB Atlas Local Testcontainer #8760

Are you sure you want to change the base?

Add MongoDB Atlas Local Testcontainer #8760

Conversation

luketn commented Jun 9, 2024

Background

Container Healthcheck

eddumelendez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luketn Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luketn Sep 19, 2024 •

edited

Loading