support Dolt #2177

coffeegoddd · 2024-01-29T22:14:32Z

What does this PR do?

This PR adds support for Dolt, a version controlled relational database with Git-like versioning semantics.

netlify · 2024-01-29T22:14:37Z

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	`f07cfd2`
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-go/deploys/66191d9aa36bee00089185df
😎 Deploy Preview	https://deploy-preview-2177--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

mdelapenya · 2024-01-29T22:19:58Z

Hi @coffeegoddd thanks for opening this PR, much appreciated! I've one major concern regarding the module structure: there a few files that are autogenerated, so for that we offer a code generation tool to create the skafolding for the module: please visit https://golang.testcontainers.org/modules/#interested-in-adding-a-new-module to learn how to start with a new module. Once we are there, I think we can continue with the review 🙏

Cheers!

mdelapenya · 2024-01-29T23:22:15Z

@coffeegoddd looks much better now! I'm letting the CI run now. I did not know about DoltDB, so I'm really ecstatic seen a new database module for a technology with +16k stars on Github. Could you tell me a bit more about the use case for this module, is it for the project itself to start using testcontainers-go? for the community to build apps with DoltDB? Both?

modules/dolt/dolt.go

coffeegoddd · 2024-01-29T23:58:34Z

@mdelapenya thanks for the help with this PR!

I wanted to add this module for Dolt to support an upcoming blog post I'm writing about working with Dolt and testcontainers.

We had a customer reach out to us wanting a blog demonstrating how Dolt can be used with testcontainers that specifically show-cases Dolt's Git-like functionality that make it really compelling as a testing database, since it makes it easy to setup state, create branches to test against, and even reset changes with dolt reset, similar to git reset.

Its possible we start using testcontainers internally, I hadn't encountered it before the customer mentioned it to our team, but it definitely makes life much easier for integration tests. Hopefully the community finds this useful too.

The customer originally asked for a Java Spring Boot + Dolt + testcontainers project blog, but I don't currently know Java so thought I'd start with Go 🤠 .

If this works well, we also have DoltgreSQL under development so I can add a module for that too.

coffeegoddd · 2024-01-30T23:41:27Z

@mdelapenya when you get a chance get you let me know what else needs to happen to get this PR landed? much appreciated 🤠

coffeegoddd · 2024-01-31T00:45:12Z

demonstration repo https://github.com/dolthub/testcontainers-go-demo

muescha · 2024-01-31T03:03:37Z

docs/modules/dolt.md

+the remote to clone, and use `WithDoltCredsPublicKey(key string)` along with `WithCredsFile(credsFile string)` to authorize the Dolt container to clone from the remote.
+
+<!--codeinclude-->
+[Example of Init script](../../modules/dolt/testdata/schema.sql)


I expect here a clone script like this script:
https://github.com/dolthub/testcontainers-go-demo/blob/main/testdata/clone-db.sh

modules/dolt/dolt.go

mdelapenya

One final question before merging this PR, that I should've probably mentioned earlier in the discussion.

Would you like to host this module in a repo from Dolt's organisation, and contribute it to the modules catalog as a community module?

modules/dolt/examples_test.go

coffeegoddd · 2024-02-06T17:32:04Z

here is the blog post I wrote for this work: https://www.dolthub.com/blog/2024-02-05-testcontainers-for-go-with-dolt/

mdelapenya · 2024-02-14T19:37:12Z

One final question before merging this PR, that I should've probably mentioned earlier in the discussion.

Would you like to host this module in a repo from Dolt's organisation, and contribute it to the modules catalog as a community module?

Hey @coffeegoddd did you have time to review my previous message? I'd prefer to have this module hosted in your organisation, so that you totally control it and evolve it using testcontainers-go as an upstream project. If you have any doubt about this, we can proceed and merge this PR into our repo, although the cadence of updates will be slower.

mdelapenya · 2024-02-22T17:01:09Z

@coffeegoddd we have just added a section in the modules docs trying to explain where to host a testcontainers-go module: https://golang.testcontainers.org/modules/#before-creating-a-new-module.

In this particular case, you being part of the vendor, I'd recommend you to host the module under your org having us linking it in our modules catalog.

Thoughts?

* main: (115 commits) chore: create TLS certs in a consistent manner (testcontainers#2478) chore(deps): bump idna from 3.6 to 3.7 (testcontainers#2480) Elasticsearch disable CA retrieval when ssl is disabled (testcontainers#2475) fix: handle dockerignore exclusions properly (testcontainers#2476) chore: prepare for next minor development cycle (0.31.0) chore: use new version (v0.30.0) in modules and examples Fix url creation to handle query params when using HTTP wait strategy (testcontainers#2466) fix: data race on container run (testcontainers#2345) fix: logging deadlock (testcontainers#2346) feat(k6):Add remote test scripts (testcontainers#2350) feat: optimizes file copies to and from containers (testcontainers#2450) fix(exec): updates the `Multiplexed` opt to combine stdout and stderr (testcontainers#2452) Upgrade neo4j module to use features from v0.29.1 of testcontainers-go (testcontainers#2463) bug:Fix AMQPS url (testcontainers#2462) chore: more compose updates in comments chore: use "docker compose" (v2) instead of "docker-compose" (v1) (testcontainers#2464) chore(deps): bump github/codeql-action from 2.22.12 to 3.24.9 (testcontainers#2459) refactor: Add Weaviate modules tests (testcontainers#2447) feat(exitcode): Add exit code sugar method (testcontainers#2342) feat: add module to support InfluxDB v1.x (testcontainers#1703) ...

mdelapenya · 2024-04-12T11:41:29Z

@coffeegoddd I went ahead and updated your PR with the latest state of the code. We are going to integrate the Dolt module into the repo, as it's better to have it at some place!

Hope the changes on top of yours are evident, but please let me know if you need me to clarify anything. Thanks!

mdelapenya

LGTM thanks!

* main: support Dolt (testcontainers#2177) chore: create TLS certs in a consistent manner (testcontainers#2478) chore(deps): bump idna from 3.6 to 3.7 (testcontainers#2480)

* /modules/dolt: wip, kinda working * /modules/dolt: get tests passing * /{.github,.vscode,docs,mkdocs,modules,sonar-project}: use modulegen tool * /modules/dolt/{dolt.go,examples_test.go}: run linter * /modules/dolt/{dolt.go,examples_test.go}: add methods for cloning * /{docs, modules}: add with creds file * /{docs,modules}: pr feedback, cleanup * /modules/dolt/examples_test.go: remove panics, lint * chore: run mod tidy * chore: include MustConnectionString method * chore: do not use named returns * chore: perform initialisation before the container has started --------- Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* main: fix: don't retry on permanent APIClient errors (testcontainers#2506) feat: support overriding the default recreate options for compose (testcontainers#2511) feat: support passing io.Reader for compose files when creating a compose instance (testcontainers#2509) chore: add funding button for testcontainers (testcontainers#2510) feat: support Ryuk for the compose module (testcontainers#2485) chore(deps): bump golang.org/x/net in modules (minio, gcloud, weaviate, compose, qdrant, couchbase, k3s, milvus, mockserver, pulsar, kafka) (testcontainers#2505) fix: fallback to URL-path when parsing auth config URL without scheme (testcontainers#2488) fix(postgres): Fix the non-default dbname error (testcontainers#2489) feat: Bump default postgres version (testcontainers#2481) support Dolt (testcontainers#2177) chore: create TLS certs in a consistent manner (testcontainers#2478) chore(deps): bump idna from 3.6 to 3.7 (testcontainers#2480) Elasticsearch disable CA retrieval when ssl is disabled (testcontainers#2475) fix: handle dockerignore exclusions properly (testcontainers#2476)

* chore: start a foundational package for interacting with Docker networks * feat: add an SSH tunnel forwarding a host port to a container * fix: rename struct * chore: pass the original context to the exposeHostPorts function * chore: start tunnel using context * chore: push goroutines to the method where they are used * fix: proper eval of first network * fix: handle dockerignore exclusions properly (#2476) * chore: only include the dockerignore if it contains ignore files * fix: the inclusions must be relative to the context * docs: document the dockerignore feature * chore: only include the dockerignore file if it exists * Elasticsearch disable CA retrieval when ssl is disabled (#2475) * skip search for CACert if ssl has been turned off * add tests with and without ssl enabled * add all config keys that disable CA gen, restrict check to version 8 * rename test to match content * chore(deps): bump idna from 3.6 to 3.7 (#2480) Bumps [idna](https://github.com/kjd/idna) from 3.6 to 3.7. - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst) - [Commits](kjd/idna@v3.6...v3.7) --- updated-dependencies: - dependency-name: idna dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: create TLS certs in a consistent manner (#2478) * fix: remove suspicious filepath.Join * chore: fix lint * fix: handle error * chore: reverse assertion for lint * feat: support generating TLS certificates on the fly * chore: apply to cockroachdb * chore: support saving the cert and priv key files to disk * chore: apply to rabbitmq * chore: simplify * chore: use in redpanda module * chore: lint * chore: set validFrom internally * fix: properly use the new API in redpanda * docs: document the TLS helpers * chore: simplify WithParent to accept the struct directly * chore: use tlscert package instead * fix: use non-deprecated API * docs: update * docs: fix examples * chore: use released version of tlscert * fix: add common name for the node cert * support Dolt (#2177) * /modules/dolt: wip, kinda working * /modules/dolt: get tests passing * /{.github,.vscode,docs,mkdocs,modules,sonar-project}: use modulegen tool * /modules/dolt/{dolt.go,examples_test.go}: run linter * /modules/dolt/{dolt.go,examples_test.go}: add methods for cloning * /{docs, modules}: add with creds file * /{docs,modules}: pr feedback, cleanup * /modules/dolt/examples_test.go: remove panics, lint * chore: run mod tidy * chore: include MustConnectionString method * chore: do not use named returns * chore: perform initialisation before the container has started --------- Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> * feat: Bump default postgres version (#2481) * Bump default postgres version * Bump to use latest pg * Bump version from non-ancient version --------- Co-authored-by: bstrausser <bstrausser@locusrobotics.com> * fix(postgres): Fix the non-default dbname error (#2489) * Fix the non-default dbname error The linked issue described in great detail an issue where we assumed everyone would use the default database user, whose home DB defaults to the postgres database. When that was not the case, the snapshots would fail silently as the user would not connect to the right database to take the commands. This PR fixes the issue by adding the dbname by default in the command, and adds a test to validate this works as intended. In addition, it also adds some logic to handle any error that does not cause the exec command to fail, such as database access failures. Run the added test to test this works as intended. Closes #2474 * Document the postgres dbname issue in the docs * fix: fallback to URL-path when parsing auth config URL without scheme (#2488) * chore(deps): bump golang.org/x/net in modules (minio, gcloud, weaviate, compose, qdrant, couchbase, k3s, milvus, mockserver, pulsar, kafka) (#2505) * chore(deps): bump golang.org/x/net in /modules/kafka Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/pulsar Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/mockserver Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/milvus Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net from 0.19.0 to 0.23.0 in /modules/k3s Bumps [golang.org/x/net](https://github.com/golang/net) from 0.19.0 to 0.23.0. - [Commits](golang/net@v0.19.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/couchbase Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/qdrant Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/compose Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/weaviate Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to 0.23.0. - [Commits](golang/net@v0.20.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/gcloud Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0. - [Commits](golang/net@v0.21.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump golang.org/x/net in /modules/minio Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0. - [Commits](golang/net@v0.21.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: support Ryuk for the compose module (#2485) * feat: add testcontainers labels to compose containers * feat: support reaper for compose * chore: increase ryuk reconnection timeout on CI * chore: cache containers on UP * chore: more tuning for compose * chore: more consistent assertion * chore: the compose stack asks for the reaper, but each container then connects to it * chore: use different error groups the first time wait is called, the context is cancelled * chore: the lookup method include cache checks * chore: update tests to make them deterministic * chore: rename local compose testss * chore: support returning the dynamic port in the helper function * chore: try with default reconnection timeout * feat: support removing networks from compose * chore: support naming test services with local and api It will allow the tests to be more deterministic, as there could be service containers started from the local test suite with the same name as in the API test suite. * Revert "chore: try with default reconnection timeout" This reverts commit 336760c. * fix: typo * chore: add funding button for testcontainers (#2510) * feat: support passing io.Reader for compose files when creating a compose instance (#2509) * feat: support passing io.Reader when creating a compose instance * docs: change title * feat: support overriding the default recreate options for compose (#2511) * feat: support overriding the default recreate options for compose * chore: validate recreation values * fix: don't retry on permanent APIClient errors (#2506) * fix: don't retry on permanent APIClient errors * fix: add more tests for un-retryable scenarios * chore: run mod tidy * chore: implement the port-forwarding correctly * chore: use new sshd image * chore: simplify channel creation to avoid allocations * fix: do not leak goroutines Detected with go.uber.org/goleak * chore: expose host internal constant * fix: update variables --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Laurent Saint-Félix <laurent.saintfelix@elastic.co> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dustin Brown <dustin@dolthub.com> Co-authored-by: Barrett Strausser <bearrito@users.noreply.github.com> Co-authored-by: bstrausser <bstrausser@locusrobotics.com> Co-authored-by: Guillaume St-Pierre <guillaume.stpierre@docker.com> Co-authored-by: Patrick Jahn <33724206+p-jahn@users.noreply.github.com>

* main: (22 commits) feat: forward host ports to a container using an SSH tunnel (testcontainers#2471) Update follow_logs.md with adding missing package (testcontainers#2513) fix: don't retry on permanent APIClient errors (testcontainers#2506) feat: support overriding the default recreate options for compose (testcontainers#2511) feat: support passing io.Reader for compose files when creating a compose instance (testcontainers#2509) chore: add funding button for testcontainers (testcontainers#2510) feat: support Ryuk for the compose module (testcontainers#2485) chore(deps): bump golang.org/x/net in modules (minio, gcloud, weaviate, compose, qdrant, couchbase, k3s, milvus, mockserver, pulsar, kafka) (testcontainers#2505) fix: fallback to URL-path when parsing auth config URL without scheme (testcontainers#2488) fix(postgres): Fix the non-default dbname error (testcontainers#2489) feat: Bump default postgres version (testcontainers#2481) support Dolt (testcontainers#2177) chore: create TLS certs in a consistent manner (testcontainers#2478) chore(deps): bump idna from 3.6 to 3.7 (testcontainers#2480) Elasticsearch disable CA retrieval when ssl is disabled (testcontainers#2475) fix: handle dockerignore exclusions properly (testcontainers#2476) chore: prepare for next minor development cycle (0.31.0) chore: use new version (v0.30.0) in modules and examples Fix url creation to handle query params when using HTTP wait strategy (testcontainers#2466) fix: data race on container run (testcontainers#2345) ...

* main: (34 commits) break: return error from Customize request option (testcontainers#2267) fix: wrong copy paste (testcontainers#2515) docs: add documentation for Exec method (testcontainers#2451) docs: document the SSHd tunnel (testcontainers#2514) fix: enhance host configuration port binding (testcontainers#2512) feat: forward host ports to a container using an SSH tunnel (testcontainers#2471) Update follow_logs.md with adding missing package (testcontainers#2513) fix: don't retry on permanent APIClient errors (testcontainers#2506) feat: support overriding the default recreate options for compose (testcontainers#2511) feat: support passing io.Reader for compose files when creating a compose instance (testcontainers#2509) chore: add funding button for testcontainers (testcontainers#2510) feat: support Ryuk for the compose module (testcontainers#2485) chore(deps): bump golang.org/x/net in modules (minio, gcloud, weaviate, compose, qdrant, couchbase, k3s, milvus, mockserver, pulsar, kafka) (testcontainers#2505) fix: fallback to URL-path when parsing auth config URL without scheme (testcontainers#2488) fix(postgres): Fix the non-default dbname error (testcontainers#2489) feat: Bump default postgres version (testcontainers#2481) support Dolt (testcontainers#2177) chore: create TLS certs in a consistent manner (testcontainers#2478) chore(deps): bump idna from 3.6 to 3.7 (testcontainers#2480) Elasticsearch disable CA retrieval when ssl is disabled (testcontainers#2475) ...

coffeegoddd added 2 commits January 29, 2024 13:34

/modules/dolt: wip, kinda working

493e490

/modules/dolt: get tests passing

9555867

coffeegoddd requested a review from a team as a code owner January 29, 2024 22:14

coffeegoddd changed the title ~~Db/dolt~~ support Dolt Jan 29, 2024

/{.github,.vscode,docs,mkdocs,modules,sonar-project}: use modulegen tool

963ab3d

mdelapenya reviewed Jan 29, 2024

View reviewed changes

modules/dolt/dolt.go Show resolved Hide resolved

/modules/dolt/{dolt.go,examples_test.go}: run linter

606c5aa

/modules/dolt/{dolt.go,examples_test.go}: add methods for cloning

83a99d4

coffeegoddd force-pushed the db/dolt branch from 918e153 to 83a99d4 Compare January 30, 2024 22:30

coffeegoddd added 2 commits January 30, 2024 14:30

Merge branch 'main' into db/dolt

ed7539f

/{docs, modules}: add with creds file

f948998

muescha reviewed Jan 31, 2024

View reviewed changes

mdelapenya reviewed Jan 31, 2024

View reviewed changes

modules/dolt/dolt.go Show resolved Hide resolved

/{docs,modules}: pr feedback, cleanup

c1fa127

mdelapenya reviewed Feb 1, 2024

View reviewed changes

mdelapenya reviewed Feb 2, 2024

View reviewed changes

modules/dolt/examples_test.go Outdated Show resolved Hide resolved

/modules/dolt/examples_test.go: remove panics, lint

09c4f55

mdelapenya added 4 commits April 12, 2024 13:19

chore: run mod tidy

81d294a

chore: include MustConnectionString method

67fd4bf

chore: do not use named returns

5d56b38

chore: perform initialisation before the container has started

f07cfd2

mdelapenya self-assigned this Apr 12, 2024

mdelapenya added the enhancement New feature or request label Apr 12, 2024

mdelapenya approved these changes Apr 12, 2024

View reviewed changes

mdelapenya merged commit b5f1e13 into testcontainers:main Apr 12, 2024
104 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support Dolt #2177

support Dolt #2177

coffeegoddd commented Jan 29, 2024

netlify bot commented Jan 29, 2024 •

edited

Loading

mdelapenya commented Jan 29, 2024

mdelapenya commented Jan 29, 2024

coffeegoddd commented Jan 29, 2024

coffeegoddd commented Jan 30, 2024

coffeegoddd commented Jan 31, 2024

muescha Jan 31, 2024

mdelapenya left a comment

coffeegoddd commented Feb 6, 2024

mdelapenya commented Feb 14, 2024

mdelapenya commented Feb 22, 2024

mdelapenya commented Apr 12, 2024

mdelapenya left a comment

support Dolt #2177

support Dolt #2177

Conversation

coffeegoddd commented Jan 29, 2024

What does this PR do?

netlify bot commented Jan 29, 2024 • edited Loading

✅ Deploy Preview for testcontainers-go ready!

mdelapenya commented Jan 29, 2024

mdelapenya commented Jan 29, 2024

coffeegoddd commented Jan 29, 2024

coffeegoddd commented Jan 30, 2024

coffeegoddd commented Jan 31, 2024

muescha Jan 31, 2024

Choose a reason for hiding this comment

mdelapenya left a comment

Choose a reason for hiding this comment

coffeegoddd commented Feb 6, 2024

mdelapenya commented Feb 14, 2024

mdelapenya commented Feb 22, 2024

mdelapenya commented Apr 12, 2024

mdelapenya left a comment

Choose a reason for hiding this comment

netlify bot commented Jan 29, 2024 •

edited

Loading