Skip to content

Commit

Permalink
support Dolt (#2177)
Browse files Browse the repository at this point in the history
* /modules/dolt: wip, kinda working

* /modules/dolt: get tests passing

* /{.github,.vscode,docs,mkdocs,modules,sonar-project}: use modulegen tool

* /modules/dolt/{dolt.go,examples_test.go}: run linter

* /modules/dolt/{dolt.go,examples_test.go}: add methods for cloning

* /{docs, modules}: add with creds file

* /{docs,modules}: pr feedback, cleanup

* /modules/dolt/examples_test.go: remove panics, lint

* chore: run mod tidy

* chore: include MustConnectionString method

* chore: do not use named returns

* chore: perform initialisation before the container has started

---------

Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
  • Loading branch information
coffeegoddd and mdelapenya authored Apr 12, 2024
1 parent 0a268b3 commit b5f1e13
Show file tree
Hide file tree
Showing 16 changed files with 980 additions and 2 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ jobs:
matrix:
go-version: [1.21.x, 1.x]
platform: [ubuntu-latest]
module: [artemis, cassandra, chroma, clickhouse, cockroachdb, compose, consul, couchbase, elasticsearch, gcloud, inbucket, influxdb, k3s, k6, kafka, localstack, mariadb, milvus, minio, mockserver, mongodb, mssql, mysql, nats, neo4j, ollama, openfga, openldap, opensearch, postgres, pulsar, qdrant, rabbitmq, redis, redpanda, registry, surrealdb, vault, weaviate]
module: [artemis, cassandra, chroma, clickhouse, cockroachdb, compose, consul, couchbase, dolt, elasticsearch, gcloud, inbucket, influxdb, k3s, k6, kafka, localstack, mariadb, milvus, minio, mockserver, mongodb, mssql, mysql, nats, neo4j, ollama, openfga, openldap, opensearch, postgres, pulsar, qdrant, rabbitmq, redis, redpanda, registry, surrealdb, vault, weaviate]
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: ${{ matrix.go-version }}
Expand Down
4 changes: 4 additions & 0 deletions .vscode/.testcontainers-go.code-workspace
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@
"name": "module / couchbase",
"path": "../modules/couchbase"
},
{
"name": "module / dolt",
"path": "../modules/dolt"
},
{
"name": "module / elasticsearch",
"path": "../modules/elasticsearch"
Expand Down
81 changes: 81 additions & 0 deletions docs/modules/dolt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Dolt

Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

## Introduction

The Testcontainers module for Dolt.

## Adding this module to your project dependencies

Please run the following command to add the Dolt module to your Go dependencies:

```
go get github.com/testcontainers/testcontainers-go/modules/dolt
```

## Usage example

<!--codeinclude-->
[Creating a Dolt container](../../modules/dolt/examples_test.go) inside_block:runDoltContainer
<!--/codeinclude-->

## Module reference

The Dolt module exposes one entrypoint function to create the Dolt container, and this function receives two parameters:

```golang
func RunContainer(ctx context.Context, opts ...testcontainers.ContainerCustomizer) (*DoltContainer, error)
```

- `context.Context`, the Go context.
- `testcontainers.ContainerCustomizer`, a variadic argument for passing options.

### Container Options

When starting the Dolt container, you can pass options in a variadic way to configure it.

#### Image

If you need to set a different Dolt Docker image, you can use `testcontainers.WithImage` with a valid Docker image
for Dolt. E.g. `testcontainers.WithImage("dolthub/dolt-sql-server:1.32.4")`.

{% include "../features/common_functional_options.md" %}

#### Set username, password and database name

If you need to set a different database, and its credentials, you can use `WithUsername`, `WithPassword`, `WithDatabase`
options.

!!!info
The default values for the username is `root`, for password is `test` and for the default database name is `test`.

#### Init Scripts

If you would like to perform DDL or DML operations in the Dolt container, add one or more `*.sql`, `*.sql.gz`, or `*.sh`
scripts to the container request, using the `WithScripts(scriptPaths ...string)`. Those files will be copied under `/docker-entrypoint-initdb.d`.

#### Clone from remotes

If you would like to clone data from a remote into the Dolt container, add an `*.sh`
scripts to the container request, using the `WithScripts(scriptPaths ...string)`. Additionally, use `WithDoltCloneRemoteUrl(url string)` to specify
the remote to clone, and use `WithDoltCredsPublicKey(key string)` along with `WithCredsFile(credsFile string)` to authorize the Dolt container to clone from the remote.

<!--codeinclude-->
[Example of Clone script](../../modules/dolt/testdata/clone-db.sh)
<!--/codeinclude-->

#### Custom configuration

If you need to set a custom configuration, you can use `WithConfigFile` option to pass the path to a custom configuration file.

### Container Methods

#### ConnectionString

This method returns the connection string to connect to the Dolt container, using the default `3306` port.
It's possible to pass extra parameters to the connection string, e.g. `tls=skip-verify` or `application_name=myapp`, in a variadic way.
<!--codeinclude-->
[Get connection string](../../modules/dolt/dolt_test.go) inside_block:connectionString
<!--/codeinclude-->
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ nav:
- modules/cockroachdb.md
- modules/consul.md
- modules/couchbase.md
- modules/dolt.md
- modules/elasticsearch.md
- modules/gcloud.md
- modules/inbucket.md
Expand Down
5 changes: 5 additions & 0 deletions modules/dolt/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
include ../../commons-test.mk

.PHONY: test
test:
$(MAKE) test-dolt
251 changes: 251 additions & 0 deletions modules/dolt/dolt.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
package dolt

import (
"context"
"database/sql"
"fmt"
"path/filepath"
"strings"

"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)

const (
rootUser = "root"
defaultUser = "test"
defaultPassword = "test"
defaultDatabaseName = "test"
)

const defaultImage = "dolthub/dolt-sql-server:1.32.4"

// DoltContainer represents the Dolt container type used in the module
type DoltContainer struct {
testcontainers.Container
username string
password string
database string
}

func WithDefaultCredentials() testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
username := req.Env["DOLT_USER"]
if strings.EqualFold(rootUser, username) {
delete(req.Env, "DOLT_USER")
delete(req.Env, "DOLT_PASSWORD")
}
}
}

// RunContainer creates an instance of the Dolt container type
func RunContainer(ctx context.Context, opts ...testcontainers.ContainerCustomizer) (*DoltContainer, error) {
req := testcontainers.ContainerRequest{
Image: defaultImage,
ExposedPorts: []string{"3306/tcp", "33060/tcp"},
Env: map[string]string{
"DOLT_USER": defaultUser,
"DOLT_PASSWORD": defaultPassword,
"DOLT_DATABASE": defaultDatabaseName,
},
WaitingFor: wait.ForLog("Server ready. Accepting connections."),
}

genericContainerReq := testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
}

opts = append(opts, WithDefaultCredentials())

for _, opt := range opts {
opt.Customize(&genericContainerReq)
}

createUser := true
username, ok := req.Env["DOLT_USER"]
if !ok {
username = rootUser
createUser = false
}
password := req.Env["DOLT_PASSWORD"]

database := req.Env["DOLT_DATABASE"]
if database == "" {
database = defaultDatabaseName
}

if len(password) == 0 && password == "" && !strings.EqualFold(rootUser, username) {
return nil, fmt.Errorf("empty password can be used only with the root user")
}

container, err := testcontainers.GenericContainer(ctx, genericContainerReq)
if err != nil {
return nil, err
}

dc := &DoltContainer{container, username, password, database}

// dolthub/dolt-sql-server does not create user or database, so we do so here
err = dc.initialize(ctx, createUser)
return dc, err
}

func (c *DoltContainer) initialize(ctx context.Context, createUser bool) error {
connectionString, err := c.initialConnectionString(ctx)
if err != nil {
return err
}

var db *sql.DB
db, err = sql.Open("mysql", connectionString)
if err != nil {
return err
}
defer func() {
rerr := db.Close()
if err == nil {
err = rerr
}
}()

if err = db.Ping(); err != nil {
return fmt.Errorf("error pinging db: %w", err)
}

// create database
_, err = db.Exec(fmt.Sprintf("CREATE DATABASE IF NOT EXISTS %s;", c.database))
if err != nil {
return fmt.Errorf("error creating database %s: %w", c.database, err)
}

if createUser {
// create user
_, err = db.Exec(fmt.Sprintf("CREATE USER IF NOT EXISTS '%s' IDENTIFIED BY '%s';", c.username, c.password))
if err != nil {
return fmt.Errorf("error creating user %s: %w", c.username, err)
}

q := fmt.Sprintf("GRANT ALL ON %s.* TO '%s';", c.database, c.username)
// grant user privileges
_, err = db.Exec(q)
if err != nil {
return fmt.Errorf("error creating user %s: %w", c.username, err)
}
}

return nil
}

func (c *DoltContainer) initialConnectionString(ctx context.Context) (string, error) {
containerPort, err := c.MappedPort(ctx, "3306/tcp")
if err != nil {
return "", err
}

host, err := c.Host(ctx)
if err != nil {
return "", err
}

connectionString := fmt.Sprintf("root:@tcp(%s:%s)/", host, containerPort.Port())
return connectionString, nil
}

func (c *DoltContainer) MustConnectionString(ctx context.Context, args ...string) string {
addr, err := c.ConnectionString(ctx, args...)
if err != nil {
panic(err)
}
return addr
}

func (c *DoltContainer) ConnectionString(ctx context.Context, args ...string) (string, error) {
containerPort, err := c.MappedPort(ctx, "3306/tcp")
if err != nil {
return "", err
}

host, err := c.Host(ctx)
if err != nil {
return "", err
}

extraArgs := ""
if len(args) > 0 {
extraArgs = strings.Join(args, "&")
}
if extraArgs != "" {
extraArgs = "?" + extraArgs
}

connectionString := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s%s", c.username, c.password, host, containerPort.Port(), c.database, extraArgs)
return connectionString, nil
}

func WithUsername(username string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_USER"] = username
}
}

func WithPassword(password string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_PASSWORD"] = password
}
}

func WithDoltCredsPublicKey(key string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_CREDS_PUB_KEY"] = key
}
}

func WithDoltCloneRemoteUrl(url string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_REMOTE_CLONE_URL"] = url
}
}

func WithDatabase(database string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_DATABASE"] = database
}
}

func WithConfigFile(configFile string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
cf := testcontainers.ContainerFile{
HostFilePath: configFile,
ContainerFilePath: "/etc/dolt/servercfg.d/server.cnf",
FileMode: 0o755,
}
req.Files = append(req.Files, cf)
}
}

func WithCredsFile(credsFile string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
cf := testcontainers.ContainerFile{
HostFilePath: credsFile,
ContainerFilePath: "/root/.dolt/creds/" + filepath.Base(credsFile),
FileMode: 0o755,
}
req.Files = append(req.Files, cf)
}
}

func WithScripts(scripts ...string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
var initScripts []testcontainers.ContainerFile
for _, script := range scripts {
cf := testcontainers.ContainerFile{
HostFilePath: script,
ContainerFilePath: "/docker-entrypoint-initdb.d/" + filepath.Base(script),
FileMode: 0o755,
}
initScripts = append(initScripts, cf)
}
req.Files = append(req.Files, initScripts...)
}
}
Loading

0 comments on commit b5f1e13

Please sign in to comment.