Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support Dolt #2177

Merged
merged 14 commits into from
Apr 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ jobs:
matrix:
go-version: [1.21.x, 1.x]
platform: [ubuntu-latest]
module: [artemis, cassandra, chroma, clickhouse, cockroachdb, compose, consul, couchbase, elasticsearch, gcloud, inbucket, influxdb, k3s, k6, kafka, localstack, mariadb, milvus, minio, mockserver, mongodb, mssql, mysql, nats, neo4j, ollama, openfga, openldap, opensearch, postgres, pulsar, qdrant, rabbitmq, redis, redpanda, registry, surrealdb, vault, weaviate]
module: [artemis, cassandra, chroma, clickhouse, cockroachdb, compose, consul, couchbase, dolt, elasticsearch, gcloud, inbucket, influxdb, k3s, k6, kafka, localstack, mariadb, milvus, minio, mockserver, mongodb, mssql, mysql, nats, neo4j, ollama, openfga, openldap, opensearch, postgres, pulsar, qdrant, rabbitmq, redis, redpanda, registry, surrealdb, vault, weaviate]
uses: ./.github/workflows/ci-test-go.yml
with:
go-version: ${{ matrix.go-version }}
Expand Down
4 changes: 4 additions & 0 deletions .vscode/.testcontainers-go.code-workspace
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@
"name": "module / couchbase",
"path": "../modules/couchbase"
},
{
"name": "module / dolt",
"path": "../modules/dolt"
},
{
"name": "module / elasticsearch",
"path": "../modules/elasticsearch"
Expand Down
81 changes: 81 additions & 0 deletions docs/modules/dolt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Dolt

Not available until the next release of testcontainers-go <a href="https://github.com/testcontainers/testcontainers-go"><span class="tc-version">:material-tag: main</span></a>

## Introduction

The Testcontainers module for Dolt.

## Adding this module to your project dependencies

Please run the following command to add the Dolt module to your Go dependencies:

```
go get github.com/testcontainers/testcontainers-go/modules/dolt
```

## Usage example

<!--codeinclude-->
[Creating a Dolt container](../../modules/dolt/examples_test.go) inside_block:runDoltContainer
<!--/codeinclude-->

## Module reference

The Dolt module exposes one entrypoint function to create the Dolt container, and this function receives two parameters:

```golang
func RunContainer(ctx context.Context, opts ...testcontainers.ContainerCustomizer) (*DoltContainer, error)
```

- `context.Context`, the Go context.
- `testcontainers.ContainerCustomizer`, a variadic argument for passing options.

### Container Options

When starting the Dolt container, you can pass options in a variadic way to configure it.

#### Image

If you need to set a different Dolt Docker image, you can use `testcontainers.WithImage` with a valid Docker image
for Dolt. E.g. `testcontainers.WithImage("dolthub/dolt-sql-server:1.32.4")`.

{% include "../features/common_functional_options.md" %}

#### Set username, password and database name

If you need to set a different database, and its credentials, you can use `WithUsername`, `WithPassword`, `WithDatabase`
options.

!!!info
The default values for the username is `root`, for password is `test` and for the default database name is `test`.

#### Init Scripts

If you would like to perform DDL or DML operations in the Dolt container, add one or more `*.sql`, `*.sql.gz`, or `*.sh`
scripts to the container request, using the `WithScripts(scriptPaths ...string)`. Those files will be copied under `/docker-entrypoint-initdb.d`.

#### Clone from remotes

If you would like to clone data from a remote into the Dolt container, add an `*.sh`
scripts to the container request, using the `WithScripts(scriptPaths ...string)`. Additionally, use `WithDoltCloneRemoteUrl(url string)` to specify
the remote to clone, and use `WithDoltCredsPublicKey(key string)` along with `WithCredsFile(credsFile string)` to authorize the Dolt container to clone from the remote.

<!--codeinclude-->
[Example of Clone script](../../modules/dolt/testdata/clone-db.sh)
<!--/codeinclude-->

#### Custom configuration

If you need to set a custom configuration, you can use `WithConfigFile` option to pass the path to a custom configuration file.

### Container Methods

#### ConnectionString

This method returns the connection string to connect to the Dolt container, using the default `3306` port.
It's possible to pass extra parameters to the connection string, e.g. `tls=skip-verify` or `application_name=myapp`, in a variadic way.

<!--codeinclude-->
[Get connection string](../../modules/dolt/dolt_test.go) inside_block:connectionString
<!--/codeinclude-->
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ nav:
- modules/cockroachdb.md
- modules/consul.md
- modules/couchbase.md
- modules/dolt.md
- modules/elasticsearch.md
- modules/gcloud.md
- modules/inbucket.md
Expand Down
5 changes: 5 additions & 0 deletions modules/dolt/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
include ../../commons-test.mk

.PHONY: test
test:
$(MAKE) test-dolt
251 changes: 251 additions & 0 deletions modules/dolt/dolt.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
package dolt

import (
"context"
"database/sql"
"fmt"
"path/filepath"
mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
"strings"

"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)

const (
rootUser = "root"
defaultUser = "test"
defaultPassword = "test"
defaultDatabaseName = "test"
)

const defaultImage = "dolthub/dolt-sql-server:1.32.4"

// DoltContainer represents the Dolt container type used in the module
type DoltContainer struct {
testcontainers.Container
username string
password string
database string
}

func WithDefaultCredentials() testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
username := req.Env["DOLT_USER"]
if strings.EqualFold(rootUser, username) {
delete(req.Env, "DOLT_USER")
delete(req.Env, "DOLT_PASSWORD")
}
}
}

// RunContainer creates an instance of the Dolt container type
func RunContainer(ctx context.Context, opts ...testcontainers.ContainerCustomizer) (*DoltContainer, error) {
req := testcontainers.ContainerRequest{
Image: defaultImage,
ExposedPorts: []string{"3306/tcp", "33060/tcp"},
Env: map[string]string{
"DOLT_USER": defaultUser,
"DOLT_PASSWORD": defaultPassword,
"DOLT_DATABASE": defaultDatabaseName,
},
WaitingFor: wait.ForLog("Server ready. Accepting connections."),
}

genericContainerReq := testcontainers.GenericContainerRequest{
ContainerRequest: req,
Started: true,
}

opts = append(opts, WithDefaultCredentials())

for _, opt := range opts {
opt.Customize(&genericContainerReq)
}

createUser := true
username, ok := req.Env["DOLT_USER"]
if !ok {
username = rootUser
createUser = false
}
password := req.Env["DOLT_PASSWORD"]

database := req.Env["DOLT_DATABASE"]
if database == "" {
database = defaultDatabaseName
}

if len(password) == 0 && password == "" && !strings.EqualFold(rootUser, username) {
return nil, fmt.Errorf("empty password can be used only with the root user")
}

container, err := testcontainers.GenericContainer(ctx, genericContainerReq)
if err != nil {
return nil, err
}

dc := &DoltContainer{container, username, password, database}

// dolthub/dolt-sql-server does not create user or database, so we do so here
err = dc.initialize(ctx, createUser)
return dc, err
}

func (c *DoltContainer) initialize(ctx context.Context, createUser bool) error {
connectionString, err := c.initialConnectionString(ctx)
if err != nil {
return err
}

var db *sql.DB
db, err = sql.Open("mysql", connectionString)
if err != nil {
return err
}
defer func() {
rerr := db.Close()
if err == nil {
err = rerr
}
}()

if err = db.Ping(); err != nil {
return fmt.Errorf("error pinging db: %w", err)
}

// create database
_, err = db.Exec(fmt.Sprintf("CREATE DATABASE IF NOT EXISTS %s;", c.database))
if err != nil {
return fmt.Errorf("error creating database %s: %w", c.database, err)
}

if createUser {
// create user
_, err = db.Exec(fmt.Sprintf("CREATE USER IF NOT EXISTS '%s' IDENTIFIED BY '%s';", c.username, c.password))
if err != nil {
return fmt.Errorf("error creating user %s: %w", c.username, err)
}

q := fmt.Sprintf("GRANT ALL ON %s.* TO '%s';", c.database, c.username)
// grant user privileges
_, err = db.Exec(q)
if err != nil {
return fmt.Errorf("error creating user %s: %w", c.username, err)
}
}

return nil
}

func (c *DoltContainer) initialConnectionString(ctx context.Context) (string, error) {
containerPort, err := c.MappedPort(ctx, "3306/tcp")
if err != nil {
return "", err
}

host, err := c.Host(ctx)
if err != nil {
return "", err
}

connectionString := fmt.Sprintf("root:@tcp(%s:%s)/", host, containerPort.Port())
return connectionString, nil
}

func (c *DoltContainer) MustConnectionString(ctx context.Context, args ...string) string {
addr, err := c.ConnectionString(ctx, args...)
if err != nil {
panic(err)
}
return addr
}

func (c *DoltContainer) ConnectionString(ctx context.Context, args ...string) (string, error) {
containerPort, err := c.MappedPort(ctx, "3306/tcp")
if err != nil {
return "", err
}

host, err := c.Host(ctx)
if err != nil {
return "", err
}

extraArgs := ""
if len(args) > 0 {
extraArgs = strings.Join(args, "&")
}
if extraArgs != "" {
extraArgs = "?" + extraArgs
}

connectionString := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s%s", c.username, c.password, host, containerPort.Port(), c.database, extraArgs)
return connectionString, nil
}

func WithUsername(username string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_USER"] = username
}
}

func WithPassword(password string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_PASSWORD"] = password
}
}

func WithDoltCredsPublicKey(key string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_CREDS_PUB_KEY"] = key
}
}

func WithDoltCloneRemoteUrl(url string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_REMOTE_CLONE_URL"] = url
}
}

func WithDatabase(database string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
req.Env["DOLT_DATABASE"] = database
}
}

func WithConfigFile(configFile string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
cf := testcontainers.ContainerFile{
HostFilePath: configFile,
ContainerFilePath: "/etc/dolt/servercfg.d/server.cnf",
FileMode: 0o755,
}
req.Files = append(req.Files, cf)
}
}

mdelapenya marked this conversation as resolved.
Show resolved Hide resolved
func WithCredsFile(credsFile string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
cf := testcontainers.ContainerFile{
HostFilePath: credsFile,
ContainerFilePath: "/root/.dolt/creds/" + filepath.Base(credsFile),
FileMode: 0o755,
}
req.Files = append(req.Files, cf)
}
}

func WithScripts(scripts ...string) testcontainers.CustomizeRequestOption {
return func(req *testcontainers.GenericContainerRequest) {
var initScripts []testcontainers.ContainerFile
for _, script := range scripts {
cf := testcontainers.ContainerFile{
HostFilePath: script,
ContainerFilePath: "/docker-entrypoint-initdb.d/" + filepath.Base(script),
FileMode: 0o755,
}
initScripts = append(initScripts, cf)
}
req.Files = append(req.Files, initScripts...)
}
}
Loading
Loading