feat(modules.clickhouse): Add zookeeper for clickhouse clusterization #1995

laskoviymishka · 2023-12-15T17:03:40Z

If you would like to perform queries against clusterized ClickHouse you need to start zookeeper near your cluster and pass link to it via config.xml

API example:

	zkhost, zkport, err := RunZookeeper(ctx)
	if err != nil {
		t.Fatal(err)
	}

	container, err := RunContainer(ctx,
		WithUsername(user),
		WithPassword(password),
		WithDatabase(dbname),
		WithZookeeper(zkhost, zkport),
	)
	if err != nil {
		t.Fatal(err)
	}

What does this PR do?

Add option: WithZookeeper to clickhouse module.

Why is it important?

ClickHouse queries (especially DDLs) vary a lot between single node and cluster. Having it a clusterized allows you to apply exactly same code to real prod clusters and in your tests.

Related issues

netlify · 2023-12-15T17:03:46Z

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	`e1e52d6`
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-go/deploys/65b7a4b3caa0e300085eab73
😎 Deploy Preview	https://deploy-preview-1995--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

eddumelendez

Thanks for your contribution. I've left some comments but in general I would rather prefer to add an example about how to do this instead of hardcoding the configuration file. Also, wonder if it is possible to configure the zookeeper section to do not assume other config values. If so, then using templates would be great, redpanda module is doing so here.

With Testcontainers, containers can be composed programmatically and with the upcoming network support in #1993 modules would improve this experience. That's why I don't see having zookeeper as part of clickhouse.

modules/clickhouse/zookeeper.go

modules/clickhouse/mounts/zk_config.xml.tpl

modules/clickhouse/clickhouse.go

docs/modules/clickhouse.md

If you would like to perform queries against clusterized ClickHouse you need to start zookeeper near your cluster and pass link to it via `config.xml` API example: ``` zk, err := RunZookeeper(ctx) if err != nil { t.Fatal(err) } container, err := RunContainer(ctx, WithUsername(user), WithPassword(password), WithDatabase(dbname), WithZookeeper(zk), ) if err != nil { t.Fatal(err) } ```

Pass clickhouse config as separate config file Move xml config into go-tmpl

laskoviymishka · 2023-12-25T08:36:24Z

@eddumelendez gentle ping on this one <3

eddumelendez

LGTM

mdelapenya

LGTM, will approve it once the comments for the docs are addressed.

Thank you so much for your work here! And @eddumelendez, thanks for the review too!

modules/clickhouse/clickhouse.go

* main: Adding inbucket module (testcontainers#2142) testifylint: enable compares rule (testcontainers#2143) Bump containerd version to v1.7.12 (testcontainers#2137) feat: Add Minio module (testcontainers#2132) Adding LogConsumers start as part of the ContainerRequest (testcontainers#2073) chore: bring back assertion for network aliases for bridge in rootless mode (testcontainers#2141) chore(deps): bump github.com/docker/compose/v2 from 2.23.3 to 2.24.0 in /modules/compose (testcontainers#2096)

* main: chore(deps): bump github.com/docker/compose/v2 in /modules/compose (testcontainers#2162) feat(modules.cockroachdb) Adds cockroachdb module (testcontainers#2131)

modules/clickhouse/clickhouse_test.go

mdelapenya · 2024-02-02T15:45:12Z

Thanks @laskoviymishka for submitting this new feature! Much appreciated 🙇

* main: feat(modules.clickhouse): Add zookeeper for clickhouse clusterization (testcontainers#1995) redpanda: allow using SASL and TLS together (testcontainers#2140) chore: do not panic in testable examples (testcontainers#2193) fix: all mounts should contain the testcontainers labels (testcontainers#2191) [redpanda] sasl test for wrong mechanism (testcontainers#2048) fix: deprecate BindMounts correctly (testcontainers#2190)

* main: (33 commits) feat (postgres): support for creating and restoring Snapshots (testcontainers#2199) fix: apply volume options only to volumes (testcontainers#2201) redpanda/test: add admin client call (testcontainers#2200) chore(deps): bump cloud.google.com/go/spanner from 1.55.0 to 1.56.0 in /modules/gcloud, cloud.google.com/go/pubsub from 1.35.0 to 1.36.1 in /modules/gcloud, cloud.google.com/go/bigquery from 1.57.1 to 1.58.0 in /modules/gcloud (testcontainers#2197) chore(deps): bump github.com/docker/docker from 25.0.1+incompatible to 25.0.2+incompatible (testcontainers#2196) fix: go doc reference broken image (testcontainers#2195) Add Support for WASM Transforms to Redpanda Module (testcontainers#2170) feat(modules.clickhouse): Add zookeeper for clickhouse clusterization (testcontainers#1995) redpanda: allow using SASL and TLS together (testcontainers#2140) chore: do not panic in testable examples (testcontainers#2193) fix: all mounts should contain the testcontainers labels (testcontainers#2191) [redpanda] sasl test for wrong mechanism (testcontainers#2048) fix: deprecate BindMounts correctly (testcontainers#2190) chore(docker_mounts): stop doing misleading logging (testcontainers#2178) fix: Add HTTPStrategy WithForcedIPv4LocalHost To Fix Docker Port Map (testcontainers#1775) chore(deps): bump github.com/docker/compose/v2 in /modules/compose (testcontainers#2162) feat(modules.cockroachdb) Adds cockroachdb module (testcontainers#2131) chore(deps): bump golang.org/x/crypto in /modules/minio (testcontainers#2161) chore(deps): bump golang.org/x/crypto in /modules/openldap (testcontainers#2165) chore(deps): bump github.com/google/uuid from 1.5.0 to 1.6.0 (testcontainers#2169) ...

laskoviymishka requested a review from a team as a code owner December 15, 2023 17:03

eddumelendez reviewed Dec 15, 2023

View reviewed changes

modules/clickhouse/zookeeper.go Outdated Show resolved Hide resolved

modules/clickhouse/zookeeper.go Outdated Show resolved Hide resolved

laskoviymishka requested a review from eddumelendez December 15, 2023 18:39

eddumelendez reviewed Dec 18, 2023

View reviewed changes

modules/clickhouse/mounts/zk_config.xml.tpl Show resolved Hide resolved

modules/clickhouse/clickhouse.go Outdated Show resolved Hide resolved

modules/clickhouse/clickhouse.go Outdated Show resolved Hide resolved

docs/modules/clickhouse.md Outdated Show resolved Hide resolved

laskoviymishka requested a review from eddumelendez December 18, 2023 21:21

laskoviymishka force-pushed the zk-ch-cluster branch 2 times, most recently from cf89f03 to 70b3a15 Compare December 22, 2023 08:57

laskoviymishka added 4 commits December 25, 2023 09:36

Remove separate zookeeper container

8f21cd3

Pass clickhouse config as separate config file Move xml config into go-tmpl

fix comments typo and review notes

46212ec

fix golangci-lint

ca29cdf

laskoviymishka force-pushed the zk-ch-cluster branch from 70b3a15 to ca29cdf Compare December 25, 2023 08:36

eddumelendez previously approved these changes Jan 22, 2024

View reviewed changes

Merge branch 'main' into zk-ch-cluster

93d81f2

mdelapenya reviewed Jan 24, 2024

View reviewed changes

modules/clickhouse/clickhouse.go Show resolved Hide resolved

modules/clickhouse/clickhouse.go Show resolved Hide resolved

mdelapenya added 2 commits January 26, 2024 16:29

docs: document panics

dd1f6e6

mdelapenya dismissed eddumelendez’s stale review via dd1f6e6 January 26, 2024 15:32

mdelapenya added 4 commits January 26, 2024 20:53

Merge branch 'main' into zk-ch-cluster

c509392

Merge branch 'main' into zk-ch-cluster

572fc52

* main: chore(deps): bump github.com/docker/compose/v2 in /modules/compose (testcontainers#2162) feat(modules.cockroachdb) Adds cockroachdb module (testcontainers#2131)

fix: use require.NoError in tests

ecc6f9a

docs: use require.NoError in docs

e1e52d6

mdelapenya changed the title ~~feat(modules.clickhouse): Add zookeeper for clikchouse clusterization~~ feat(modules.clickhouse): Add zookeeper for clickhouse clusterization Jan 29, 2024

mdelapenya self-assigned this Jan 29, 2024

mdelapenya added the feature New functionality or new behaviors on the existing one label Jan 29, 2024

mdelapenya reviewed Jan 29, 2024

View reviewed changes

modules/clickhouse/clickhouse_test.go Show resolved Hide resolved

mdelapenya merged commit bc67dc3 into testcontainers:main Feb 2, 2024
80 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(modules.clickhouse): Add zookeeper for clickhouse clusterization #1995

feat(modules.clickhouse): Add zookeeper for clickhouse clusterization #1995

laskoviymishka commented Dec 15, 2023 •

edited

Loading

netlify bot commented Dec 15, 2023 •

edited

Loading

eddumelendez left a comment

laskoviymishka commented Dec 25, 2023

eddumelendez left a comment

mdelapenya left a comment

mdelapenya commented Feb 2, 2024

feat(modules.clickhouse): Add zookeeper for clickhouse clusterization #1995

feat(modules.clickhouse): Add zookeeper for clickhouse clusterization #1995

Conversation

laskoviymishka commented Dec 15, 2023 • edited Loading

What does this PR do?

Why is it important?

Related issues

netlify bot commented Dec 15, 2023 • edited Loading

✅ Deploy Preview for testcontainers-go ready!

eddumelendez left a comment

Choose a reason for hiding this comment

laskoviymishka commented Dec 25, 2023

eddumelendez left a comment

Choose a reason for hiding this comment

mdelapenya left a comment

Choose a reason for hiding this comment

mdelapenya commented Feb 2, 2024

laskoviymishka commented Dec 15, 2023 •

edited

Loading

netlify bot commented Dec 15, 2023 •

edited

Loading