Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka cluster example #1984

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
e29b4c9
Simplify `KafkaContainerCluster#start`
bsideup Oct 7, 2020
3f1cddb
When an image version is not specified, use `latest` as the default t…
rnorth Oct 11, 2020
bc7aaaf
Add workflow for Update Gradle Wrapper Action. (#3297)
cristiangreco Oct 11, 2020
edd2a1c
Always continue on error for examples CI (#3339)
rnorth Oct 12, 2020
5cc47ef
Bump snakeyaml from 1.25 to 1.27 in /core (#3252)
dependabot[bot] Oct 12, 2020
7d1734c
Bump lombok from 1.18.12 to 1.18.14 in /examples (#3322)
dependabot[bot] Oct 12, 2020
aeec90a
Bump r2dbc-mariadb from 0.8.3-beta1 to 0.8.4-rc in /modules/mariadb (…
dependabot[bot] Oct 12, 2020
413965a
Bump testng from 7.2.0 to 7.3.0 in /examples (#3068)
dependabot[bot] Oct 12, 2020
1845cfb
Bump assertj-core from 3.17.1 to 3.17.2 in /core (#3251)
dependabot[bot] Oct 12, 2020
05a30f6
Bump aws-java-sdk-dynamodb from 1.11.865 to 1.11.880 in /modules/dyna…
dependabot[bot] Oct 12, 2020
8b17881
Bump mockito-core from 3.5.11 to 3.5.13 in /core (#3275)
dependabot[bot] Oct 12, 2020
66e5f8a
Bump elasticsearch-rest-client from 7.9.1 to 7.9.2 in /modules/elasti…
dependabot[bot] Oct 12, 2020
759a441
Increase memory limits used in example (#3340)
rnorth Oct 12, 2020
831b639
Bump mockito-core from 3.5.11 to 3.5.13 in /modules/junit-jupiter (#3…
dependabot[bot] Oct 12, 2020
61c4119
Bump tomcat-jdbc from 9.0.37 to 9.0.39 in /modules/jdbc-test (#3338)
dependabot[bot] Oct 12, 2020
0fc92bd
Bump junit from 4.13 to 4.13.1 in /examples (#3328)
dependabot[bot] Oct 12, 2020
b2db18f
Bump mariadb-java-client from 2.6.2 to 2.7.0 in /modules/mariadb (#3278)
dependabot[bot] Oct 12, 2020
1261fee
Bump solr-solrj from 8.6.2 to 8.6.3 in /examples (#3321)
dependabot[bot] Oct 12, 2020
b6b828e
Bump postgresql from 42.2.16 to 42.2.17 in /examples (#3323)
dependabot[bot] Oct 12, 2020
0f58f36
Bump postgresql from 42.2.16 to 42.2.17 in /modules/junit-jupiter (#3…
dependabot[bot] Oct 12, 2020
8558994
Bump postgresql from 42.2.16 to 42.2.17 in /modules/spock (#3330)
dependabot[bot] Oct 12, 2020
5b70c58
Bump tomcat-jdbc from 9.0.37 to 9.0.39 in /modules/jdbc (#3333)
dependabot[bot] Oct 12, 2020
7681f12
Bump postgresql from 42.2.16 to 42.2.17 in /modules/postgresql (#3334)
dependabot[bot] Oct 12, 2020
304c2d7
Bump aws-java-sdk-sqs from 1.11.860 to 1.11.880 in /modules/localstac…
dependabot[bot] Oct 12, 2020
aa8e5ed
Bump httpclient from 4.5.12 to 4.5.13 in /modules/junit-jupiter (#3326)
dependabot[bot] Oct 12, 2020
98a8260
Bump httpclient from 4.5.12 to 4.5.13 in /modules/spock (#3329)
dependabot[bot] Oct 12, 2020
a094294
Bump cucumber-junit from 6.7.0 to 6.8.1 in /examples (#3325)
dependabot[bot] Oct 12, 2020
c1e8234
Bump org.springframework.boot from 2.3.3.RELEASE to 2.3.4.RELEASE in …
dependabot[bot] Oct 12, 2020
325c8b2
Bump aws-java-sdk-s3 from 1.11.870 to 1.11.880 in /modules/localstack…
dependabot[bot] Oct 13, 2020
4801ed8
Bump zt-exec from 1.10 to 1.12 in /core (#3253)
dependabot[bot] Oct 13, 2020
b2b8e62
Bump s3 from 2.14.21 to 2.15.7 in /modules/localstack (#3335)
dependabot[bot] Oct 13, 2020
e4c4cfe
Bump cucumber-java from 6.6.0 to 6.8.1 in /examples (#3324)
dependabot[bot] Oct 13, 2020
b073523
Remove GitHub Actions cache restore keys (#3342)
rnorth Oct 13, 2020
8ab49ee
Allow users to specify a MongoDB database name (#2980)
silaev Oct 13, 2020
0065801
Add GCloud module for Google Cloud Datastore, Firestore, PubSub, and …
eddumelendez Oct 13, 2020
3bf8d76
Use a lighter weight image for MultiplePortsExposedTest (#3343)
rnorth Oct 13, 2020
ae5010a
docker-machine: get full remote daemon URL, to allow for use of custo…
vcvitaly Oct 14, 2020
abf0f9a
Fix remote gradle cache 400 InvalidArgument error (#3346)
rnorth Oct 14, 2020
a33622f
Add gcloud endpoint accessors (#3344)
rnorth Oct 15, 2020
ee987ed
Merge branch 'master' into seglo/multi-broker-kafka
seglo Oct 15, 2020
7b55496
Merge branch 'seglo/multi-broker-kafka' into seglo/multi-broker-kafka
bsideup Feb 6, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions .github/workflows/ci-examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,10 @@ jobs:
java-version: '1.8'
- name: Cache Gradle Home files
uses: actions/cache@v1
continue-on-error: true
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-home-testmatrix-examples-${{ hashFiles('**/*.gradle') }}
restore-keys: |
${{ runner.os }}-gradle-home-testmatrix-examples-
- id: set-matrix
working-directory: ./examples/
env:
Expand All @@ -48,11 +47,10 @@ jobs:
java-version: '1.8'
- name: Cache Gradle Home files
uses: actions/cache@v1
continue-on-error: true
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-home-examples-${{matrix.gradle_args}}_check-${{ hashFiles('**/*.gradle') }}
restore-keys: |
${{ runner.os }}-gradle-home-examples-${{matrix.gradle_args}}_check-
- name: Clear existing docker image cache
run: docker image prune -af
- name: Build and test Examples with Gradle (${{matrix.gradle_args}})
Expand Down
4 changes: 0 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,6 @@ jobs:
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-home-testmatrix-${{ hashFiles('**/*.gradle') }}
restore-keys: |
${{ runner.os }}-gradle-home-testmatrix-
- id: set-matrix
env:
# Since we override the tests executor,
Expand All @@ -51,8 +49,6 @@ jobs:
with:
path: ~/.gradle/caches
key: ${{ runner.os }}-gradle-home-${{matrix.gradle_args}}_check-${{ hashFiles('**/*.gradle') }}
restore-keys: |
${{ runner.os }}-gradle-home-${{matrix.gradle_args}}_check-
- name: Clear existing docker image cache
run: docker image prune -af
- name: Build and test with Gradle (${{matrix.gradle_args}})
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/update-gradle-wrapper.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Update Gradle Wrapper

on:
schedule:
- cron: "0 0 * * *"

jobs:
update-gradle-wrapper:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2

- name: Update Gradle Wrapper
uses: gradle-update/update-gradle-wrapper-action@74a035c
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
labels: dependencies

- uses: gradle/wrapper-validation-action@e2c57ac
8 changes: 4 additions & 4 deletions core/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -192,11 +192,11 @@ dependencies {

compile 'com.github.docker-java:docker-java-transport-zerodep:3.2.5'

shaded "org.yaml:snakeyaml:1.25"
shaded "org.yaml:snakeyaml:1.27"

shaded 'org.glassfish.main.external:trilead-ssh2-repackaged:4.1.2'

shaded 'org.zeroturnaround:zt-exec:1.10', {
shaded 'org.zeroturnaround:zt-exec:1.12', {
exclude(group: 'org.slf4j')
}

Expand All @@ -205,13 +205,13 @@ dependencies {
testCompile 'com.rabbitmq:amqp-client:5.9.0'
testCompile 'org.mongodb:mongo-java-driver:3.12.7'

testCompile ('org.mockito:mockito-core:3.5.11') {
testCompile ('org.mockito:mockito-core:3.5.13') {
exclude(module: 'hamcrest-core')
}
// Synthetic JAR used for MountableFileTest and DirectoryTarResourceTest
testCompile files('testlib/repo/fakejar/fakejar/0/fakejar-0.jar')

testCompile 'org.assertj:assertj-core:3.17.1'
testCompile 'org.assertj:assertj-core:3.17.2'
testCompile project(':test-support')

jarFileTestCompileOnly "org.projectlombok:lombok:${lombok.version}"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,12 @@ private TransportConfig resolveTransportConfig() throws InvalidConfigurationExce

DockerMachineClient.instance().ensureMachineRunning(machineName);

String dockerDaemonIpAddress = DockerMachineClient.instance().getDockerDaemonIpAddress(machineName);
String dockerDaemonUrl = DockerMachineClient.instance().getDockerDaemonUrl(machineName);

log.info("Docker daemon IP address for docker machine {} is {}", machineName, dockerDaemonIpAddress);
log.info("Docker daemon URL for docker machine {} is {}", machineName, dockerDaemonUrl);

return TransportConfig.builder()
.dockerHost(URI.create("tcp://" + dockerDaemonIpAddress + ":2376"))
.dockerHost(URI.create(dockerDaemonUrl))
.sslConfig(
new LocalDirectorySSLConfig(
Paths.get(System.getProperty("user.home") + "/.docker/machine/certs/").toString()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,18 @@ public void ensureMachineRunning(@NonNull String machineName) {
}
}

/**
* @deprecated Use getDockerDaemonUrl(@NonNull String machineName) for connection to docker-machine
*/
@Deprecated
public String getDockerDaemonIpAddress(@NonNull String machineName) {
return runShellCommand(executableName, "ip", machineName);
}

public String getDockerDaemonUrl(@NonNull String machineName) {
return runShellCommand(executableName, "url", machineName);
}

public boolean isMachineRunning(String machineName) {
String status = runShellCommand("docker-machine", "status", machineName);
return status.trim().equalsIgnoreCase("running");
Expand Down
4 changes: 2 additions & 2 deletions core/src/main/java/org/testcontainers/utility/Versioning.java
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@ public boolean isValid() {

@Override
public String getSeparator() {
return "";
return ":";
}

@Override
public String toString() {
return "";
return "latest";
}

@Override
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
package org.testcontainers.dockerclient;

import com.github.dockerjava.api.DockerClient;
import com.github.dockerjava.api.model.Image;
import org.junit.Test;
import org.testcontainers.DockerClientFactory;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.startupcheck.OneShotStartupCheckStrategy;
import org.testcontainers.utility.DockerImageName;

import java.util.List;

public class AmbiguousImagePullTest {

@Test(timeout = 30_000)
public void testNotUsingParse() {
DockerClient client = DockerClientFactory.instance().client();
List<Image> alpineImages = client.listImagesCmd()
.withImageNameFilter("alpine:latest")
.exec();
for (Image alpineImage : alpineImages) {
client.removeImageCmd(alpineImage.getId()).exec();
}

try (
final GenericContainer<?> container = new GenericContainer<>(DockerImageName.parse("alpine"))
.withCommand("/bin/sh", "-c", "sleep 0")
.withStartupCheckStrategy(new OneShotStartupCheckStrategy())
) {
container.start();
// do nothing other than start and stop
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ public void testParsing() {
canonicalName = unversionedPart + versionSeparator + version;
} else {
combined = unversionedPart;
canonicalName = unversionedPart;
canonicalName = unversionedPart + ":latest";
}

VisibleAssertions.context("For " + combined);
Expand All @@ -124,7 +124,7 @@ public void testParsing() {
if (version != null) {
assertEquals(combined + " has version part: " + version, version, imageName.getVersionPart());
} else {
assertEquals(combined + " has no version specified", "", imageName.getVersionPart());
assertEquals(combined + " has automatic 'latest' version specified", "latest", imageName.getVersionPart());
}
assertEquals(combined + " has canonical name: " + canonicalName, canonicalName, imageName.asCanonicalNameString());

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ public class CmdModifierTest {
// }

// memory {
private long memoryInBytes = 8 * 1024 * 1024;
private long memorySwapInBytes = 12 * 1024 * 1024;
private long memoryInBytes = 32 * 1024 * 1024;
private long memorySwapInBytes = 64 * 1024 * 1024;

@Rule
public GenericContainer memoryLimitedRedis = new GenericContainer<>(DockerImageName.parse("redis:3.0.2"))
Expand All @@ -40,6 +40,6 @@ public void testHostnameModified() throws IOException, InterruptedException {
@Test
public void testMemoryLimitModified() throws IOException, InterruptedException {
final Container.ExecResult execResult = memoryLimitedRedis.execInContainer("cat", "/sys/fs/cgroup/memory/memory.limit_in_bytes");
assertEquals("8388608", execResult.getStdout().trim());
assertEquals(String.valueOf(memoryInBytes), execResult.getStdout().trim());
}
}
Original file line number Diff line number Diff line change
@@ -1,29 +1,29 @@
package generic;

import static org.slf4j.LoggerFactory.getLogger;

import org.junit.Rule;
import org.junit.Test;
import org.slf4j.Logger;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.output.Slf4jLogConsumer;
import org.testcontainers.utility.DockerImageName;

import static org.slf4j.LoggerFactory.getLogger;

public class MultiplePortsExposedTest {
private static final Logger log = getLogger(MultiplePortsExposedTest.class);


@Rule
// rule {
public GenericContainer container = new GenericContainer(DockerImageName.parse("orientdb:3.0.13"))
.withExposedPorts(2424, 2480)
public GenericContainer<?> container = new GenericContainer<>(DockerImageName.parse("testcontainers/helloworld:1.1.0"))
.withExposedPorts(8080, 8081)
.withLogConsumer(new Slf4jLogConsumer(log));
// }

@Test
public void fetchPortsByNumber() {
Integer firstMappedPort = container.getMappedPort(2424);
Integer secondMappedPort = container.getMappedPort(2480);
Integer firstMappedPort = container.getMappedPort(8080);
Integer secondMappedPort = container.getMappedPort(8081);
}

@Test
Expand All @@ -39,6 +39,6 @@ public void getHostOnly() {
@Test
public void getHostAndMappedPort() {
String address =
container.getHost() + ":" + container.getMappedPort(2424);
container.getHost() + ":" + container.getMappedPort(8080);
}
}
44 changes: 44 additions & 0 deletions docs/modules/gcloud.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# GCloud Module

!!! note
This module is INCUBATING. While it is ready for use and operational in the current version of Testcontainers, it is possible that it may receive breaking changes in the future. See [our contributing guidelines](/contributing/#incubating-modules) for more information on our incubating modules policy.

Testcontainers module for the Google's [Cloud SDK](https://cloud.google.com/sdk/).

Currently, the module supports `Datastore`, `Firestore`, `Pub/Sub` and `Spanner` emulators. In order to use it, you should use the following classes:

* DatastoreEmulatorContainer
* FirestoreEmulatorContainer
* PubSubEmulatorContainer
* SpannerEmulatorContainer

## Usage example

Running GCloud as a stand-in for Google Datastore during a test:

<!--codeinclude-->
[Creating a Datastore container](../../modules/gcloud/src/test/java/org/testcontainers/containers/DatastoreEmulatorContainerTest.java) inside_block:creatingDatastoreEmulatorContainer
<!--/codeinclude-->

And how to start it:

<!--codeinclude-->
[Starting a Datastore container](../../modules/gcloud/src/test/java/org/testcontainers/containers/DatastoreEmulatorContainerTest.java) inside_block:startingDatastoreEmulatorContainer
<!--/codeinclude-->

## Adding this module to your project dependencies

Add the following dependency to your `pom.xml`/`build.gradle` file:

```groovy tab='Gradle'
testCompile "org.testcontainers:gcloud:{{latest_version}}"
```

```xml tab='Maven'
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>gcloud</artifactId>
<version>{{latest_version}}</version>
<scope>test</scope>
</dependency>
```
4 changes: 2 additions & 2 deletions examples/cucumber/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies {
implementation 'org.seleniumhq.selenium:selenium-remote-driver:3.141.59'
implementation 'org.seleniumhq.selenium:selenium-firefox-driver:3.141.59'
implementation 'org.seleniumhq.selenium:selenium-chrome-driver:3.141.59'
testImplementation 'io.cucumber:cucumber-java:6.6.0'
testImplementation 'io.cucumber:cucumber-junit:6.7.0'
testImplementation 'io.cucumber:cucumber-java:6.8.1'
testImplementation 'io.cucumber:cucumber-junit:6.8.1'
testImplementation 'org.testcontainers:selenium'
}
4 changes: 2 additions & 2 deletions examples/disque-job-queue/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ repositories {
}

dependencies {
compileOnly "org.projectlombok:lombok:1.18.12"
annotationProcessor "org.projectlombok:lombok:1.18.10"
compileOnly "org.projectlombok:lombok:1.18.14"
annotationProcessor "org.projectlombok:lombok:1.18.14"
implementation 'biz.paluch.redis:spinach:0.3'
implementation 'com.google.code.gson:gson:2.8.6'
implementation 'com.google.guava:guava:23.0'
Expand Down
2 changes: 1 addition & 1 deletion examples/kafka-cluster/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ dependencies {
compile 'org.apache.kafka:kafka-clients:2.3.1'
testCompile 'org.assertj:assertj-core:3.14.0'
testCompile 'com.google.guava:guava:23.0'
testCompile 'org.slf4j:slf4j-simple:+'
testCompile 'org.slf4j:slf4j-simple:1.7.30'
}
2 changes: 1 addition & 1 deletion examples/linked-container/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ dependencies {
compileOnly 'org.slf4j:slf4j-api:1.7.30'
implementation 'com.squareup.okhttp3:okhttp:4.9.0'
implementation 'org.json:json:20180813'
testImplementation 'org.postgresql:postgresql:42.2.16'
testImplementation 'org.postgresql:postgresql:42.2.17'
testImplementation 'ch.qos.logback:logback-classic:1.2.3'
testImplementation 'org.testcontainers:postgresql'
}
Expand Down
2 changes: 1 addition & 1 deletion examples/mongodb-container/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ repositories {

dependencies {
testImplementation 'org.testcontainers:testcontainers'
testImplementation 'junit:junit:4.13'
testImplementation 'junit:junit:4.13.1'
testCompileClasspath 'org.jetbrains:annotations:20.1.0'
}
2 changes: 1 addition & 1 deletion examples/redis-backed-cache-testng/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ dependencies {
implementation 'com.google.guava:guava:23.0'
testImplementation 'org.testcontainers:testcontainers'
testImplementation 'ch.qos.logback:logback-classic:1.2.3'
testImplementation 'org.testng:testng:7.2.0'
testImplementation 'org.testng:testng:7.3.0'
}

test {
Expand Down
2 changes: 1 addition & 1 deletion examples/redis-backed-cache/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,6 @@ dependencies {
implementation 'com.google.code.gson:gson:2.8.6'
implementation 'com.google.guava:guava:23.0'
testImplementation 'org.testcontainers:testcontainers'
testImplementation 'junit:junit:4.13'
testImplementation 'junit:junit:4.13.1'
testImplementation 'ch.qos.logback:logback-classic:1.2.3'
}
2 changes: 1 addition & 1 deletion examples/selenium-container/build.gradle
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
plugins {
id 'java'
id 'org.springframework.boot' version '2.3.3.RELEASE'
id 'org.springframework.boot' version '2.3.4.RELEASE'
}
apply plugin: 'io.spring.dependency-management'

Expand Down
4 changes: 2 additions & 2 deletions examples/solr-container/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ repositories {
}

dependencies {
compileOnly "org.projectlombok:lombok:1.18.10"
compileOnly "org.projectlombok:lombok:1.18.14"
annotationProcessor "org.projectlombok:lombok:1.18.10"

implementation 'org.apache.solr:solr-solrj:8.6.2'
implementation 'org.apache.solr:solr-solrj:8.6.3'

testImplementation 'org.testcontainers:testcontainers'
testImplementation 'org.testcontainers:solr'
Expand Down
2 changes: 1 addition & 1 deletion examples/spring-boot/build.gradle
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
plugins {
id 'java'
id 'org.springframework.boot' version '2.3.3.RELEASE'
id 'org.springframework.boot' version '2.3.4.RELEASE'
}
apply plugin: 'io.spring.dependency-management'

Expand Down
Loading