GenericContainer broken after upgrade from 10.9 to 10.10 #785

jonesbusy · 2024-06-24T15:51:57Z

Hi,

I've facing an issue after upgrade testcontainer node from 10.9 to 10.10 which any other changes

Expected Behaviour

Same as 10.9.0, it should work

Actual Behaviour

Doesn't work, tested with different framework (Backstage and one JS vite application). It's the same

Testcontainer Logs

[DEBUG] Testing container runtime strategy "TestcontainersHostStrategy"...
[DEBUG] Found custom configuration: dockerHost: "tcp://localhost:2475"
[DEBUG] Testing container runtime strategy "ConfigurationStrategy"...
[DEBUG] Container runtime strategy "ConfigurationStrategy" does not work
[DEBUG] Testing container runtime strategy "UnixSocketStrategy"...
[DEBUG] Testing container runtime strategy "RootlessUnixSocketStrategy"...
[DEBUG] Testing container runtime strategy "NpipeSocketStrategy"...

This line is different from

[DEBUG] Container runtime strategy "ConfigurationStrategy" does not work

On 10.9 is

[DEBUG] Container runtime strategy "ConfigurationStrategy" works

Steps to Reproduce

On code

    wireMockContainer = await new GenericContainer("wiremock/wiremock:3.5.4-1")
      .withExposedPorts(8080)
      .withCommand(["--verbose"])
      .withWaitStrategy(Wait.forLogMessage(/port:.*8080/))
      .withStartupTimeout(120000)
      .start();

Environment Information

Running node 20 LTS
I'm running testcontainer with kubedock and following env set

DOCKER_HOST = 'tcp://localhost:2475'
TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX  = '*********-docker-io.***************.ch/'

I can provide more details if needed

Thanks

The text was updated successfully, but these errors were encountered:

cristianrgreco · 2024-06-24T16:16:05Z

Hi @jonesbusy,

I gotta say I'm stumped by this one 😄

These are the changes between 10.9 and 10.10 (https://github.com/testcontainers/testcontainers-node/releases/tag/v10.10.0):

🚀 Features
- Add ollama module @ilopezluna (https://github.com/testcontainers/testcontainers-node/pull/778)
- Add RabbitMQ module @gundersong (https://github.com/testcontainers/testcontainers-node/pull/770)

🐛 Bug Fixes
- GenericContainerBuilder false pull policy should not pull image @schummar (https://github.com/testcontainers/testcontainers-node/pull/771)

🧹 Maintenance
- Update minor deps @cristianrgreco (https://github.com/testcontainers/testcontainers-node/pull/780)
- Use new sshd:1.2.0 image @mdelapenya (https://github.com/testcontainers/testcontainers-node/pull/758)

There are no changes which would affect how a container runtime is resolved, which is what the error in this issue describes.

The features are 2 additional modules which do not affect the core. The bug fix resolves whether or not images should be force pulled, something which happens after a container runtime is resolved. The new SSHD image affects the port forwarding functionality which you do not use. The dependency updates are all minor. The core dependencies have not even been updated. These changes have all passed the builds and smoke tests.

I would like some more evidence that this is an issue with Testcontainers and not a change to your kubedock environment before investigating further.

jonesbusy · 2024-06-24T16:51:26Z

Sure I will provide evidence

Could it be minor dependencies upgraded in #780 ?

No change on the infrastructure. Basically renovate found the upgrade. This is the only package upgraded and it's dependencies

The main branch is building (I've run it again to be sure. All works with 10.9)

The only difference I can see on the DEBUG log is the ConfigurationStrategy that fail but doesn't give me the reason

On the renovate branch

On main branch with 10.9 everything is good

It happens on my 2 application using testcontainers (Spotify Backstage and one very simple Vite application). Same error

Despite

DEBUG = 'testcontainers*'

Do you think we can get more logs why the ConfigurationStrategy fail ?

Regards,

rudemex · 2024-06-24T19:11:05Z

I have the same problem

zotijs · 2024-06-25T09:06:24Z

@jonesbusy can you provide you gitlab ci job configuration? I have the same issue on 10.9.0 with @tesctontainers/mysql and with this job configuration:

test:
  stage: test
  extends: .test_application
  services:
    - name: docker:dind
      alias: docker
      # explicitly disable tls to avoid docker startup interruption
      command: ['--tls=false']
  variables:
    DOCKER_HOST: 'tcp://docker:2375'
    DOCKER_TLS_CERTDIR: ''
    DEBUG: testcontainers*
  only:
    - merge_requests

Docker executor
node lts alpine image

cristianrgreco · 2024-06-25T09:08:25Z

I've tried to reproduce this issue in GitLab and cannot:

.gitlab-ci.yml:

test:
  image: node:20-alpine
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DEBUG: testcontainers*
  script:
    - npm install
    - npm test

test.ts:

import { GenericContainer } from "testcontainers";

describe("Test", () => {
  it("1", async () => {
    const container = await new GenericContainer("alpine").start();
    await container.stop();
  });
});

package.json dependencies:

"devDependencies": {
  "@types/jest": "^29.5.12",
  "jest": "^29.7.0",
  "testcontainers": "^10.10.0",
  "ts-jest": "^29.1.5",
  "typescript": "^5.5.2"
}

Could someone please help me get a reproducer?

jonesbusy · 2024-06-25T09:13:52Z

Hi @zotijs I'm not using gitlab. I have Jenkins and K8S agent and use https://github.com/joyrex2001/kubedock to proxy the docker socket call to Kubernetes and create testcontainers

The only vars I'm using on my Jenkinsfile is

DOCKER_HOST = 'tcp://localhost:2475'
TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX  = '*********-docker-io.***************.ch/'
DEBUG = 'testcontainers*'

zotijs · 2024-06-25T09:32:29Z

@cristianrgreco I will try to create a reproduction repository, because the repository that I am having the issue is private

jonesbusy · 2024-06-25T11:02:11Z

Adding more logs I get

On 10.10.0

[11:18:31.368+02:00] - 2024-06-25T09:18:31.185Z testcontainers [DEBUG] Testing container runtime strategy "ConfigurationStrategy"...
[11:18:31.368+02:00] - 2024-06-25T09:18:31.197Z modem Sending: {
[11:18:31.368+02:00] -   path: '/info',
[11:18:31.368+02:00] -   method: 'GET',
[11:18:31.368+02:00] -   headers: { 'User-Agent': 'tc-node/10.10.0' },
[11:18:31.368+02:00] -   key: undefined,
[11:18:31.368+02:00] -   cert: undefined,
[11:18:31.368+02:00] -   ca: undefined,
[11:18:31.368+02:00] -   hostname: 'localhost',
[11:18:31.368+02:00] -   port: '2475'
[11:18:31.368+02:00] - }
[11:18:31.368+02:00] - 2024-06-25T09:18:31.205Z modem Received: {"ID":"com.joyrex2001.kubedock","Labels":[],"MemTotal":0,"Name":"kubedock","OperatingSystem":"kubernetes","ServerVersion":"0.16.0"}
[11:18:31.368+02:00] - 2024-06-25T09:18:31.230Z testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" does not work

On 10.9.0

[12:59:11.118+02:00] - 2024-06-25T10:59:10.930Z modem Sending: {
[12:59:11.118+02:00] -   path: '/info',
[12:59:11.118+02:00] -   method: 'GET',
[12:59:11.118+02:00] -   headers: { 'User-Agent': 'tc-node/10.9.0' },
[12:59:11.118+02:00] -   key: undefined,
[12:59:11.118+02:00] -   cert: undefined,
[12:59:11.118+02:00] -   ca: undefined,
[12:59:11.118+02:00] -   hostname: 'localhost',
[12:59:11.118+02:00] -   port: '2475'
[12:59:11.118+02:00] - }
[12:59:11.118+02:00] - 2024-06-25T10:59:10.938Z modem Received: {"ID":"com.joyrex2001.kubedock","Labels":[],"MemTotal":0,"Name":"kubedock","OperatingSystem":"kubernetes","ServerVersion":"0.16.0"}
[12:59:11.118+02:00] - 2024-06-25T10:59:10.974Z testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" works

Not sure it help.

Response is identical

zotijs · 2024-06-25T20:15:08Z

@cristianrgreco sorry for the delayed response. So a similar setup to what you have provided works for me too, gitlab-pipeline. I will create a branch with something similar to what I am actually doing in the private repo. But I am wondering if the issue is the group runners that we are using inside the organization. Anyhow I will come back with a more realistic example

jonesbusy · 2024-07-08T04:28:01Z

Hi,

This is the logs after testing the 10.10.1

[06:25:49.508+02:00] - 2024-07-08T04:25:49.272Z modem Received: {"ID":"com.joyrex2001.kubedock","Labels":[],"MemTotal":0,"Name":"kubedock","OperatingSystem":"kubernetes","ServerVersion":"0.16.0"}
[06:25:49.509+02:00] - 2024-07-08T04:25:49.274Z testcontainers [TRACE] Fetching remote container runtime socket path...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.274Z testcontainers [TRACE] Resolving host...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.274Z testcontainers [TRACE] Fetching Compose info...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.298Z testcontainers [TRACE] Looking up host IPs...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.300Z testcontainers [TRACE] Initialising clients...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" does not work: TypeError: Cannot convert undefined or null to object
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Checking container runtime strategy "UnixSocketStrategy"...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Container runtime strategy "UnixSocketStrategy" is not applicable
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Checking container runtime strategy "RootlessUnixSocketStrategy"...
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Container runtime strategy "RootlessUnixSocketStrategy" is not applicable
[06:25:49.509+02:00] - 2024-07-08T04:25:49.301Z testcontainers [DEBUG] Checking container runtime strategy "NpipeSocketStrategy"...
[06:25:49.510+02:00] - 2024-07-08T04:25:49.302Z testcontainers [DEBUG] Container runtime strategy "NpipeSocketStrategy" is not applicable

Container runtime strategy "ConfigurationStrategy" does not work: TypeError: Cannot convert undefined or null to object

Any idea ?

cristianrgreco · 2024-07-08T08:54:39Z

@jonesbusy That's progress, seems there's some issue with the configuration used by the ConfigurationStrategy. Unfortunately the error doesn't pinpoint much so I've raised #794 to print a stacktrace if there is one. I'll release this as a patch soon and let you know and hopefully you won't mind retrying?

jonesbusy · 2024-07-08T09:08:27Z

@cristianrgreco Sure I will try as soon as it's released. Thanks

cristianrgreco · 2024-07-08T19:06:54Z

Thanks @jonesbusy. Released in 10.10.2

jonesbusy · 2024-07-09T04:07:57Z

This is the stackstrace of 10.10.2

[06:04:18.991+02:00] - 2024-07-09T04:04:18.874Z testcontainers [DEBUG] TypeError: Cannot convert undefined or null to object
[06:04:18.991+02:00] -     at Function.keys (<anonymous>)
[06:04:18.991+02:00] -     at initStrategy (/home/jenkins/agent/workspace/****-10.10.2/node_modules/testcontainers/src/container-runtime/clients/client.ts:125:22)
[06:04:18.991+02:00] -     at getContainerRuntimeClient (/home/jenkins/agent/workspace/*****-10.10.2/node_modules/testcontainers/src/container-runtime/clients/client.ts:51:22)
[06:04:18.991+02:00] -     at GenericContainer.start (/home/jenkins/agent/workspace/e_feature_test-containe*******r-10.10.2/node_modules/testcontainers/src/generic-container/generic-container.ts:81:20)
[06:04:18.991+02:00] -     at /home/jenkins/agent/workspace/****-10.10.2/src/ts/api.test.ts:23:25
[06:04:18.991+02:00] -     at async Promise.all (index 0)
[06:04:18.991+02:00] -     at callSuiteHook (file:///home/jenkins/agent/workspace/****-10.10.2/node_modules/@vitest/runner/dist/index.js:700:23)
[06:04:18.991+02:00] -     at runSuite (file:///home/jenkins/agent/workspace/****-10.10.2/node_modules/@vitest/runner/dist/index.js:892:27)
[06:04:18.991+02:00] -     at runSuite (file:///home/jenkins/agent/workspace/*****-10.10.2/node_modules/@vitest/runner/dist/index.js:909:15)
[06:04:18.991+02:00] -     at runFiles (file:///home/jenkins/agent/workspace/*******-10.10.2/node_modules/@vitest/runner/dist/index.js:958:5)
[06:04:18.991+02:00] - 2024-07-09T04:04:18.874Z testcontainers [DEBUG] Checking container runtime strategy "UnixSocketStrategy"...

cristianrgreco · 2024-07-09T07:53:17Z

Thanks @jonesbusy, turns out this is your issue: ab46673#diff-07adb1d2976c5e124c3804996a054095cdc770c58ec8a1b7af5c24f37da21ee7R109, added as part of the Ollama module.

Will be fixed in #795.

Thanks for your help in debugging this one, I'm happy with the error logs we've achieved around the container runtime

cristianrgreco · 2024-07-09T10:22:23Z

Will leave issue open until released and fix confirmed. ETA for release within next few hours.

cristianrgreco · 2024-07-09T14:33:54Z

@jonesbusy Please try 10.10.3

jonesbusy · 2024-07-09T14:35:38Z

Just done

All work as expected

Thank you so much for the help!

cristianrgreco · 2024-07-09T14:48:15Z

Thanks to you for raising such a detailed issue and for helping so much in debugging, always a pleasure! 🙂

cristianrgreco added the triage Investigation required label Jun 24, 2024

cristianrgreco mentioned this issue Jun 25, 2024

[Gitlab CI] Could not find a working container runtime strategy #786

Closed

cristianrgreco mentioned this issue Jun 25, 2024

Add logs around resolving a container runtime #787

Merged

cristianrgreco mentioned this issue Jul 8, 2024

Log stacktrace when container runtime strategy throws #794

Merged

cristianrgreco mentioned this issue Jul 9, 2024

Docker runtimes may be undefined #795

Merged

cristianrgreco closed this as completed in #795 Jul 9, 2024

cristianrgreco reopened this Jul 9, 2024

jonesbusy closed this as completed Jul 9, 2024

msahakyan-korio mentioned this issue Aug 4, 2024

"Could not find a working container runtime strategy" when starting MySqlContainer #813

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GenericContainer broken after upgrade from 10.9 to 10.10 #785

GenericContainer broken after upgrade from 10.9 to 10.10 #785

jonesbusy commented Jun 24, 2024

cristianrgreco commented Jun 24, 2024

jonesbusy commented Jun 24, 2024

rudemex commented Jun 24, 2024 •

edited

Loading

zotijs commented Jun 25, 2024

cristianrgreco commented Jun 25, 2024 •

edited

Loading

jonesbusy commented Jun 25, 2024 •

edited

Loading

zotijs commented Jun 25, 2024

jonesbusy commented Jun 25, 2024

zotijs commented Jun 25, 2024

jonesbusy commented Jul 8, 2024

cristianrgreco commented Jul 8, 2024

jonesbusy commented Jul 8, 2024

cristianrgreco commented Jul 8, 2024

jonesbusy commented Jul 9, 2024

cristianrgreco commented Jul 9, 2024 •

edited

Loading

cristianrgreco commented Jul 9, 2024 •

edited

Loading

cristianrgreco commented Jul 9, 2024

jonesbusy commented Jul 9, 2024

cristianrgreco commented Jul 9, 2024

GenericContainer broken after upgrade from 10.9 to 10.10 #785

GenericContainer broken after upgrade from 10.9 to 10.10 #785

Comments

jonesbusy commented Jun 24, 2024

cristianrgreco commented Jun 24, 2024

jonesbusy commented Jun 24, 2024

rudemex commented Jun 24, 2024 • edited Loading

zotijs commented Jun 25, 2024

cristianrgreco commented Jun 25, 2024 • edited Loading

jonesbusy commented Jun 25, 2024 • edited Loading

zotijs commented Jun 25, 2024

jonesbusy commented Jun 25, 2024

zotijs commented Jun 25, 2024

jonesbusy commented Jul 8, 2024

cristianrgreco commented Jul 8, 2024

jonesbusy commented Jul 8, 2024

cristianrgreco commented Jul 8, 2024

jonesbusy commented Jul 9, 2024

cristianrgreco commented Jul 9, 2024 • edited Loading

cristianrgreco commented Jul 9, 2024 • edited Loading

cristianrgreco commented Jul 9, 2024

jonesbusy commented Jul 9, 2024

cristianrgreco commented Jul 9, 2024

rudemex commented Jun 24, 2024 •

edited

Loading

cristianrgreco commented Jun 25, 2024 •

edited

Loading

jonesbusy commented Jun 25, 2024 •

edited

Loading

cristianrgreco commented Jul 9, 2024 •

edited

Loading

cristianrgreco commented Jul 9, 2024 •

edited

Loading