[BUG] Problem to monitor specific docker container events during docker resources conversion from v1 to v2.16 in production

[damienklotz77]

Description

I am working on an application which include docker-compose to deploy docker-compose services.
When development has started, docker-compose was in version v1 and we released our project in production which deployed several docker-compose resources in docker-compose v1 version.
Now, we intend to release a new version of the project in production with docker-compose v2.
We do not want to use the compatibility flag in case this compatibility is not supported in the future.
The challenge is to make the project able to "convert" docker resources in production from compose v1 conventions to v2 conventions.

With compose v2.15.1, it was working properly by use "compose up --no-start" to recreate containers with new names.
But with compose v2.16.0, with the new recreate logic, the mecanism is broken with new prefix added to create event. Indeed we listen to docker events to monitor containers states. With new prefix added to canonical name for docker create event, we cannot monitor anymore the docker container status.

Why do this recreate logic has been changed ?

Steps To Reproduce

OS : Ubuntu 22.04

Let's install the three following docker-compose versions on the VM :

• v1 (v1.29.2)
• v2.15.1
• v2.16.0

sudo curl -Lo /usr/local/bin/docker-compose-1.29.2 https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
sudo curl -Lo /usr/local/bin/docker-compose-2.15.1 https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-Linux-x86_64
sudo curl -Lo /usr/local/bin/docker-compose-2.16.0 https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-Linux-x86_64
sudo chmod +x /usr/local/bin/docker-compose-1.29.2
sudo chmod +x /usr/local/bin/docker-compose-2.15.1
sudo chmod +x /usr/local/bin/docker-compose-2.16.0

Let's consider following docker-compose file :

services:
  myservice:
    image: 'hello-world'

We listen to docker events :
docker events

Let's execute following command with docker-compose v1 :

docker-compose-1.29.2 -p myproject up --no-start

And then we execute the following command with docker-compose v2.15.1 :

docker-compose-2.15.1 -p myproject up --no-start

We obtain the following docker events :

container create [...] ([...], com.docker.compose.version=1.29.2, [...], name=myproject_myservice_1, [...])
container attach [...] ([...], com.docker.compose.version=1.29.2, [...], name=myproject_myservice_1, [...])
container rename [...] ([...], com.docker.compose.version=1.29.2, [...], name=3aac21ccfe30_myproject_myservice_1, [...])
container create [...] ([...], com.docker.compose.version=2.15.1, [...], name=myproject-myservice-1, [...])
container destroy [...] ([...] com.docker.compose.version=1.29.2, [...], name=3aac21ccfe30_myproject_myservice_1, [...])

We observe that the second docker event "create" has property name equal to "myproject-myservice-1".

Let's clean project :
docker-compose-2.15.1 -p myproject down

Let's execute following command with docker-compose v1 again :

docker-compose-1.29.2 -p myproject up --no-start

And then we execute the following command with docker-compose v2.16.0 this time :

docker-compose-2.16.0 -p myproject up --no-start

We obtain following docker events :

container create [...] ([...], com.docker.compose.version=1.29.2, [...], name=myproject_myservice_1, [...])
container attach [...] ([...], com.docker.compose.version=1.29.2, [...], name=myproject_myservice_1, [...])
container create [...] ([...], com.docker.compose.version=2.16.0, [...], name=829e738869c4_myproject-myservice-1, [...])
container destroy [...] ([...], com.docker.compose.version=1.29.2, [...], name=myproject_myservice_1, [...])
container rename [...] ([...], com.docker.compose.version=2.16.0, [...], name=myproject-myservice-1, [...])

We observe that the second docker event "create" has property name equal to "829e738869c4_myproject-myservice-1".
A prefix has been added to the container name.

Is this change in behavior a desired behavior or is it a bug ?

Thank you,

Damien Klotz

Compose Version

1.29.2
2.15.1
2.16.0

Docker Environment

23.0.3

Anything else?

No response

[ndeloof]

The convergence logic relies on a hash being computed based on yaml configuration, but as we switched from Python to Go we can't make this hash equivalent between compose v1 and v2. So, up will recreate containers for containers created by the other version of compose.

While backward compatibility and interoperability has been a major requirement while developing Compose v2, it still comes with a few breaking chances (so the major release number bump). Please not this only happens as you switch to Compose v2.

Is there any reason you need to use both tools on the same application ?

[damienklotz77]

I think I was not clear enough in my description of the problem.

The problem is the difference of behavior between :
compose create v1 => compose recreate v2.15.1
AND
compose create v1 => compose recreate v2.16.0

I just noticed in changelog v2.16 :
Improved logging when recreating a service container. Pull request compose#10236
#10236

I think pull request 10236 is linked to my observation.

[ndeloof]

ok, I'll look into this later this week

[damienklotz77]

Hi, I moved reproduce steps to dedicated "reproduce step" and I added a description to better explain our problem with docker-compose v2.1.6.0 change.

[ndeloof]

Is this change in behavior a desired behavior or is it a bug ?

Actually, this is desired:
The logic to replace an obsolete container has changed on #10236 so we can track containers being re-created. Compose v1 and earlier used to rename old container with a temporary name, create new one then remove old one.
New logic creates replacement container with a temporary name, stops and remove old one, then rename new one to expected name. This is required so we can create (but not start) the replacement container before the old one is stopped, and doing so can detect replacement is in progress in another process.

This is really internal implementation detail, so could you please explain how you noticed this change and what's your use case monitoring such events ?

[ndeloof]

By the way, you should better rely on container labels, not names, to track compose resources.