Healtcheck not updated

[wehrstedt]

Describe the bug

An image specifies the health check with various arguments, e. g. the following health check:

HEALTHCHECK --interval=5s --timeout=2s --start-period=10s --retries=3 CMD [ "/bin/bash", "/healthcheck.sh" ]

container: ($> docker inspect kjn2kj3498)

// ...
"Healthcheck": {
  "Test": [
    "CMD",
    "/bin/bash",
    "/healthcheck.sh"
  ],
  "Interval": 5000000000,
  "Timeout": 2000000000,
  "StartPeriod": 10000000000,
  "Retries": 3
},

The image author updates the health check parameters, e. g. timeout will be increased to 15s, and retries to 4:

HEALTHCHECK --interval=5s --timeout=15s --start-period=10s --retries=4 CMD [ "/bin/bash", "/healthcheck.sh" ]

Watchtower updates the container by pulling the new image and recreate a new container of the new image. But the new container runs with the old healthcheck arguments instead of the updated ones:

$> docker inspect ui43j3na

// ...
"Healthcheck": {
  "Test": [
    "CMD",
    "/bin/bash",
    "/healthcheck.sh"
  ],
  "Interval": 5000000000,
  "Timeout": 2000000000,
  "StartPeriod": 10000000000,
  "Retries": 3
},

If you manually re-create the container, e. g. using docker-compose pull and docker-compose up -d, the updated health check parameters will be applied to the new container:

$> docker inspect asdn2904ds

// ...
  "Healthcheck": {
    "Test": [
        "CMD",
        "/bin/bash",
        "/healthcheck.sh"
    ],
    "Interval": 60000000000,
    "Timeout": 15000000000,
    "StartPeriod": 600000000000,
    "Retries": 4
  },

Steps to reproduce

1. Create a simplified dockerfile:

FROM ubuntu

HEALTHCHECK --interval=5s --timeout=2s --start-period=10s --retries=3 CMD [ "/usr/bin/sleep", "1s"  ]

ENTRYPOINT [ "/usr/bin/sleep", "10m" ]

2. build the image: docker build --no-cache -t healthcheck-image .
3. publish the image: docker push healthcheck-image
4. Create a container + watchtower with the following compose file:

version: '3'
services:
  healthcheck-test:
    image: healtcheck-image
    labels:
      - com.centurylinklabs.watchtower.enable=true

  watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 10
    environment:
      - WATCHTOWER_LABEL_ENABLE=true

5. Inspect the container to check the healthcheck configuration

$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 2s 10s 3}

6. Update the healthcheck timeout and retries Parameter

FROM ubuntu

HEALTHCHECK --interval=5s --timeout=15s --start-period=10s --retries=4 CMD [ "/usr/bin/sleep", "1s"  ]

ENTRYPOINT [ "/usr/bin/sleep", "10m" ]

7. rebuild and republish the image
8. Wait for watchtower to update the container
9. re-check the container healthcheck configuration

$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 2s 10s 3}

Expected:

$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 15s 10s 4}

Do the same with docker-compose up -d --force-recreate. The expected healthcheck configuration will be applied.

Expected behavior

The container should be recreated with the updated healtcheck configuration

Screenshots

No response

Environment

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 9
  Running: 2
  Paused: 0
  Stopped: 7
 Images: 347
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 12.27GiB
 Name: Wehrstedt
 ID: DDGC:2DRA:UQYD:5LEP:PAUW:AA4Y:XA6G:MJD5:SCWX:U6MM:3D5Z:Z2IV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.17.0.0/12, Size: 20
   Base: 192.168.0.0/16, Size: 24

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Your logs

Attaching to watchtower-test_watchtower_1, watchtower-test_healthcheck-test_1
watchtower_1        | time="2023-03-21T11:43:49Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="Making sure everything is sane before starting"
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="Retrieving running containers"
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="There are no additional watchtower containers"
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="Watchtower HTTP API skipped."
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Watchtower 1.5.3"
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Using no notifications"
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Only checking containers using enable label"
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Scheduling first run: 2023-03-21 11:44:00 +0000 UTC"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Checking containers for updated images"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Retrieving running containers"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Trying to load authentication credentials." container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Loaded auth credentials for user <user>, on registry <registry-url>/healtcheck-image:latest, from file /config.json"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Got image name: <registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Credentials loaded"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Checking if pull is needed" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Building challenge URL" URL="https://<registry-url>/v2/"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://<registry-url>/jwt/auth\",service=\"container_registry\"" status="401 Unauthorized"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Checking challenge header content" realm="https://<registry-url>/jwt/auth" service=container_registry
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Setting scope for auth token" image=<registry-url>/healtcheck-image scope="repository:<registry-url>/healtcheck-image:pull"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Credentials found."
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Parsing image ref" host=<registry-url> image=healtcheck-image normalized="<registry-url>/healtcheck-image:latest" tag=latest
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://<registry-url>/v2/healtcheck-image/manifests/latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Could not do a head request for \"<registry-url>/healtcheck-image:latest\", falling back to regular pull." container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://<registry-url>/jwt/auth\\\",service=\\\"container_registry\\\",scope=\\\"repository:healtcheck-image:pull\\\",error=\\\"insufficient_scope\\\"\"" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Pulling image" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=info msg="Found new <registry-url>/healtcheck-image:latest image (feb17fe278fe)"
watchtower_1        | time="2023-03-21T11:44:30Z" level=info msg="Stopping /watchtower-test_healthcheck-test_1 (f29ee537d965) with SIGTERM"
watchtower_1        | time="2023-03-21T11:44:40Z" level=debug msg="Skipped another update already running."
watchtower_1        | time="2023-03-21T11:44:40Z" level=debug msg="Scheduled next run: 2023-03-21 11:44:50 +0000 UTC"
watchtower_1        | time="2023-03-21T11:44:40Z" level=debug msg="Removing container f29ee537d965"
watchtower-test_healthcheck-test_1 exited with code 137
watchtower_1        | time="2023-03-21T11:44:41Z" level=info msg="Creating /watchtower-test_healthcheck-test_1"
watchtower_1        | time="2023-03-21T11:44:41Z" level=debug msg="Starting container /watchtower-test_healthcheck-test_1 (de887b1bc2fc)"
watchtower_1        | time="2023-03-21T11:44:41Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=1 notify=no

Additional context

No response

[simskij]

Interesting! Yeah, I think we copy the current container spec to the teeth, including any healthcheck declarations that it might have. Will have to look into how to honor changes to it.