vscode-dev-containers: it appears that the behavior of initializeCommand has changed

[marcindulak]

• VSCode Version:

code --version
1.72.2
d045a5eda657f4d7b676dedbfa7aab8207f8a075
x64

• Local OS Version:

cat /etc/*release | grep PRETTY
PRETTY_NAME="Ubuntu 20.04.5 LTS"

• Remote OS Version:

cat /etc/*release | grep PRETTY
PRETTY_NAME="Ubuntu 22.04.1 LTS"

• Remote Extension/Connection Type: Containers
  0.255.4 of https://github.com/microsoft/vscode-dev-containers - the tag is missing in github see Some releases/tags are missing on vscode-dev-containers github #7376
• Logs:

grep "Couldn't" ~/.config/Code/logs/20221017T*/exthost1/ms-vscode-remote.remote-containers/remoteContainers-*.log
[2022-10-17T16:23:06.406Z] Couldn't find env file: /tmp/vscode-dev-containers-initializeCommand-order/.devcontainer/env.container

• docker compose

docker compose version
Docker Compose version v2.10.2

• docker

docker version
Client: Docker Engine - Community
 Version:           20.10.18
 API version:       1.41
 Go version:        go1.18.6
 Git commit:        b40c2f6
 Built:             Thu Sep  8 23:11:45 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.18
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.6
  Git commit:       e42327a
  Built:            Thu Sep  8 23:09:37 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Steps to Reproduce:

It appears that initializeCommand changed the behavior somewhere between 0.251.0 (good) and 0.255.2 (bad) release of https://github.com/microsoft/vscode-dev-containers. This seems to be the same issue as #7366.

1. Create the repro files

   mkdir .devcontainer
   

   cat .devcontainer/docker-compose.yml
   version: "3.7"
   
   services:
     container:
       container_name: container
       image: ubuntu:22.04
       env_file:
         - .env.container
       command: sleep infinity
   

   cat .devcontainer/devcontainer.json
   {
    "name": "Container",
        "dockerComposeFile": "docker-compose.yml",
        "service": "container",
    "workspaceFolder": "/",
    "settings": {
    },
    "extensions": [
    ],
    "initializeCommand": "echo KEY=VALUE > .devcontainer/.env.container"
   }
   

2. To reproduce, in order to eliminate the interference of docker volumes,
   before any fresh start of code ., remove the file and vscode docker volume

   rm -f .devcontainer/.env.container
   docker volume rm vscode

3. Start code . from the root of this directory and allow for "Reopen in Container". A popup with "An error occurred setting up the container" appears. More logs are povided in the Logs section above. A successful container will start, and have the KEY=VALUE among its environment variables.

Does this issue occur when you try this locally?: Yes
Does this issue occur when you try this locally and all extensions are disabled?: Cannot disable the vscode-dev-containers extension, since it is needed to demonstrate the problem

[tcm0116]

I'm seeing this same issue as well. Reverting ms-vscode-remote.remote-containers to version 0.251.0 allows the initializeCommand to execute before any docker compose commands as expected.

[Jacques-Peeters]

Same here

[chrmarti]

Could you append the log? (F1 > Dev Containers: Show Container Log)

[marcindulak]

Example failed log:

[12475 ms] Dev Containers 0.255.4 in VS Code 1.72.2 (d045a5eda657f4d7b676dedbfa7aab8207f8a075).
[12474 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[12637 ms] 1.41
[12656 ms] Start: Run: /snap/code/111/usr/share/code/code --ms-enable-electron-run-as-node /home/ubuntu/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/\
devContainersSpecCLI.js read-configuration --workspace-folder /tmp/tmp --log-level debug --log-format json --config /tmp/tmp/.devcontainer/devcontainer.json --include-merged-configu\
ration --mount-workspace-git-root true
[13600 ms] (node:1192426) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.f\
rom() methods instead.
[13601 ms] (Use `code --trace-deprecation ...` to show where the warning was created)
[13617 ms] @devcontainers/cli 0.20.0. Node.js v16.14.2. linux 5.15.0-48-generic x64.
[13617 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/tmp/tmp
[13731 ms] Start: Run: docker-compose version --short
[15394 ms] Docker Compose version: 1.25.0
[15394 ms] Start: Run: docker-compose -f /tmp/tmp/.devcontainer/docker-compose.yml config
[16531 ms]
[16532 ms] Couldn't find env file: /tmp/tmp/.devcontainer/.env.container

[16532 ms] Exit code 1
[16532 ms] Error: Command failed: docker-compose -f /tmp/tmp/.devcontainer/docker-compose.yml config
    at vc (/home/ubuntu/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1859:904)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async EO (/home/ubuntu/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1787:19035)
    at async _ae (/home/ubuntu/.vscode/extensions/ms-vscode-remote.remote-containers-0.255.4/dist/spec-node/devContainersSpecCLI.js:1994:26821)
[16553 ms] Exit code 1

[chrmarti]

What has changed is that we now need to look up the image or Dockerfile in the docker-compose.yml in order to read the image metadata from the base image, so we can compute the complete configuration of the dev container. (The image metadata is new.)

To read the docker-compose.yml we use docker-compose config because that gives us an accurate view of the Docker Compose configuration (e.g., including env variable replacements and override files).

I suggest you update your setup such that docker-compose config works even before initializeCommand runs. We could look into running initializeCommand before that, but reading initializeCommand from the configuration while we are computing the complete configuration could potentially be cyclic (maybe not today because the image metadata cannot yet include an initializeCommand).

[marcindulak]

Others who are affected by this change may have different needs for initializeCommand, but I need a way to dynamically inject environment variables into .devcontainer/docker-compose.yml.

The KEY=VALUE in "initializeCommand": "echo KEY=VALUE > .devcontainer/.env.container" are coming from an API call, they are not static strings to be hard-coded and the .devcontainer/.env.container file that contains them not to be commited to the repository.

My current workaround is to create a wrapper over /usr/bin/docker-compose, which touches the missing devcontainer/.env.container file, but would prefer to use a built-in vscode support instead

#!/bin/bash

if test -d .devcontainer;
then
touch .devcontainer/.env.container
fi

/usr/bin/docker-compose.orig "$@"

As for the usefulness of docker-compose config for providing the Docker Compose configuration, it may be insufficient, since .devcontainer/docker-compose.yml will still be changed by the consequent initializeCommand and the information retrieved initially by docker-compose config previously will be no longer accurate.

[tcm0116]

For reference, below are the results of a quick search where it's suggested to create a .env file in the initializeCommand in order to dynamically generate the required environment for the container:

#6586

https://stackoverflow.com/questions/70227082/configuring-environment-variables-prior-to-creation-of-dev-container

#3981

The documentation for initializeCommand is a little squishy in that it doesn't specifically say that's it's run before any docker commands:

A command string or list of command arguments to run on the host machine before the container is created.

If this is going to remain a permanent change, then I suggest two things:

1. Update the documentation to be more clear about the sequence of events and the limitations of what can be done in initializeCommand.
2. Provide a mechanism to dynamically setup the required environment for the image.

[marcindulak]

Please let us hear whether a solution for injecting sensitive values or variables dynamically into docker-compose.yml will be supported by vscode remote-containers.

An older issue used initializeCommand as part of the workaround for injecting secrets into containers #4841 (comment). This method used initializeCommand to build an image, and used that image as devcontainer. I don't know whether this is something remote-containers are oficially supporting. That solution targeted also only the devcontainer itself, and not other containers possibly defined in docker-compose.yml, as opposed to the method by dynamically writing .env file.

The method of dynamically injecting possibly sensitive values will need to rely on vscode only, and not a presence of environment variables in a terminal session, since people may start vscode from a launcher and not terminal.

[chrmarti]

I just realized that we can skip reading the full config when there is an initializeCommand. (This won't allow us to check host requirement passed down from the base image upfront, but I think that's acceptable in this case.)

[dBotLFG]

Why was this closed? AFAICT initializeCommand is not working. We were using this to generate .devcontainer/.env which seems to be the crux of this bug report. Is initializeCommand fixed, or has the documentation been updated, or am I missing a workaround that explains how .env is supposed to be created now?

[chrmarti]

@dBotLFG I closed it when I added a fix to the code. I'll let you know when the fix is available in a new pre-release version, so you can give it a try. Thanks.

[chrmarti]

This fix is available in Dev Containers 0.264.0-pre-release. Let me know if that fixes this issue for you. Thanks.

[dBotLFG]

Yes, that fixed the issue for me. initializeCommand is now able to dynamically create .devcontainer/.env for use with docker-compose.yml with the fix you provided in 0.264.0-pre-release . Thank you very much