Ko-built Windows images aren't pullable/runnable

[imjasonh]

Reported in Tekton's #windows Slack channel: https://tektoncd.slack.com/archives/C020SPF0B7Y/p1620277036006500 -- copied here for posterity and to aid future discussion.

tl;dr: Images that Ko builds for Windows don't appear to be pullable, possibly due to KO_DATA_PATH and the /var/run/ko directory in the tar file.

It sounds like we could benefit from some Windows CI at least, to ensure that Windows images built with ko are runnable. We could have a GitHub Actions Windows runner pull and run a ko-built image, or after we cut a release with #339 we could build and run ko within CI to detect breakages before merging pull requests.

The specific issue might require us to have Windows-specific logic in gobuild.go, to change where kodata ends up in the image.

---

@aiden-deloryn

I have been doing some testing with a Windows node and have encountered a problem which prevents the entrypoint image from running (and as a result the TaskRun's pod won't initialise). Describing the pod reveals the following error under Events:

Failed to pull image "10.115.11.41:30002/tekton/entrypoint-bff0a22da108bc2f16c818c97641a296@sha256:1f900110f0103c208200d0be7e5614836d036845ac934d093192826dd2b94d84": rpc error: code = Unknown desc = failed to register layer: re-exec error: exit status 1: output: mkdir \\?\C:\ProgramData\docker\tmp\hcs523070999\var\run\ko: expected relative path

After inspecting the container image, I can see that the path /var/run/ko is injected into the image as an environment variable called KO_DATA_PATH. According to the Ko documentation this path is used to bundle static assets from a kodata sub-directory into the image's KO_DATA_PATH directory. I'm not sure why this extra path is being appended to the beginning - \\?\C:\ProgramData\docker\tmp\hcs523070999.

As this problem occurs when processing an image layer it can also be reproduced by running docker pull <image>, after building the Windows image with ko publish ./cmd/entrypoint --platform=windows/amd64. Note: for testing purposes, I'm using a Ko base image override of github.com/tektoncd/pipeline/cmd/entrypoint: mcr.microsoft.com/windows/servercore:ltsc2019

I haven't used Ko previously so I'm not all that familiar with it. I'm also not sure if this is a Docker issue or a Ko issue. My Ko version is 0.8.2 and Windows Docker runtime version 20.10.4 (Mirantis). I can't inspect the image using Docker because I cannot pull it. It seems either Ko is producing an incorrect image layer, or Docker is interpreting/converting the path incorrectly. Is there any way I can troubleshoot Ko further to see what's going on?

---

@aiden-deloryn

Your suggestion does sound plausible to me. Perhaps a relative path could be the solution (as the error seems to suggest). Maybe just removing the root path separator would be enough? (i.e. var/run/ko)
If we need a more specific location, I've found some directories in nanoserver (smallest image) which might be suitable:
C:\ProgramData\ >>> for application data that is not user-specific
C:\Windows\Temp\ >>> for temporary files
C:\Users\ContainerUser\ >>> user's home directory
C:\Users\ContainerUser\AppData\Local\ >>> user-specific application data
C:\Users\ContainerUser\AppData\Local\Temp\ >>> user's temp directory

---

@adamrehn:

Looking at the implementation of tarAddDirectories(), it looks like the intended file permissions for the kodata directory are world readable and executable, but not writable? If static data files don't need to be writable then C:\ProgramData\ko might be the ideal candidate, since the primary benefit of opting for a temporary or user-specific directory would be that the resulting files can be modified. (Avoiding user-specific directories also sidesteps any issues around determining whether a given Windows container image uses the ContainerUser or ContainerAdministrator user by default.)

[imjasonh]

I briefly hacked up ko to make /var/run/ko and /ko-app both relative dirs (var/run/ko and ko-app), and built a simple Windows image using it. With these changes, the image fails to pull with:

docker: failed to register layer: re-exec error: exit status 1: output: hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3).

edit: the image I tested with is here: gcr.io/imjasonh/ghatest2-3b797f3433e868656fc12cc06198c02c@sha256:f0631af760a78fea65ca3633e03bf568c451568bae7f894d250bf9df449df01e

[aiden-deloryn]

I can confirm similar results using relative paths, although I saw a slightly different error message:

failed to register layer: re-exec error: exit status 1: output: mkdir \\?\C:\ProgramData\docker\tmp\hcs136446283\var\run\ko: The system cannot find the path specified

Using the skopeo command line utility I was able to copy an entrypoint image created by ko from Docker Hub to a local directory on my computer. This made it possible to un-tar the image layers and inspect the directory contents. Under <image-layer>/var/run/ko I found the kodata files as expected:

HEAD
LICENSE
refs/
third_party/

I then created a simple image for comparison using this Dockerfile:

FROM mcr.microsoft.com/windows/servercore:ltsc2019

COPY my-folder var/run/ko

This resulted in two top level folders being created; Files and Hives. The data from the COPY command is located in <image-layer>/Files/var/run/ko. The main difference here is the extra parent directory 'Files'. This seems to have been generated by Docker when building the image.

[imjasonh]

Yeah, you can also explore it online here: https://explore.ggcr.dev/?image=gcr.io%2Fimjasonh%2Fghatest2-3b797f3433e868656fc12cc06198c02c%40sha256%3Af0631af760a78fea65ca3633e03bf568c451568bae7f894d250bf9df449df01e

I don't understand how docker build knows to inject the Files/ and Hives/ directories. From what I can tell from this post, Hives/ seems to be there to be able to record additions (and by whiteout files, deletions) on Windows registry values, and Files/ holds the same for the filesystem, like non-Windows images.

It's possible that tarBinary for Windows containers needs to prepend the Files/ directory prefix in order for this to work?

[jonjohnsonjr]

I ran into this recently, maybe they know? https://github.com/buildpacks/imgutil/blob/5a2568dd25b62b6f9620d7d94cb1abf2ddbe0d76/layer/windows_baselayer.go

[aiden-deloryn]

Here is a comparison of a configuration file generated by Ko and one generated by Docker for reference:

Ko generated config file

{
    "architecture": "amd64",
    "author": "github.com/google/ko",
    "created": "2021-04-09T20:34:41.3608362Z",
    "history": [
        {
            "created": "2020-05-07T05:09:25.0463439Z",
            "created_by": "Apply image 1809-RTM-amd64"
        },
        {
            "created": "2021-04-09T20:34:41.3608362Z",
            "created_by": "Install update ltsc2019-amd64"
        },
        {
            "author": "ko",
            "created": "0001-01-01T00:00:00Z",
            "created_by": "ko publish ko://github.com/tektoncd/pipeline/cmd/entrypoint",
            "comment": "kodata contents, at $KO_DATA_PATH"
        },
        {
            "author": "ko",
            "created": "0001-01-01T00:00:00Z",
            "created_by": "ko publish ko://github.com/tektoncd/pipeline/cmd/entrypoint",
            "comment": "go build output, at ko-app/entrypoint"
        }
    ],
    "os": "windows",
    "rootfs": {
        "type": "layers",
        "diff_ids": [
            "sha256:a7ba3db29ebb3a32e6a9c912d6ab5941bd981a0c39a420e6786c70a66babb80b",
            "sha256:30275bef3136de22586acb46a3a28617738e9c2d0df13e546d6b2c3119ab269d",
            "sha256:1cd897584015e8248109cfd98233c0d89302a68c3ec115538460210685ed810e",
            "sha256:547733b18d0a2be0d9462ea75a5284e4326c8a623522e6525635786888ec76cf"
        ]
    },
    "config": {
        "Cmd": [
            "c:\\windows\\system32\\cmd.exe"
        ],
        "Entrypoint": [
            "ko-app/entrypoint"
        ],
        "Env": [
            "PATH=ko-app",
            "KO_DATA_PATH=var/run/ko"
        ]
    },
    "os.version": "10.0.17763.1879"
}

Docker generated config file

{
    "architecture": "amd64",
    "config": {
        "Hostname": "",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": [
            "c:\\windows\\system32\\cmd.exe"
        ],
        "Image": "sha256:152749f71f8fd6004056d15c7fd5791563072703171eb8dbd3e66b2ee67f8287",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": null
    },
    "container_config": {
        "Hostname": "",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": [
            "cmd",
            "/S",
            "/C",
            "#(nop) COPY dir:77c5636325457486302b0915a625e3205113750c5aae8dac34bb9d2db932b573 in var\\run\\ko "
        ],
        "Image": "sha256:152749f71f8fd6004056d15c7fd5791563072703171eb8dbd3e66b2ee67f8287",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": null
    },
    "created": "2021-05-11T03:18:43.9711165Z",
    "docker_version": "20.10.4",
    "history": [
        {
            "created": "2020-05-07T05:09:25.0463439Z",
            "created_by": "Apply image 1809-RTM-amd64"
        },
        {
            "created": "2021-04-09T20:34:41.3608362Z",
            "created_by": "Install update ltsc2019-amd64"
        },
        {
            "created": "2021-05-11T03:18:43.9711165Z",
            "created_by": "cmd /S /C #(nop) COPY dir:77c5636325457486302b0915a625e3205113750c5aae8dac34bb9d2db932b573 in var\\run\\ko "
        }
    ],
    "os": "windows",
    "os.version": "10.0.17763.1879",
    "rootfs": {
        "type": "layers",
        "diff_ids": [
            "sha256:a7ba3db29ebb3a32e6a9c912d6ab5941bd981a0c39a420e6786c70a66babb80b",
            "sha256:30275bef3136de22586acb46a3a28617738e9c2d0df13e546d6b2c3119ab269d",
            "sha256:e6cb1711b093584dcf7a549c63815f8d50cc866b753b0c825b388e378f8c714e"
        ]
    }
}

It seems the Docker generated config file has an additional container_config property. The ConfigFile definition (https://github.com/google/go-containerregistry/blob/main/pkg/v1/config.go) does not have a field for container_config, and I don't see a reference to this in OCI Image Configuration: https://github.com/opencontainers/image-spec/blob/master/config.md#properties

This is just an observation and I'm not sure if container_config is even needed.

[imjasonh]

If I move the binary from /var/run/ko to Files/ko and create an empty Hives/ directory, I can get docker run to fail slightly differently (image here):

docker: Error response from daemon: container e112da5454b73c27dc054d70e0c81d0f808ca98e9d1626ae60148eb1b7fd5c5f encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {"CommandLine":"Files/ko","User":"ContainerUser","WorkingDirectory":"C:\\","CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}.

(this is with config.Entrypoint = "Files/ko")

edit: I've been hacking apart ko to try to successfully build a Windows container image, diff here: main...imjasonh:windows

[imjasonh]

I've made a bit more progress (I think), latest code at main...imjasonh:windows

With this change, the ko-built image is pullable at least, but still fails to run:

gcr.io/imjasonh/windows/ko-98b8c7facdad74510a7cae0cd368eb4e@sha256:d416c86566c1b19e71634ed12aefde8ad8257c0bfa141f8a4cc1ae3ddfc65eee: Pulling from imjasonh/windows/ko-98b8c7facdad74510a7cae0cd368eb4e
e4800203e906: Pulling fs layer
b99cd9a3c11a: Pulling fs layer
b99cd9a3c11a: Verifying Checksum
b99cd9a3c11a: Download complete
e4800203e906: Verifying Checksum
e4800203e906: Download complete
e4800203e906: Pull complete
b99cd9a3c11a: Pull complete
Digest: sha256:d416c86566c1b19e71634ed12aefde8ad8257c0bfa141f8a4cc1ae3ddfc65eee
Status: Downloaded newer image for gcr.io/imjasonh/windows/ko-98b8c7facdad74510a7cae0cd368eb4e@sha256:d416c86566c1b19e71634ed12aefde8ad8257c0bfa141f8a4cc1ae3ddfc65eee
gcr.io/imjasonh/windows/ko-98b8c7facdad74510a7cae0cd368eb4e@sha256:d416c86566c1b19e71634ed12aefde8ad8257c0bfa141f8a4cc1ae3ddfc65eee
docker: Error response from daemon: container 34d7364a2dbfd34f55163f9d60c841ea9e4a81711bbe20080132883d93190efa encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: Access is denied. (0x5) extra info: {"CommandLine":"C:\\ko-app\\ko.exe","User":"ContainerUser","WorkingDirectory":"C:\\","CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}.
Error: Process completed with exit code 125.

This image was built from that branch with:

KO_DOCKER_REPO=gcr.io/imjasonh/windows KO_DEFAULTBASEIMAGE=mcr.microsoft.com/windows/nanoserver:1809 go run ./ publish ./ --platform=windows/amd64

This builds a ko image, based on Microsoft's proprietary nanoserver base image. Buildpacks has some code to be able to produce Windows images not based on proprietary base images, but honestly I'd settle for anything at this point. Baby steps.

[TBBle]

Just came across this, you might find the information in microsoft/hcsshim#853 (which includes a bunch of information from Buildpacks too) and the implementation in microsoft/hcsshim#901 useful as a reference on Windows container image layout.