As a Developer I want to be able to build Windows Containers with BuildKit

[bplasmeijer]

The windows BuildKit is only supported on Linux

[jsturtevant]

It is possible to use buildx (which uses buildkit) if there are no RUN commands in the docker file:

docker buildx create --name img-builder --use
docker buildx build --no-cache --pull --push \
  --platform "windows/amd64" -t $imagetag -f $dockerfile .
docker buildx rm img-builder

ARG nanoserverTag="mcr.microsoft.com/windows/nanoserver:1809-amd64"
# othertags list: curl -L https://mcr.microsoft.com/v2/windows/nanoserver/tags/list

FROM ${nanoserverTag}
COPY some.exe "/Windows/System32/some.exe"

It would be interesting to see if could use some windows emulator connected to Binfmt_misc as QEMU is used for other architectures

[jorgearteiro]

There is an issue logged on BuildKit repo moby/buildkit#616

[TBBle]

@jsturtevant If you wanted to go down that path, I believe Wine can be used with binfmt_misc, although I'm not sure how well that would work with namespaces since there's a WineServer and configuration directory to create, and you probably don't want that in the final image...

It would be easier to prototype that on a raw Linux box than via the Linux support in Docker on Windows, I think.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Current status of my work on moby/buildkit#616 is that it's stalled waiting on a pile of 4 containerd PRs to be reviewed and advanced. The top of the stack is containerd/containerd#4419, but there's some outstanding design questions on the top two PRs regarding HCS behaviours, see microsoft/hcsshim#853, but the bottom two are just pending containerd maintainer review to merge or rework as appropriate.

And after that, will come fixing path-related issues in the BuildKit LLB layer, for which I'd only scratched the surface of in moby/buildkit#1621, exploring the solution space, and which needs to be reworked per maintainer feedback.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Status unchanged, still waiting on merging or feedback on containerd support for Windows container creation/export/mount support, and then continuing the work on BuildKit itself, starting from moby/buildkit#1621

[immuzz]

@kevpar Do you have any information on the containerd PRs mentioned above?

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Still working through containerd PRs per above. One of four has landed, and one needs to be rewritten or at least iterated upon further. I've also opened PR on hcsshim again to remove some duplication of code with the containerd PRs.

That said, Docker is looking at supporting containerd on Windows for 21.x (moby/moby#41455), which might mean we get BuildKit on Docker+containerd even without that containerd PR stack, as the parts I was trying to implement in containerd (WCOW layer management) are provided by Docker when Docker uses containerd, as far as I know.

I haven't looked closely at how BuildKit integrates with Docker though, it's possible it'll still need fixes in containerd, and either way, BuildKit itself has higher-level issues to be resolved, e.g., moby/buildkit#1621.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Not much has changed since December. The containerd PRs are now in a reasonable state, two are ready-to-go and one (containerd/containerd#4419) is functional but possibly needs some discussion about the test suite changes, and depends on a hcsshim feature PR (microsoft/hcsshim#901).

Some recent testing suggests that that hcsshim PR might have issues with Windows 10 20H2 that don't appear on Windows Server LTSC2019, but I haven't definitely proved that as I have only tested the latter via CI pipelines so far. Edit: Still using CI pipelines, I confirmed that the PR works on RS5 on AppVeyor (VS 2019 image), but fails on my personal Windows 10 20H2 box, suggesting that some part of the reverse engineering used to build that PR was incorrect or incomplete.

[immuzz]

@weijuans-msft can you provide clarity here as part of your effort to provide containerd parity with Docker developer experience.

[ghost]

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

[TBBle]

Compared to January, I've identified and have a fix for the issue that microsoft/hcsshim#901 shows on Windows 10 20H2, but that PR is still pending merge, and then the fix will be up for PR.

The containerd PRs (containerd/containerd#4399, containerd/containerd#4415, and containerd/containerd#4419) should all be in a reasonable state to flow through, although the latter depends on microsoft/hcsshim#901, often triggers a now-known containerd issue (containerd/containerd#4924) in its test suite, and also frequently triggers an unknown (hcsshim?) issue that I've yet been unable to replicate outside containerd.

It's possible that the Docker/containerd integration for Windows (moby/moby#41455 is the best place to track that) will mean some or all of those PRs stop being critical-path for BuildKit on Windows, and then the BuildKit-local platform issues (i.e. filesystem handling and similar) can be addressed.

[weijuans-msft]

@TBBle great work here! I changed this to "In Progress" as I am seeing you and lots of others in the moby and containerd community working on this. Thank you.

[ghost]

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

[pl4nty]

GitHub Actions hosted Windows runners are pinned to Docker 24.0.7, so I wrote some steps to setup BuildKit manually:

- name: Setup containerd
  run: |
    $version = "1.7.15"
    curl.exe -L https://github.com/containerd/containerd/releases/download/v$version/containerd-$version-windows-amd64.tar.gz -o containerd.tar.gz
    tar.exe xvf containerd.tar.gz

    .\bin\containerd.exe config default | Out-File .\bin\config.toml -Encoding ascii
    .\bin\containerd.exe --register-service
    Start-Service containerd

- name: Setup BuildKit
  run: |
    $version = "v0.13.1"
    curl.exe -L https://github.com/moby/buildkit/releases/download/$version/buildkit-$version.windows-amd64.tar.gz -o buildkit.tar.gz
    tar.exe xvf buildkit.tar.gz
    
    .\bin\buildkitd.exe --register-service --debug --containerd-worker=true --service-name buildkitd
    Start-Service buildkitd

- name: Setup Docker Buildx
  run: |
    $version = "v0.13.1"
    curl.exe -L https://github.com/docker/buildx/releases/download/$version/buildx-$version.windows-amd64.exe -o $env:ProgramData\Docker\cli-plugins\docker-buildx.exe

- uses: docker/setup-buildx-action@v3.2.0
  with:
    driver: remote
    endpoint: npipe:////./pipe/buildkitd

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@profnandaa, @iankingori, @lucillex, please provide an update or close this issue.

[ntrappe-msft]

@pl4nty This is great, thank you! If you want, you can submit a PR for your script under our helpful_tools folder so we can add more visibility.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@profnandaa, @iankingori, @lucillex, please provide an update or close this issue.

[profnandaa]

// Update: Buildkit 0.13.0 is now released with windows binaries - Release v0.13.0 · moby/buildkit (github.com) 🎉 Thank you all who have continued to test the releases, WCOW support is still experimental but we are on the way to stabilization. We will appreciate any feedback; this will aid in our prioritization especially on closing the Linux/Windows feature-parity gap. Feel free to open the issue at moby/buildkit.

// UPDATE: Stabilization efforts still going on. We have had several releases since v0.13.0, now at v0.14.1. Aiming for GA by September 2024. Feel free to give it a try and open any issues on moby/buildkit.

[FrankRichterAnsys]

If you want to try out HyperV Isolation, BuildKit master branch (as of moby/buildkit#4364) can pass through the necessary options from its own buildkit.toml config file. I haven't tested this, but it may be as simple as passing the SandboxIsolation=1 flag, similar to what containerd/cri does:

[worker.containerd.runtime.options]
  SandboxIsolation = "1"
  ScaleCpuLimitsToSandbox = true

I was trying that, but it didn't work for me.
Trying to build an image on a Windows 11 machine "FROM" the servercore:ltsc2019-amd64 image always ends in The container operating system does not match the host operating system, even with quite recent containerd (1.7.18) and buildkit (v0.14.1) versions.
I'd be grateful for suggestions for other things to try...

[TBBle]

Drat, so either that change wasn't correct, wasn't sufficient, or it didn't flow through into hcsshim for some reason. I'm not sure if containerd will have any useful logs for this and I'm not sure if the relevant options are ever written to disk before being passed to hcsshim.

[slonopotamus]

See instructions for easy experimental way to set up building & running Windows containers using nerdctl and BuildKit.

[FrankRichterAnsys]

See instructions for easy experimental way to set up building & running Windows containers using nerdctl and BuildKit.

These work well enough if the host Windows matches the container Windows.
In my case, however, I want to use different Windows versions - which requires HyperV isolation.
The comment I quoted is explicitly about that, and the contained suggestion was what "it didn't work for me" is referring to.

[slonopotamus]

I'm talking about the topic in general - the way to use BuildKit for building containers on Windows.

[jkroepke]

I tried to build a https://github.com/microsoft/windows-host-process-containers-base-image/ which is considered as special.

While build looks fine, exporting the images causes unknown errors:

#7 exporting to image
#7 exporting layers
#7 exporting layers 0.2s done
#7 ERROR: failed to commit 5am92igktp6scjpj0kc7ejaj3 to oumlt8yasx34v7sb7a31zzo9f during finalize: failed to reimport snapshot: hcsshim::ImportLayer failed in Win32: Cannot create a file when that file already exists. (0xb7): unknown
------
 > exporting to image:
------
ERROR: failed to solve: failed to commit 5am92igktp6scjpj0kc7ejaj3 to oumlt8yasx34v7sb7a31zzo9f during finalize: failed to reimport snapshot: hcsshim::ImportLayer failed in Win32: Cannot create a file when that file already exists. (0xb7): unknown

Ref: https://github.com/prometheus-community/windows_exporter/actions/runs/10078511673/job/27863581327#step:17:266

[profnandaa]

@jkroepke -- if you don't mind, could you please open this issue on https://github.com/moby/buildkit so that we track it properly.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@profnandaa, @iankingori, @lucillex, please provide an update or close this issue.

[jkroepke]

@profnandaa moby/buildkit#5196

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@profnandaa, @iankingori, @lucillex, please provide an update or close this issue.

[henryborchers]

I'm getting excited with every bit of news that shows that we are getting closer to being able to use Buildkit to build Windows container. I cannot wait to be able to use RUN with --cache or --secret. I see that there are commits being added to buildkit to accommodate windows containers but I'm not sure where everything is at. From what I can tell is that Windows containers with buildkit work for some specific cases but it's still very experimental.

[profnandaa]

@henryborchers -- thanks for the feedback. Also tells us the new features users are looking forward to. Our first priority right now was to have full parity with the classic builder, and then move to the nice parts. Please swing by the moby/buildkit repo and open or upvote any issues that are important to your usecase.

[slonopotamus]

I doubt there will be high adoption until there is this.

[TBBle]

And for implementing Windows in builder-next to be viable, we need first-class containerd support on Windows (as we don't support the runc executor), i.e. moby/moby#47955 and packaging/distribution decisions.

In the meantime, a nerdctl/c8d/buildkit stack Windows installer is our best chance for wider adoption and testing, as you're well aware. ^_^