Support for nerdctl build on Windows

[TBBle]

Trivially bring up BuildKit support in nerdctl. It was basically all-there already, just disabled.

Resolves #627 assuming you have Buildkitd functional on Windows, of course.

Resolves #2173 which just turned out to be a warning that no error had occurred.

Tested with a trivial Dockerfile that was already working with buildctl build and nerdctl run, and buildkitd set up as a service matching the existing rootful instructions (i.e. configured to be in the default containerd namespace instead of the default buildkit containerd namespace) per moby/buildkit#616 (comment).

nerdctl build --buildkit-host "npipe:////./pipe/buildkitd" . -t docker.io/tbble/supersimpledocker
nerdctl build . -t docker.io/tbble/supersimpledocker

Possible outstanding things:

Removing the CLI defaults, while correct because they were not actually used, isn't super-nice from the UX perspective. It might be better to report which paths will be auto-detected, but that requires some further refactoring, and might also be too noisy? Simply saying <AUTODETECTED> or something would work, perhaps.

I've only tested this on Windows, and haven't checked if the test suite covers the buildkit socket detection code on Linux/FreeBSD, so it's possible I mangled something when moving that code. And I only tested the default namespace, but since Windows buildkitd doesn't change its default listening address for different containerd namespaces, this seems fine for now.

Actually, the buildkitd source doesn't appear to use different sockets for different containerd namespaces on Linux either, AFAICT. Is that logic actually syncing with a systemd unit configuration or something, rather than buildkitd's logic? It would make sense for buildkitd to work this way, at least for the containerd worker, it just doesn't seem to, or I didn't find the implementation when I went looking.

[TBBle]

Okay, I broke something when moving the socket-finding logic, the rootless tests are panicking. I'll look into that later today now.

[AkihiroSuda]

Needs rebase

[iankingori]

@TBBle do you have some free cycles to complete this? buildkit v0.13.0-rc2 includes the windows buildkitd binary and would be great to have nerdctl working as well. Happy to pick this up if you are swamped

[TBBle]

I can quickly rebase it tonight, as it looks like the conflicting changes are simply adjacent to my changes, rather than actual conflicting changes. The requested changes don't look too bad, so I can probably get those done too tonight.

[TBBle]

Okay, I've rebased it, and we'll let CI confirm I didn't break anything. I am not currently in a position to test it locally tonight though.

Also, I added b5859e1 for #2587 (review) but it's mildly out-of-scope and slightly touches a handful of files so happy to have that split out into a separate PR if maintainers prefer. That one also makes Go 1.19 the minimum, but go.mod already states Go 1.20 as minimum so I assume that's probably fine. (I noticed that Go 1.21 actually makes the go.mod version a hard minimum, rather than a strong recommendation, which is nice.)

A future TODO is to get a buildkitd running on Windows under CI, which should then automatically try to run the existing build test, which ought to work as-is...

Edit: Test failure looks unrelated, and the logs show that (unlike master) the build-test skip logs the expected npipe:////./pipe/buildkitd and doesn't skip due to platform support, but after trying buildctl. FreeBSD doesn't seem to even try to run TestBuild, so I can't confirm the correct socket (in /var/run) and TestBuild passes in normal integration and skips in rootless, just like in the main branch, with the same attempted socket names in rootless mode.

(So an extra future TODO to consider: Rig up buildkitd to work with rootless in the test suite, perhaps. Although maybe there's no value in that except testing the relevant socket.)

So yeah, apart from what looks like a spurious failure in one of the rootless integration passes, CI seems to say it's fine.

[iankingori]

Okay, I've rebased it, and we'll let CI confirm I didn't break anything. I am not currently in a position to test it locally tonight though.

I should have some time to test this today, thanks.

[iankingori]

I got nerdctl build to run but with a couple of problems:

1.Nerdctl sets a default windows path on container images:

Path=c:\Windows\System32;c:\Windows

This is a universal problem we've seen with Buildkit as well that we need to address but unrelated to this PR.

2.Nerdctl Build does not work when progress is auto or tty

When progress=auto which defaults to tty we get the following:

Nerdctl

Command: nerdctl build . -t core:1

time="2024-02-28T23:10:18-08:00" level=debug msg="Choosing the buildkit host \"npipe:////./pipe/buildkitd\", candidates=[npipe:////./pipe/buildkitd]"
time="2024-02-28T23:10:18-08:00" level=debug msg="Chosen buildkit host \"npipe:////./pipe/buildkitd\""
time="2024-02-28T23:10:19-08:00" level=debug msg="worker labels: map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproject.buildkit.worker.containerd.uuid:6ceace98-92bc-4a83-807b-133ad2fd9275 org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:WIN-LIQNSMAKV3L org.mobyproject.buildkit.worker.network: org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:windows]"
time="2024-02-28T23:10:19-08:00" level=debug msg="running C:\\Tools\\Buildkit\\bin\\buildctl.exe [--addr=npipe:////./pipe/buildkitd build --progress=tty --frontend=dockerfile.v0 --local=context=. --output=type=docker,name=docker.io/library/core:1 --local=dockerfile=C:\\code\\debug --opt=filename=Dockerfile]"
time="2024-02-28T23:10:19-08:00" level=fatal msg="no image was built"

Buildkit

Response:

failed to copy to tar: rpc error: code = Unknown desc = write /dev/stdout: The pipe is being closed.
6532 v0.13.0-rc2 C:\Tools\Buildkit\bin\buildctl.exe --addr=npipe:////./pipe/buildkitd build --progress=tty --frontend=dockerfile.v0 --local=context=. --output=type=docker,name=docker.io/library/core:1 --local=dockerfile=C:\code\debug --opt=filename=Dockerfile
github.com/moby/buildkit/session/filesync.writeTargetFile
        /src/session/filesync/diffcopy.go:145
github.com/moby/buildkit/session/filesync.(*fsSyncAttachable).DiffCopy
        /src/session/filesync/filesync.go:339
github.com/moby/buildkit/session/filesync._FileSend_DiffCopy_Handler
        /src/session/filesync/filesync.pb.go:392
github.com/moby/buildkit/util/grpcerrors.StreamServerInterceptor
        /src/util/grpcerrors/intercept.go:33
github.com/moby/buildkit/session.NewSession.ChainStreamServer.func6.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:49
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.StreamServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:446
github.com/moby/buildkit/session.NewSession.ChainStreamServer.func6.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:49
github.com/moby/buildkit/session.NewSession.ChainStreamServer.func6
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:58
google.golang.org/grpc.(*Server).processStreamingRPC
        /src/vendor/google.golang.org/grpc/server.go:1644
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1741
google.golang.org/grpc.(*Server).serveStreams.func1.1
        /src/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

This particular issue goes away when we switch to progress=plain

Nerdctl

Command: nerdctl build --progress=plain . -t core:1

Response:

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 176B done
#1 DONE 0.0s

#2 [internal] load metadata for mcr.microsoft.com/windows/servercore:ltsc2022
#2 DONE 0.1s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2022@sha256:8a75266be74ad7a904470e18057f6fd62055cf7028172307fefc67aff37dfd10
#4 resolve mcr.microsoft.com/windows/servercore:ltsc2022@sha256:8a75266be74ad7a904470e18057f6fd62055cf7028172307fefc67aff37dfd10 0.0s done
#4 DONE 0.1s

#4 [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2022@sha256:8a75266be74ad7a904470e18057f6fd62055cf7028172307fefc67aff37dfd10
#4 extracting sha256:d7662b0a97ab33c03d2f9da1cbaf9628276b35d8445cec2791dd15133452c1d1
#4 extracting sha256:d7662b0a97ab33c03d2f9da1cbaf9628276b35d8445cec2791dd15133452c1d1 82.4s done
#4 DONE 82.5s

#4 [1/2] FROM mcr.microsoft.com/windows/servercore:ltsc2022@sha256:8a75266be74ad7a904470e18057f6fd62055cf7028172307fefc67aff37dfd10
#4 extracting sha256:4abd29fcbfc570cbbbaefefc279496e8f41ab12aa0e2e608ffd71916ba664a81
#4 extracting sha256:4abd29fcbfc570cbbbaefefc279496e8f41ab12aa0e2e608ffd71916ba664a81 91.0s done
#4 DONE 173.5s

#5 [2/2] RUN "echo Hello Nerdctl!!"
#5 1.755 Hello Nerdctl!!
#5 DONE 2.7s

#6 exporting to docker image format
#6 exporting layers
#6 exporting layers 2.4s done
#6 exporting manifest sha256:318cf6a359222644681c0d0b561a939a70fa6bcec90fe152724b1235bb129df9 0.0s done
#6 exporting config sha256:57db8b05d1bb3c16224cc694ac90a87c05f1f27a7ef19886baa0056a9009cbfb 0.0s done
#6 sending tarball
#6 sending tarball 36.9s done
#6 DONE 39.3s
unpacking docker.io/library/core:1 (sha256:318cf6a359222644681c0d0b561a939a70fa6bcec90fe152724b1235bb129df9)...
Loaded image: docker.io/library/core:1

Short Term Fix:
We could default windows progress to plain and/or log this as a know issue

Long Term Fix:
Investigate how to pass the Windows Stdout Handle from Nerdctl to Buildkit. I suspect we need to separate the progress text stream from the tar file stream.

[TBBle]

Huh, weird, I didn't see the TTY issue in my original testing. I wonder if something changed since then, or if my test setup was so trivial it didn't trigger this. If I recall correctly, my test setup here didn't actually have anything in the context except the Dockerfile.

[AkihiroSuda]

Can we merge this and track other issues in follow-up ?

[iankingori]

Sounds good to me

[TBBle]

Rebased to main since I was going to fix the typo that turned out to not be a typo, so instead we get a trivial rebase.

[AkihiroSuda]

Thanks

[slonopotamus]

Doesn't seem to work, see containerd/containerd#10154 (comment)

[TBBle]

Oops, looks like we never opened an issue for number 2 in #2587 (comment).