BUILDKIT ignores "insecure-registries". server gave HTTP response to HTTPS client. x509: certificate error #1642
Comments
|
Hm, I'm really struggling to reproduce this actually. I can seem to get something very similar to this if the target image doesn't exist - then when using the base image in a Could you please share (redact any confidential information):
|
|
Can you also post the output of |
|
Hello @crazy-max and @jedevc! docker build -t some-tag .
docker build -t some-tag --build-arg VAR1 .
docker build -t some-tag --pull --build-arg VAR1 .Here are our versions below. docker buildx ls |
|
Getting exactly the same error. Using a Docker registry on a Raspberry Pi 4. |
|
Another +1, didn't realize this was the same issue. |
|
Does anyone here have a reproducible example? I've managed to repro in the case if "the target image doesn't exist" - but at least from the above, it looks like people are having other issues. If anyone could share a reproducible example (e.g. a docker compose setup, or a github repo, or anything really), or even just indicate what the registry used is and how it's setup would massively help in tracking this down. Without that, it's kind of just guessing in the dark, there's so many different setups and registry combinations, that it's impossible to just try them all 😢 |
|
@jedevc, thanks a lot for your help. I have this GitHub repo. It's not a MRE, but I hope it helps. Use the build.sh and then publish.sh scripts. Specify the custom Docker Registry host as a parameter. Also, you might want to comment out some of the lines in these files to avoid building and pushing more than what's necessary to reproduce this issue. My Docker Registry runs on a Raspberry Pi and I'm building and pushing from a Mac M1, if that info helps. |
|
Happens for our devs as well. Interestingly our jenkins servers are not hitting this issue (even tho I just upgraded OS, hence docker on them a week ago). |
|
We are seeing this in our CI which builds many containers in parallel using DinD. We have a local pull-through cache which currently is HTTP only for reasons. What is interesting is that the problem is intermittent. The same builder will pull the base image (specified in the For example, a success: and a failure on the same base image, using the same build command and Dockerfile: This had worked reliably before we switched to buildkit. Our server's "insecure-registries": [ "http://registrycache.<redacted>.com:5001"],
"registry-mirrors": ["http://registrycache.<redacted>.com:5001"],(and yes I know and the DinD service container entrypoint shares the same config: services:
- name: docker:24-dind
alias: docker
entrypoint: ["dockerd-entrypoint.sh"]
command:
[
"--insecure-registry",
"registrycache.<redacted>.com:5001",
"--registry-mirror",
"http://registrycache.<redacted>.com:5001",
]I will see if I can narrow down a self-contained MRE but for now I will have to disable buildkit in my environment. |
|
I'm afraid at the moment there is only one option - to set |
|
The last report from the maintainers here was that they couldn't reproduce the specific error (only related ones), so it's not exactly fair to characterize them as not caring about it; they're a fairly busy team and likely just haven't had a chance to get back to this yet and try to reproduce again. 😅 ❤️ This happens to be something I ran into recently, so I was able to get the same error with the following minimal reproducer for the two related problems here (insecure-registry from Docker not applying to BuildKit and insecure registry-mirror from Docker in BuildKit): 👀 # set up a registry and start up Docker-in-Docker
docker network create --internal foo # internal so Docker/BuildKit can't hit the internet and we can be certain it's using our registry/mirror for sure
docker run -dit --rm --network foo --name foo-registry registry
docker run -dit --rm --network foo --name foo-dind --privileged docker:dind dockerd --insecure-registry foo-registry:5000 --registry-mirror http://foo-registry:5000
# validate our configuration
docker exec foo-dind docker info | grep -B2 foo-registry
# running crane on the host network so we can copy in an image to test/run with
registry="$(docker container inspect --format '{{ .NetworkSettings.Networks.foo.IPAddress }}' foo-registry)"
docker run --rm --network host gcr.io/go-containerregistry/crane cp --insecure docker.io/library/hello-world:linux "$registry:5000/library/hello-world:linux" &> /dev/null
# make sure it works with Docker
docker exec foo-dind docker run --rm foo-registry:5000/library/hello-world:linux > /dev/null
docker exec foo-dind docker rmi foo-registry:5000/library/hello-world:linux
docker exec foo-dind docker run --rm hello-world:linux > /dev/null
docker exec foo-dind docker rmi hello-world:linux
# test BuildKit (both of these should fail)
printf 'FROM foo-registry:5000/library/hello-world:linux\nRUN ["/hello"]' | docker exec -i foo-dind docker buildx build --no-cache -
printf 'FROM hello-world:linux\nRUN ["/hello"]' | docker exec -i foo-dind docker buildx build --no-cache -Failing Output:c0e1321ec9a61c25f72f7c8a6fe44324594e70dac5b09c2dbf8607668029c286
94a25edc05815f711bf175f1ed932040f5506b87a206bc6c0707bb21b8d9510d
a934704a869b3ebcf72e59d13b37b72475ba15e967610c3bd73e75530d044110
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
Experimental: false
Insecure Registries:
foo-registry:5000
127.0.0.0/8
Registry Mirrors:
http://foo-registry:5000/
Unable to find image 'foo-registry:5000/library/hello-world:linux' locally
linux: Pulling from library/hello-world
719385e32844: Pulling fs layer
719385e32844: Verifying Checksum
719385e32844: Download complete
719385e32844: Pull complete
Digest: sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Status: Downloaded newer image for foo-registry:5000/library/hello-world:linux
Untagged: foo-registry:5000/library/hello-world:linux
Untagged: foo-registry:5000/library/hello-world@sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Deleted: sha256:9c7a54a9a43cca047013b82af109fe963fde787f63f9e016fdc3384500c2823d
Deleted: sha256:01bb4fce3eb1b56b05adf99504dafd31907a5aadac736e36b27595c8b92f07f1
Unable to find image 'hello-world:linux' locally
linux: Pulling from library/hello-world
719385e32844: Pulling fs layer
719385e32844: Verifying Checksum
719385e32844: Download complete
719385e32844: Pull complete
Digest: sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Status: Downloaded newer image for hello-world:linux
Untagged: hello-world:linux
Untagged: hello-world@sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Deleted: sha256:9c7a54a9a43cca047013b82af109fe963fde787f63f9e016fdc3384500c2823d
Deleted: sha256:01bb4fce3eb1b56b05adf99504dafd31907a5aadac736e36b27595c8b92f07f1
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 74B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/hello-world:linux
#3 ERROR: failed to do request: Head "https://foo-registry:5000/v2/library/hello-world/manifests/linux?ns=docker.io": http: server gave HTTP response to HTTPS client
------
> [internal] load metadata for docker.io/library/hello-world:linux:
------
Dockerfile:1
--------------------
1 | >>> FROM hello-world:linux
2 | RUN ["/hello"]
--------------------
ERROR: failed to solve: hello-world:linux: failed to do request: Head "https://foo-registry:5000/v2/library/hello-world/manifests/linux?ns=docker.io": http: server gave HTTP response to HTTPS clientIt's noted in some of the previous comments, and it's not mentioned on https://docs.docker.com/engine/reference/commandline/dockerd/#insecure-registries, but https://docs.docker.com/registry/insecure/ does suggest that When I update Successful Output:1b4f558bf37fbd97f0543da03c39196b838cf5fe477f5c056987228377efe92d
0ae051678f3216a64e55bd2439237e22060c1d9afd2a425aed2c5a9a5c6ef483
9f5072a5b80e41ef027c731464cc07aebd6ff559a9f40266a15791f63cc41ffc
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
Experimental: false
Insecure Registries:
foo-registry:5000
127.0.0.0/8
Registry Mirrors:
http://foo-registry:5000/
Unable to find image 'foo-registry:5000/library/hello-world:linux' locally
linux: Pulling from library/hello-world
719385e32844: Pulling fs layer
719385e32844: Verifying Checksum
719385e32844: Download complete
719385e32844: Pull complete
Digest: sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Status: Downloaded newer image for foo-registry:5000/library/hello-world:linux
Untagged: foo-registry:5000/library/hello-world:linux
Untagged: foo-registry:5000/library/hello-world@sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Deleted: sha256:9c7a54a9a43cca047013b82af109fe963fde787f63f9e016fdc3384500c2823d
Deleted: sha256:01bb4fce3eb1b56b05adf99504dafd31907a5aadac736e36b27595c8b92f07f1
Unable to find image 'hello-world:linux' locally
linux: Pulling from library/hello-world
719385e32844: Pulling fs layer
719385e32844: Verifying Checksum
719385e32844: Download complete
719385e32844: Pull complete
Digest: sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Status: Downloaded newer image for hello-world:linux
Untagged: hello-world:linux
Untagged: hello-world@sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
Deleted: sha256:9c7a54a9a43cca047013b82af109fe963fde787f63f9e016fdc3384500c2823d
Deleted: sha256:01bb4fce3eb1b56b05adf99504dafd31907a5aadac736e36b27595c8b92f07f1
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 74B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/hello-world:linux
#3 DONE 0.1s
#4 [1/2] FROM docker.io/library/hello-world:linux@sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
#4 resolve docker.io/library/hello-world:linux@sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d 0.0s done
#4 extracting sha256:719385e32844401d57ecfd3eacab360bf551a1491c05b85806ed8f1b08d792f6 done
#4 sha256:719385e32844401d57ecfd3eacab360bf551a1491c05b85806ed8f1b08d792f6 2.46kB / 2.46kB done
#4 sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d 2.07kB / 2.07kB done
#4 sha256:7e9b6e7ba2842c91cf49f3e214d04a7a496f8214356f41d81a6e6dcad11f11e3 525B / 525B done
#4 sha256:9c7a54a9a43cca047013b82af109fe963fde787f63f9e016fdc3384500c2823d 1.47kB / 1.47kB done
#4 DONE 0.1s
#5 [2/2] RUN ["/hello"]
#5 0.218
#5 0.218 Hello from Docker!
#5 0.218 This message shows that your installation appears to be working correctly.
#5 0.218
#5 0.218 To generate this message, Docker took the following steps:
#5 0.218 1. The Docker client contacted the Docker daemon.
#5 0.218 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
#5 0.218 (amd64)
#5 0.218 3. The Docker daemon created a new container from that image which runs the
#5 0.218 executable that produces the output you are currently reading.
#5 0.218 4. The Docker daemon streamed that output to the Docker client, which sent it
#5 0.218 to your terminal.
#5 0.218
#5 0.218 To try something more ambitious, you can run an Ubuntu container with:
#5 0.218 $ docker run -it ubuntu bash
#5 0.218
#5 0.218 Share images, automate workflows, and more with a free Docker ID:
#5 0.218 https://hub.docker.com/
#5 0.218
#5 0.218 For more examples and ideas, visit:
#5 0.218 https://docs.docker.com/get-started/
#5 0.218
#5 DONE 0.2s
#6 exporting to image
#6 exporting layers 0.0s done
#6 writing image sha256:ccf2e23200821ad83198e15400f250a69c4d9eabd8948eeaedb0dd2131509dc1 done
#6 DONE 0.0s
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 100B done
#2 DONE 0.0s
#3 [internal] load metadata for foo-registry:5000/library/hello-world:linux
#3 DONE 0.0s
#4 [1/2] FROM foo-registry:5000/library/hello-world:linux@sha256:726023f73a8fc5103fa6776d48090539042cb822531c6b751b1f6dd18cb5705d
#4 CACHED
#5 [2/2] RUN ["/hello"]
#5 0.220
#5 0.220 Hello from Docker!
#5 0.220 This message shows that your installation appears to be working correctly.
#5 0.220
#5 0.220 To generate this message, Docker took the following steps:
#5 0.220 1. The Docker client contacted the Docker daemon.
#5 0.220 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
#5 0.220 (amd64)
#5 0.220 3. The Docker daemon created a new container from that image which runs the
#5 0.220 executable that produces the output you are currently reading.
#5 0.220 4. The Docker daemon streamed that output to the Docker client, which sent it
#5 0.220 to your terminal.
#5 0.220
#5 0.220 To try something more ambitious, you can run an Ubuntu container with:
#5 0.220 $ docker run -it ubuntu bash
#5 0.220
#5 0.220 Share images, automate workflows, and more with a free Docker ID:
#5 0.220 https://hub.docker.com/
#5 0.220
#5 0.220 For more examples and ideas, visit:
#5 0.220 https://docs.docker.com/get-started/
#5 0.220
#5 DONE 0.2s
#6 exporting to image
#6 exporting layers done
#6 writing image sha256:8c9eada21969c3a185c994222d584698cee2ad8015347dd2df38a3c8a6dce78b done
#6 DONE 0.0sI'm not sure why the example above with (There is arguably a bug here where BuildKit is interpreting Docker's configuration in a slightly different way from Docker itself, but it's not unfair for BuildKit to have not wanted to implement the same "which type of insecure?" heuristic as Docker did because it is not the most reliable/efficient thing to try and detect correctly, which is why I think this might be more of a documentation issue since Docker doesn't mind the protocol prefix and BuildKit works correctly with it. 😅) |
|
To highlight from @tianon 's comment: If you hit this error with |
|
So as it turns out, apparently using Also, apparently |
|
To anyone coming across this in the future - this issue should now be resolved by moby/moby#45992, which will be released as part of moby 25.0. |
|
Hey @jedevc, I just tried docker 25 and when using in dind (started with both Is this meant to work following your change or am I confused about all of this? Some versioning information: I see also a dependency to buidkit when building: The only way I found to make it work is to use the following buildkit.toml passed to [registry."docker.io"]
mirrors = ["dockerproxy:6000"]
[registry."dockerproxy:6000"]
http = true |
|
As 25.0, it's still not fixed. |
|
@victornoel @woshikid The case described above by @victornoel (#1642 (comment)) is expected; looking at the build progress output, that is a build running in a container-builder; When creating a container builder, you're creating an isolated BuildKit instance that runs inside a container; such a container has no direct relation with the daemon it's running on, and does not inherit configuration from the daemon (only using configuration options as passed by the user when creating the container / containerised builder). The Moby pull-request that closed this ticket (moby/moby#45992) fixed this issue when using the BuildKit builder that's embedded in the Docker Engine Daemon; this builder is configured using the daemon-configuration options set for the Docker Engine Daemon, and therefore will be configured with the insecure registry settings that are set in the daemon config. Perhaps it's possible for I should add that Docker Engine v25 allows you to use containerd's image store instead of the "graph-driver" store for containers and images; https://docs.docker.com/storage/containerd/. When using the Docker Daemon with the containerd image store ("snapshotters") enabled, the Docker Engine gains support for storing multi-arch / multi-platform images. If you're creating a container-builder for that reason (to be able to build multi-arch images with multiple architectures), it may be worth giving that option a try. I should add some disclaimers;
|
|
@thaJeztah thank you for the explanation, I knew I was most certainly missing something :P so when is this buildkit embedded in docker used? When I call if I understand correctly what you are saying, I can use containred image store + Sorry to turn this ticket into a support thread :P |
|
Ok, it seems I "just" need not to execute Now I "just" need to understand how to enable containerd image store from the command line because I don't have much control on the daemon.json file unfortunately. |
|
So The builder that's used is configurable; by default, If you type docker builder ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
custom-net-build docker-container
custom-net-build0 desktop-linux inactive
multi-arch-builder * docker-container
multi-arch-builder0 desktop-linux inactive
default docker
default default running v0.12.4+3b6880d2a00f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
desktop-linux docker
desktop-linux desktop-linux running v0.12.4+3b6880d2a00f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
remote docker
remote remote running v0.10.6+d52b2d5 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6You can switch the builder to use using docker builder use desktop-linuxAfter that, it should show as the currently selected builder; docker builder ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
custom-net-build docker-container
custom-net-build0 desktop-linux inactive
multi-arch-builder docker-container
multi-arch-builder0 desktop-linux inactive
default docker
default default running v0.12.4+3b6880d2a00f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
desktop-linux * docker
desktop-linux desktop-linux running v0.12.4+3b6880d2a00f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
remote docker
remote remote running v0.10.6+d52b2d5 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6You can also inspect the builder to get more information about the features it provides. In my case, I have the containerd image-store integration enabled, and the builder shows that it uses the docker builder inspect desktop-linux
Name: desktop-linux
Driver: docker
Last Activity: 2024-01-26 15:08:49 +0000 UTC
Nodes:
Name: desktop-linux
Endpoint: desktop-linux
Status: running
Buildkit: v0.12.4+3b6880d2a00f
Platforms: linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
Labels:
org.mobyproject.buildkit.worker.containerd.namespace: moby
org.mobyproject.buildkit.worker.containerd.uuid: 1cc22aa9-1955-4d97-96a7-b6153b61614b
org.mobyproject.buildkit.worker.executor: containerd
org.mobyproject.buildkit.worker.hostname: docker-desktop
org.mobyproject.buildkit.worker.moby.host-gateway-ip: 192.168.65.254
org.mobyproject.buildkit.worker.network: host
org.mobyproject.buildkit.worker.selinux.enabled: false
org.mobyproject.buildkit.worker.snapshotter: overlayfs
GC Policy rule#0:
All: false
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
Keep Duration: 48h0m0s
Keep Bytes: 2.764GiB
GC Policy rule#1:
All: false
Keep Duration: 1440h0m0s
Keep Bytes: 20GiB
GC Policy rule#2:
All: false
Keep Bytes: 20GiB
GC Policy rule#3:
All: true
Keep Bytes: 20GiB |
|
Thank you again for sharing all this knowledge 👍 |
|
For people using buildkitd in docker-container here is simple way how to get it working with http: root = "/var/lib/buildkit"
insecure-entitlements = [ "network.host", "security.insecure" ]
[log]
format = "text"
[registry."YOUR_HOST:YOUR_PORT"]
http = true
insecure = trueAnd then run this bash init_docker_builder() {
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1
docker run --rm --privileged -d --name buildx_buildkit_my_builder0 --volume ./buildkitd.toml:/buildkitd.toml --network host moby/buildkit:latest --config /buildkitd.toml
docker run --privileged --rm tonistiigi/binfmt --install all
docker buildx create --name my_builder --platform=linux/arm64,linux/amd64 --driver=docker-container
export BUILDKIT_HOST=docker-container://buildx_buildkit_my_builder0
export BUILDX_BUILDER=my_builder
docker buildx install
docker buildx use --builder my_builder --default --global
}
init_docker_builderI faced error |
|
Any reason you're not just using the |
|
@tianon |
|
I think that's just |
Oh, I didn't know it. Now I do. Thank you! |
Sorry, what? Where do you see this? Buildx defaults to using latest release of buildkit when it deploys container builders (sometimes we give couple of extra days of testing after a release before switching buildx default, but that is an exception). |
|
@tonistiigi this seems like some issue with Arch packaging system. If I accidentally called things not by their names, I'm sorry |
|
Luckily I did btrfs snapshot before update so I can investigate a bit I entered Since I update my system quite often, it seems like docker packaged in arch is really lagging behind and only now they decided to update it. |
|
@nikelborm That is the buildkit that is embedded into docker daemon(see driver==docker). You updated docker to 25 and that comes with buildkit v0.12.5 . If you use |
|
Just installed Docker 26.1.2 in an VM with Oracle Linux 8.9 and the problem remains. Executing the command "docker build ." results in failure if the base image is in an insecure registry. Looks like the "insecure-registries" property in the file "/etc/docker/daemon.json" is still being ignored. I can manually pull the same image without problem. Using "export DOCKER_BUILDKIT=0" still works. docker builder version |
|
I'm still getting "http: server gave HTTP response to HTTPS client" with Docker version 27.1.1, build 6312585. |
|
Same. Issue is not fixed |
|
I fixed using |
|
It seems to be fixed in v27.1.2 Update: |
|
It seems to be broken again with v27.3.1 |
Hello! After upgrading to Docker Engine v23.0.1 our CI process started failing due to errors accessing internal Harbor registry.
As I see BuildKit became default in versions 23+.
Unfortunately Docker BuildKit disregards configuration
insecure-registriesin/etc/docker/daemon.json.All attempts to build image fail with the following errors. Needless to say that address of our Harbor registry was added to
insecure-registries. It used to work normally with previous versions of Docker.If we place commands like
docker pull harbor.company.local/hub/library/node:16.17.0-alpine3.16`before
docker build, all images are pulled successfully, which is proves that docker daemon is configured correctly and all other processes keep working.Thanks to the discussion in #1613, we managed to implement a workaround with setting
BUILDKIT_NO_CLIENT_TOKEN=true. 👍For the sake of completeness, I have to mention another issue #1589, which is related to the same or very similar problem I believe.
Another workaround was to
export DOCKER_BUILDKIT=0.When this variable is set, the following message appear in the logs, which is expected, I'm convinced.
Anyway, we still shall appreciate if you can fix this behaviour and make BuildKit rely on
insecure-registriesin/etc/docker/daemon.json.The text was updated successfully, but these errors were encountered: