Failure to pull image during build via FROM behind a proxy

[ghost]

Description

nerdctl pull works. nerdctl build for the same image name in FROM doesn't work:

$ env | grep -i proxy
NO_PROXY=localhost,127.0.0.1,<censored>
http_proxy=http://<censored>:80
HTTPS_PROXY=http://<censored>:80
HTTP_PROXY=http://<censored>:80

$ colima start --runtime containerd
INFO[0000] starting colima
INFO[0000] runtime: containerd
INFO[0000] preparing network ...                         context=vm
INFO[0000] creating and starting ...                     context=vm
INFO[0031] provisioning ...                              context=containerd
INFO[0031] starting ...                                  context=containerd
INFO[0037] done

$ cat Dockerfile
FROM node:14-alpine

$ nerdctl build -t test .
[+] Building 30.1s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                       0.1s
 => => transferring dockerfile: 57B                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                          0.1s
 => => transferring context: 2B                                                                                                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/node:14-alpine                                                                                   30.0s
------
 > [internal] load metadata for docker.io/library/node:14-alpine:
------
Dockerfile:1
--------------------
   1 | >>> FROM node:14-alpine
   2 |
--------------------
error: failed to solve: node:14-alpine: failed to do request: Head "https://registry-1.docker.io/v2/library/node/manifests/14-alpine": dial tcp 44.205.64.79:443: i/o timeout
FATA[0030] unrecognized image format
FATA[0030] exit status 1

$ nerdctl pull node:14-alpine
docker.io/library/node:14-alpine:                                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0a37794193aa27ba8af16a2da0998a3e0c551fcc4ebec2ce95b1615032daf820:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:78673749e0d950695922b7136b37398ee45b55e3658937326f6892312db11ed1: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:798752c1e2a0bc426a575910df9dd7bebf6a5993f14129fcea71b39137e6aef3:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:16677254a57ae0b50f9cc248748074ac0a92fd7f72dec5bc8398e30899545a17:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3af31e9dbfcf77f14c1b1c67ae02f0896e7d5426589108db17c80778439c9f1a:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c60ddf63b62240f708d8e614432f811fa775fdc3ea506e2a6bbcca3df6f3ab50:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 16.6s                                                                    total:  40.5 M (2.4 MiB/s)

Additionally, if I colima ssh and try curl for the failed IP address - the HTTP request works. Only the FROM image pull during nerdctl build seems to be affected, i.e. it's not using the proxy settings.

Version

Colima Version:
colima version 0.4.4
git commit: 8bb1101
runtime: containerd
arch: x86_64
client: v0.20.0
server: v1.5.8

Lima Version: not sure how to look that up, I just installed colima via brew today.

Qemu Version: not sure.

Operating System

• macOS Intel
• macOS M1
• Linux

Reproduction Steps

in the description

Expected behaviour

nerdctl build not to fail on image pull

Additional context

No response

[abiosoft]

Nerdctl behaves a bit differently for the local image cache.

Can you try building the image in the buildkit namespace and see if it gets pulled successfully?

nerdctl --namespace buildkit build -t test .

[ghost]

Same result, sorry:

$ nerdctl --namespace buildkit build -t test .
[+] Building 30.0s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                      0.0s
 => => transferring dockerfile: 57B                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                           0.0s
 => ERROR [internal] load metadata for docker.io/library/node:14-alpine                                                                                  30.0s
------
 > [internal] load metadata for docker.io/library/node:14-alpine:
------
Dockerfile:1
--------------------
   1 | >>> FROM node:14-alpine
   2 |
--------------------
error: failed to solve: node:14-alpine: failed to do request: Head "https://registry-1.docker.io/v2/library/node/manifests/14-alpine": dial tcp 44.205.64.79:443: i/o timeout
FATA[0030] exit status 1

While nerdctl pull for the same image still works.

[ghost]

Any chance the Minikube I've got installed might somehow contaminate Colima's environment?

[ghost]

Tried all the same stuff after switching to Docker runtime and the official Docker CLI. Same result for trying to build - I/O timeout, but now docker pull node:14-alpine also fails with an I/O timeout:

$ docker pull node:14-alpine
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

That is, with nerdctl + containerd results were much better.

Looking into Docker's docs, http proxies must be configured in the process environment for the Docker service. Not sure where to look for that in Colima's VM, but I found that /etc/environment does set all the right proxy env variables (mirrors the ones I have in my host machine's env).

[ghost]

Solved it:

After modifying /etc/rc.conf as described in #161 (comment) , both Docker + Docker CLI and containerd + nerdctl successfully passed all scenarios.