Image pulls from embedded registry fail if --bind-address is set (and not 127.0.0.1)

[achristianson]

Environmental Info:
K3s Version: v1.30.0+k3s1

Node(s) CPU architecture, OS, and Version: Linux viking 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux

Cluster Configuration: 3 servers 4 agents with embedded registry enabled

Describe the bug:

When I have --bind-address set on my masters, pods scheduled on the masters cannot pull from 127.0.0.1:6443 because it isn't bound to 127.0.0.1 --it's bound to a different IP.

Steps To Reproduce:

Set up a cluster with embedded registry. Set --bind-address to anything other than 127.0.0.1.

• Installed K3s:

Expected behavior:

When a pod is scheduled, k3s is expected to pull from the correct address when --bind-address is set.

Actual behavior:

When a pod is scheduled, k3s tries to pull from 127.0.0.1 but since k3s is bound to a different IP, it gets connection refused.

Additional context / logs:

[brandond]

I can take a look at addressing this when I bump the embedded spegel version. I will note that this currently only affects servers, as agents do not support the --bind-address flag, so the listener address cannot be configured. Might want to promote that to an agent flag as well I guess.

[achristianson]

Might want to promote that to an agent flag as well I guess.

Having the bind flag on agents would definitely be desirable (and make the overall config options more consistent).

[brandond]

there are already some changes staged in this space in

• Enable serving supervisor metrics #10019

[brandond]

One of the problems here is that spegel needs to be able to identify which requests are from the local containerd instance and should proxy to other nodes, as opposed to those from other nodes that should not be reproxied. Right now this is done by identifying requests to localhost, and I can't think of another really good way to do that without opening more ports. We may just need to set it up so that it binds to localhost plus the configured bind address.

[brandond]

OK, I've modified that PR to always bind the supervisor on the loopback addresses, in addition to the requested address

[brandond]

Note that the embedded registry still will not work on nodes where IPv6 is the primary address family due to #9897

[aganesh-suse]

Validated on master branch with version v1.30.2-rc2+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
embedded-registry: true
bind-address: x.x.x.x

registries.yaml:

mirrors:
  private.registry.com:
    endpoint:
      - private.registry.com
  docker.io:
    endpoint:
      - private.registry.com      
  k8s.gcr.io:
    endpoint:
      - private.registry.com      
configs:
  private.registry.com:
    auth:
      username: <username>
      password: <password>
    tls:
      ca_file: /home/ubuntu/ca.pem

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s && sudo cp registries.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.30.2-rc2+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

Replication Results:

• k3s version used for replication:

$ k3s -v
k3s version v1.30.1+k3s1 (80978b5b)
go version go1.22.2

$ journalctl -xeu k3s-agent | grep 'received image event' 

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.30.2-rc2+k3s1 (b4d4ed8f)
go version go1.22.4

Sample spegel logs from the journal logs on the agent node:

$ journalctl -xeu k3s-agent | grep 'received image event'  
Jun 14 17:50:51 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:51Z" level=info msg="spegel 2024/06/14 17:50:51 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"docker.io/rancher/mirrored-pause:3.6@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c\" \"type\"=\"CREATE\""
Jun 14 17:50:51 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:51Z" level=info msg="spegel 2024/06/14 17:50:51 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"docker.io/rancher/mirrored-pause@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c\" \"type\"=\"CREATE\""
Jun 14 17:50:55 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:55Z" level=info msg="spegel 2024/06/14 17:50:55 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/mytestcontainer:unprivileged@sha256:7e418465981575a9abef4ee16a80c562a2d2d171e591c1475c38347ef3ec2a72\" \"type\"=\"CREATE\""
Jun 14 17:50:55 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:55Z" level=info msg="spegel 2024/06/14 17:50:55 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/mytestcontainer@sha256:7e418465981575a9abef4ee16a80c562a2d2d171e591c1475c38347ef3ec2a72\" \"type\"=\"CREATE\""
Jun 14 17:51:29 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:51:29Z" level=info msg="spegel 2024/06/14 17:51:29 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/nginx:latest@sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\" \"type\"=\"CREATE\""
Jun 14 17:51:29 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:51:29Z" level=info msg="spegel 2024/06/14 17:51:29 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/nginx@sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\" \"type\"=\"CREATE\""