cluster reboots don't preserve node ip when additional networks are attached

[lazedo]

What happened:
node ip wasn't preserved

What you expected to happen:
node ip preserved

How to reproduce it (as minimally and precisely as possible):
start cluster, attach a network to a node, restart container

Anything else we need to know?:
entrypoint script assumes head for detecting ip which doesn't match the omitted (by head) existing
the node ip in 172.18.x.x the network attached is 192.168.x.x. the ip from attached network may be listed first

  # fixup IPs in manifests ...
  curr_ipv4="$( (head -n1 <(timeout 5 getent ahostsv4 "$(hostname)") | cut -d' ' -f1) || true)"

Environment:
ubuntu-2204

• kind version: (use kind version): 0.15
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-08-08T21:15:39Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info):

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 7
  Running: 7
  Paused: 0
  Stopped: 0
 Images: 350
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-47-generic
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.35GiB
 Name: ubuntu-2104
 ID: 3BEI:3ZQB:VGSQ:67ZG:TA4Q:SYOX:UOTH:LPA5:UUFA:PAST:FRWN:KKAO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: lazedo
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

[aojea]

I don't really understand your comment about head sorry, can you be more specific and add an example?

can you paste the output of getent ahostsv4 "$(hostname)" ?

[lazedo]

curr_ipv4="$( (head -n1 <(timeout 5 getent ahostsv4 "$(hostname)") | cut -d' ' -f1) || true)"
head -n1 only gets first ip from getent ahostsv4 "$(hostname)", without head it lists all ips

root@west-worker5:/# getent ahostsv4 "$(hostname)"
192.168.61.3    STREAM west-worker5
192.168.61.3    DGRAM  
192.168.61.3    RAW    
192.168.36.3    STREAM 
192.168.36.3    DGRAM  
192.168.36.3    RAW    
172.19.0.7      STREAM 
172.19.0.7      DGRAM  
172.19.0.7      RAW    
root@west-worker5:/# 

root@west-worker5:/# head -n1 <(timeout 5 getent ahostsv4 "$(hostname)")
192.168.61.3    STREAM west-worker5
root@west-worker5:/# 

the node ip is 172.19.0.7 the ips for attached networks are 192.168.61.3 and 192.168.36.3

[aojea]

why are there 3 different ips on that container?

[BenTheElder]

What you expected to happen:

node ip preserved

so, while I think it may be a bug how we detect the IP (probably we want the one from the kind network specifically, ignoring for a moment the experimental option to override the network name), preserving the IP on reboot is not guaranteed anyhow and is not an intended goal

clusters can function fine with the IP changing, we re-roll the certs and nearly everything in the cluster should be using the node name or else kubernetes.default, not the IP.

why are there 3 different ips on that container?

docker network connect was used to add additional networks to the nodes per:

start cluster, attach a network to a node, restart container

[aojea]

docker network connect was used to add additional networks to the nodes per:

we can't support that at this moment, multi network is not a thing in Kubernetes ... yet ;)

[BenTheElder]

For an immediate workaround, depending on what you are trying to do exactly, it may be more appropriate to attach other containers to the kind network than attach kind nodes to another network.

[lazedo]

i'll try to figure out a solution to preserve the IP from the original network, don't worry. i'm using multus on some nodes and need to attach the network to the node to provide other interfaces to pods

[BenTheElder]

I don't think there's a good way to do this without an active component on the host because we would need to know which network the IP ranges fall into.

If people are mutating the networks on their own outside of kind even the kind network may be different, even if we plumbed in the range on startup or something.

As far as I know docker / podman only expose this info via the host socket inspecting the network or containers.