Flannel daemon set not working on CentOS 7.9 with error message 'iface variable - Could not find valid interface matching'

[weiyshay]

I am trying to set up k8s on a CentOS host but failed to install flannel with below error message:

I0602 03:33:17.123079       1 main.go:278] Could not find valid interface matching eth0: failed to find IPv4 address for interface eth0
E0602 03:33:17.123099       1 main.go:309] Failed to find interface to use that matches the interfaces and/or regexes provided

I searched around and it seems that this is a general configuration issue which normally caused by the iface name not existing or incorrect, while I am sure the name and the ip address are correct on my host and I added the ifname to the original
daemonset configuration yaml as below:

diff kube-flannel.yml.orig  kube-flannel.yml -rup
--- kube-flannel.yml.orig       2022-06-01 22:57:47.132126202 -0400
+++ kube-flannel.yml    2022-06-01 23:01:08.813126202 -0400
@@ -200,6 +200,7 @@ spec:
         args:
         - --ip-masq
         - --kube-subnet-mgr
+        - --iface=eth0
         resources:
           requests:
             cpu: "100m"

My env:

• Flannel version: https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
• Backend used (e.g. vxlan or udp): vxlan
• Etcd version: don't know how to check but I am using the default
• Kubernetes version (if used): kubelet-1.23.6 kubeadm-1.23.6 kubectl-1.23.6

ip addr show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:02:15:12 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.12/24 brd 192.168.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe02:1512/64 scope link
valid_lft forever preferred_lft forever

ip route show

default via 192.168.2.1 dev eth0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.12

kubectl logs -n kube-system kube-flannel-ds-2kxbv

I0602 03:33:15.923363 1 main.go:207] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[eth0] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0602 03:33:15.923449 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0602 03:33:16.122620 1 kube.go:121] Waiting 10m0s for node controller to sync
I0602 03:33:16.122648 1 kube.go:398] Starting kube subnet manager
I0602 03:33:17.122777 1 kube.go:128] Node controller sync successful
I0602 03:33:17.122805 1 main.go:227] Created subnet manager: Kubernetes Subnet Manager - vm2-centos79
I0602 03:33:17.122811 1 main.go:230] Installing signal handlers
I0602 03:33:17.122907 1 main.go:463] Found network config - Backend type: vxlan
I0602 03:33:17.123079 1 main.go:278] Could not find valid interface matching eth0: failed to find IPv4 address for interface eth0
E0602 03:33:17.123099 1 main.go:309] Failed to find interface to use that matches the interfaces and/or regexes provided

[discostur]

Also ran into the same problem:

I0602 12:15:05.091233       1 main.go:207] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0602 12:15:05.091341       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0602 12:15:05.295923       1 kube.go:121] Waiting 10m0s for node controller to sync
I0602 12:15:05.296240       1 kube.go:398] Starting kube subnet manager
I0602 12:15:06.296365       1 kube.go:128] Node controller sync successful
I0602 12:15:06.296403       1 main.go:227] Created subnet manager: Kubernetes Subnet Manager - k8s-dev03.intern.customer-virt.eu
I0602 12:15:06.296418       1 main.go:230] Installing signal handlers
I0602 12:15:06.296546       1 main.go:463] Found network config - Backend type: vxlan
I0602 12:15:06.296579       1 match.go:195] Determining IP address of default interface
E0602 12:15:06.296658       1 main.go:270] Failed to find any valid interface to use: failed to get default interface: protocol not available

CentOS 8.x does work
CentOS 7.x is broken

However, rolling back to flannel v0.17 does work. So there must be any change is the new 0.18 version which breaks c7 support ...

[weiyshay]

Rolling back to v0.17 can successfully launch the flannel container on my host while the coredns stuck in 'ContainerCreating' now.

#kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d8c4cb4d-js6rs 0/1 ContainerCreating 0 7m42s
kube-system coredns-6d8c4cb4d-tcvtt 0/1 ContainerCreating 0 7m42s
kube-system etcd-vm2-centos79 1/1 Running 0 7m53s
kube-system kube-apiserver-vm2-centos79 1/1 Running 0 7m53s
kube-system kube-controller-manager-vm2-centos79 1/1 Running 0 7m54s
kube-system kube-flannel-ds-cll2z 1/1 Running 0 7m43s
kube-system kube-proxy-42x5s 1/1 Running 0 7m43s
kube-system kube-scheduler-vm2-centos79 1/1 Running 0 7m53s

#kubectl logs -n kube-system coredns-6d8c4cb4d-js6rs
Error from server (BadRequest): container "coredns" in pod "coredns-6d8c4cb4d-js6rs" is waiting to start: ContainerCreating

[manuelbuil]

There was an update on the go netlink library which apparently breaks centos7, perhaps because it uses a very old kernel

[discostur]

True, could be related to the netlink update (#1565). Also this is just a beta release:

github.com/vishvananda/netlink v1.2.0-beta

Edit: opened an issue in the netlink project

[manuelbuil]

True, could be related to the netlink update (#1565). Also this is just a beta release:

github.com/vishvananda/netlink v1.2.0-beta

Edit: opened an issue in the netlink project

Good idea! Thanks. Do you know if the maintainers of that project respond quickly?

[discostur]

No, sorry. First time that i visited their repository ...

[manuelbuil]

The new release 0.18.1 should fix the issue with centos 7. Could you please try?

[ViokingTung]

The new release 0.18.1 should fix the issue with centos 7. Could you please try?

yes,fixed, release 0.18.1 went well with centos 7

[discostur]

@manuelbuil can confirm this fixes the issue and compatibility is given with centos 7 - thanks ;)

[manuelbuil]

@manuelbuil can confirm this fixes the issue and compatibility is given with centos 7 - thanks ;)

Thanks to you! We just updated the netlink library to the new version with the fix you reported

[manuelbuil]

As this is fixed, I think we can close it