networkd starts before udev #1767

crawford · 2017-01-17T18:23:25Z

Issue Report

Bug

CoreOS Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1284.2.0
VERSION_ID=1284.2.0
BUILD_ID=2017-01-10-0545
PRETTY_NAME="Container Linux by CoreOS 1284.2.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

All

Expected Behavior

udev is able to rename interfaces.

Actual Behavior

Errors like the following are seen because networkd starts before udev:

Jan 16 23:55:16 localhost systemd-udevd[1315]: Error changing net interface name 'eth2' to 'ens802f0': Device or resource busy
Jan 16 23:55:16 localhost systemd-udevd[1315]: could not rename interface '4' from 'eth2' to 'ens802f0': Device or resource busy

The text was updated successfully, but these errors were encountered:

qrpike · 2017-02-28T07:54:59Z

I am getting this error on only 1 machine. Latest CoreOS stable. It's an intel NIC if that matters.

dm0- · 2017-03-21T20:34:43Z

Is there a reliable way to reproduce this error? I haven't seen it, and the upstream systemd-networkd.service has After=systemd-udevd.service on all channels.

ndenev · 2017-05-05T11:37:49Z

We're also seeing this. While systemd-udevd.service does start shortly before systemd-networkd.service, there are Error changing net interface name entries, and from boot to boot I get a different combinations of eth* and eno* interfaces (some renamed some not)

[stg] root@k8s1-5 system # journalctl -o short-precise -u systemd-networkd.service | head -n2
-- Logs begin at Fri 2017-05-05 10:36:45 UTC, end at Fri 2017-05-05 11:32:44 UTC. --
May 05 10:36:45.647795 localhost systemd[1]: Starting Network Service...
[stg] root@k8s1-5 system # journalctl -o short-precise -u systemd-udevd.service | head -n2
-- Logs begin at Fri 2017-05-05 10:36:45 UTC, end at Fri 2017-05-05 11:32:50 UTC. --
May 05 10:36:45.642651 localhost systemd[1]: Starting udev Kernel Device Manager...
[stg] root@k8s1-5 system #

May 05 10:36:45.775588 localhost kernel: tg3 0000:02:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b2
May 05 10:36:45.775763 localhost kernel: tg3 0000:02:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.775922 localhost kernel: tg3 0000:02:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.776117 localhost kernel: tg3 0000:02:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.791031 localhost kernel: tg3 0000:02:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b3
May 05 10:36:45.791126 localhost kernel: tg3 0000:02:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.791217 localhost kernel: tg3 0000:02:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.791295 localhost kernel: tg3 0000:02:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.807288 localhost kernel: tg3 0000:03:00.0 eth2: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b4
May 05 10:36:45.807408 localhost kernel: tg3 0000:03:00.0 eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.807497 localhost kernel: tg3 0000:03:00.0 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.807782 localhost kernel: tg3 0000:03:00.0 eth2: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.829230 localhost kernel: tg3 0000:03:00.1 eth3: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:66:63:b5
May 05 10:36:45.829326 localhost kernel: tg3 0000:03:00.1 eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
May 05 10:36:45.829426 localhost kernel: tg3 0000:03:00.1 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
May 05 10:36:45.829535 localhost kernel: tg3 0000:03:00.1 eth3: dma_rwctrl[00000001] dma_mask[64-bit]
May 05 10:36:45.832921 localhost systemd-networkd[404]: eth3: IPv6 enabled for interface: Success
May 05 10:36:45.958443 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
May 05 10:36:45.972112 localhost systemd-networkd[404]: eth0: IPv6 enabled for interface: Success
May 05 10:36:46.101679 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 05 10:36:46.342269 localhost systemd-networkd[404]: eth2: IPv6 enabled for interface: Success
May 05 10:36:46.468596 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
May 05 10:36:46.468763 localhost systemd-networkd[404]: eth1: IPv6 enabled for interface: Success
May 05 10:36:46.593586 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
May 05 10:36:50.273248 localhost systemd-udevd[1953]: Error changing net interface name 'eth2' to 'eno3': Device or resource busy
May 05 10:36:50.385293 localhost kernel: tg3 0000:03:00.1 eno4: renamed from eth3
May 05 10:36:50.385493 localhost kernel: tg3 0000:02:00.0 eno1: renamed from eth0
May 05 10:36:50.273259 localhost systemd-udevd[1953]: could not rename interface '4' from 'eth2' to 'eno3': Device or resource busy
May 05 10:36:50.288118 localhost systemd-udevd[1974]: Error changing net interface name 'eth1' to 'eno2': Device or resource busy
May 05 10:36:50.288124 localhost systemd-udevd[1974]: could not rename interface '3' from 'eth1' to 'eno2': Device or resource busy
May 05 10:36:51.877535 localhost systemd-networkd[3063]: eno4: IPv6 enabled for interface: Success
May 05 10:36:52.003593 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eno4: link is not ready
May 05 10:36:52.024205 localhost systemd-networkd[3063]: eno1: IPv6 enabled for interface: Success
May 05 10:36:52.734042 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
May 05 10:36:55.827881 localhost kernel: tg3 0000:02:00.0 eno1: Link is up at 1000 Mbps, full duplex
May 05 10:36:55.827989 localhost kernel: tg3 0000:02:00.0 eno1: Flow control is off for TX and off for RX
May 05 10:36:55.828127 localhost kernel: tg3 0000:02:00.0 eno1: EEE is disabled
May 05 10:36:55.828760 localhost systemd-networkd[3063]: eno1: Gained carrier
May 05 10:36:55.828987 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready
May 05 10:36:57.473690 localhost systemd-networkd[3063]: eno1: Gained IPv6LL

This is on Stable-1298.7.0

ndenev · 2017-05-08T14:24:39Z

Of course setting debug in the kernel config line makes this work :(

alialshamali · 2017-05-10T09:05:35Z

I am getting the same error, when I reboot the server, its doesn't come up. After looking at the log I find

Error changing net interface name 'eth0' to 'eno1' : Device or resource busy

Only after rebooting the server it come up and take the configured IP address.

Is there any fix?

Just to add its a intel NIC too

ndenev · 2017-05-10T11:35:20Z

I resorted to disabling the "predictable" interface names altogether and use ethX naming.

alialshamali · 2017-05-24T08:09:32Z

Thanks @ndenev
Seems to be a workaround, but wondering what is the root cause. I am trying to analyze to figure what in the kernel is causing this behavior. Its very strange.

alialshamali · 2017-07-20T09:38:14Z

I notice the issue was resolved when I removed the multiple ip addresses from the interface and restart ed the interface. Can this be related? Once I did that I no longer have this issue.

paulmcgoldrick · 2017-08-03T17:02:15Z

Same issue.

[Thu Aug  3 16:45:49 2017] IPv6: ADDRCONF(NETDEV_CHANGE): eth7: link becomes ready
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.1 eno2: renamed from eth1
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.2 eno3: renamed from eth2
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.0 eno1: renamed from eth0
[Thu Aug  3 16:45:51 2017] igb 0000:01:00.3 eno4: renamed from eth3
[Thu Aug  3 16:45:51 2017] ixgbe 0000:44:00.0 enp68s0f0: renamed from eth6
[Thu Aug  3 16:45:51 2017] ixgbe 0000:43:00.0 enp67s0f0: renamed from eth4
[Thu Aug  3 16:45:51 2017] ixgbe 0000:43:00.1 enp67s0f1: renamed from eth5

Aug 03 16:45:47 localhost systemd-udevd[1425]: Error changing net interface name 'eth7' to 'enp68s0f1': Device or resource busy
Aug 03 16:45:47 localhost systemd-udevd[1425]: could not rename interface '9' from 'eth7' to 'enp68s0f1': Device or resource busy

coreos-16 etc # cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1409.6.0
VERSION_ID=1409.6.0
BUILD_ID=2017-07-06-0128
PRETTY_NAME="Container Linux by CoreOS 1409.6.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

awsiv · 2017-08-11T07:45:36Z

I'm seeing the same issue as well:

Aug 10 09:44:59 systemd-udevd[975]: Error changing net interface name 'eth6' to 'ens1f0': Device or resource busy
Aug 10 09:44:59 systemd-udevd[975]: could not rename interface '8' from 'eth6' to 'ens1f0': Device or resource busy

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1409.7.0
VERSION_ID=1409.7.0
BUILD_ID=2017-07-19-0005
PRETTY_NAME="Container Linux by CoreOS 1409.7.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

awsiv · 2017-09-25T07:04:34Z

Any updates on this?

In the initramfs persistent ifnames were disabled. This caused problems because when the renaming was done after the initramfs, a race made it fail, as originally reported in coreos/bugs#1767 Reverts the booteninge commit "systemd: add module to disable network device renaming" and widens the networkd match rule for DigitalOcean.

crawford added area/usability component/distro kind/bug kind/support priority/P0 team/os labels Jan 17, 2017

xeji mentioned this issue Apr 17, 2018

nixos/networkd: fails non-deterministically with predictable interface names NixOS/nixpkgs#39069

Closed

pothos mentioned this issue Feb 13, 2020

Revert "systemd: add module to disable network device renaming" flatcar/bootengine#6

Merged

pothos mentioned this issue Feb 25, 2020

sys-kernel/bootengine: Re-enable persistent ifnames flatcar-archive/coreos-overlay#190

Merged

goochjj mentioned this issue Mar 4, 2020

Interface Naming broken in 2345 Stable flatcar/Flatcar#36

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

networkd starts before udev #1767

networkd starts before udev #1767

crawford commented Jan 17, 2017

qrpike commented Feb 28, 2017

dm0- commented Mar 21, 2017

ndenev commented May 5, 2017

ndenev commented May 8, 2017

alialshamali commented May 10, 2017 •

edited

Loading

ndenev commented May 10, 2017

alialshamali commented May 24, 2017

alialshamali commented Jul 20, 2017

paulmcgoldrick commented Aug 3, 2017 •

edited

Loading

awsiv commented Aug 11, 2017 •

edited

Loading

awsiv commented Sep 25, 2017

networkd starts before udev #1767

networkd starts before udev #1767

Comments

crawford commented Jan 17, 2017

Issue Report

Bug

CoreOS Version

Environment

Expected Behavior

Actual Behavior

qrpike commented Feb 28, 2017

dm0- commented Mar 21, 2017

ndenev commented May 5, 2017

ndenev commented May 8, 2017

alialshamali commented May 10, 2017 • edited Loading

ndenev commented May 10, 2017

alialshamali commented May 24, 2017

alialshamali commented Jul 20, 2017

paulmcgoldrick commented Aug 3, 2017 • edited Loading

awsiv commented Aug 11, 2017 • edited Loading

awsiv commented Sep 25, 2017

alialshamali commented May 10, 2017 •

edited

Loading

paulmcgoldrick commented Aug 3, 2017 •

edited

Loading

awsiv commented Aug 11, 2017 •

edited

Loading