nixos/networkd: fails non-deterministically with predictable interface names

[xeji]

Issue description

When using systemd-networkd together with predictable interface names, there seems to be a race condition between networkd activating the interfaces and udevd renaming them, resulting in non-deterministic failure of interface renaming and setup.

For an example, see these hydra builds: failed and succeeded

Issue occurs on 18.03 and current master (i.e. with systemd 237 and 238).

It is probably an upstream bug (but systemd has not confirmed this yet). I found two similar reports: systemd/systemd#7293 and coreos/bugs#1767

Since the corresponding test is part of the tested jobs for the nixos-unstable-small and release-18.03-small channels, the sporadically failing test can delay the channels.

I suggest removing this test from the tested jobs for the time being until this is fixed.

Steps to reproduce

Although the issue seems non-deterministic in nature (see hydra builds), I can quite reliably reproduce it running the test on my local machine:
nix-build '<nixpkgs/nixos/tests/predictable-interface-names.nix>' -A vm-test-run-predictableInterfaceNames-with-networkd

Technical details

nixos 18.03.132008.ad771371fb2 or master e0c9a25

[Mic92]

Note that our udev rules for network interfaces are not in sync with what systemd upstream uses.
So that bugs in our system are probably different from what other linux distributions report.

[xeji]

The only relevant change to udev rules that I found so far is that we use a custom version of 80-net-setup-link.rules when predictable interface names are enabled. Not sure why the built-in upstream solution doesn't work for us.

[xeji]

After some tests, here's what I think is happening: Stage 1 udev discovers and adds the network interfaces with their original ethX names. Renaming doesn't happen until stage 2. But networkd tries to configure all interfaces it finds, so there's a race condition between stage 2 udev and networkd.

This can be (sort of) "solved" by an ugly hack: Add ExecStartPre=sleep 5 to networkd to allow time for renaming. Fixes the test but slows down the boot and still does not guarantee success.

A better solution might be to let stage 1 udev rename the interfaces. I tried but haven't quite figured out how to do this. We might try to go back to the upstream way of predictable interface names controlled by the net.ifnames kernel boot option.

@shlevy maybe you can help? (I found your f756369 from a long time ago, which was reverted after a few days)

[shlevy]

Hmm not sure here, sorry!

[xeji]

Fixed by applying predictable interface names in stage 1 already.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixosstag.fcio.net/t/predictable-network-interface-names-in-initrd/4055/1