systemd-networkd: Generated network configuration breaks resolved DBus API

[tadfisher]

Issue description

network-interfaces-systemd.nix generates /etc/systemd/network/99-main.network which matches all network interfaces, in order to enforce the networking.useDHCP option. This has the effect of forcing all network interfaces to be "managed" by systemd-networkd.

Upstream does not allow managed interfaces to be configured via the org.freedesktop.resolve1 DBus API, as the following transcript demonstrates:

$ busctl call org.freedesktop.resolve1 \
> /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager \
> SetLinkDNS 'ia(iay)' 3 2 2 4 209 222 18 222 2 4 209 222 18 218
Link tun0 is managed.

$ dbus-send --print-reply --system --type=method_call --dest=org.freedesktop.resolve1 /org/freedesktop/resolve1/li
nk/_33 org.freedesktop.DBus.Properties.GetAll string:''
method return time=1474790774.718633 sender=:1.0 -> destination=:1.5 serial=6 reply_serial=2
   array [
      dict entry(
         string "ScopesMask"
         variant             uint64 0
      )
      dict entry(
         string "DNS"
         variant             array [
            ]
      )
# -- snip --

$ cat /run/systemd/netif/links/3
# This is private data. Do not parse.
ADMIN_STATE=configured
OPER_STATE=routable
NETWORK_FILE=/nix/store/dcn980w7p3vwg0jyhfvkbaczdl6kcf2i-network-units/99-main.network
DNS=
NTP=
DOMAINS=
ROUTE_DOMAINS=
LLMNR=yes
MDNS=no
ADDRESSES=
ROUTES=

$ cat /etc/systemd/network/99-main.network
[Match]

[Network]
DHCP=none

Note that the primary motivation for using the resolved DBus API is to manage per-interface DNS, which I do not believe is possible via openresolv. Invoking this API is a clean way to prevent DNS leaks over tun0 when connecting to an OpenVPN server, for example. In this example, tun0 is not configured via networking.interfaces."tun0" and an explicitly-matching .network configuration does not exist.

I've been able to work around this problem with the following in configuration.nix:

# Prevent networkd from managing unconfigured links.                                                    
systemd.network.networks."99-main".enable = false;

A solution that seems reasonable to me would be to treat networking.useDHCP = null; differently from true or false, and refrain from generating a default .network configuration which matches all interfaces. However, this changes the semantics of that option when networking.useNetworkd is enabled.

Steps to reproduce

This can be demonstrated with the following configuration:

networking.useNetworkd = true;
services.resolved.enable = true;

Then using busctl as shown above to attempt setting per-interface DNS or DOMAIN options on any interface which is not configured via networking.interfaces.<name>.

Technical details

• System: 16.09beta430.c4469ed (Flounder)
• Nix version: nix-env (Nix) 1.11.4
• Nixpkgs version: "16.09beta430.c4469ed"

[Mic92]

Resolved might not work because nss module for resolved is ignored currently by libc.
See this issue

[groxxda]

Should we add an option enableCatchAll that conditionally enables 99-main.network? It should be turned on by default I guess (at least if no interfaces are configured explicitly).

The simplest network use case is probably "get ip, domain, gateway from dhcp on all interfaces (that may appear)" and should be covered without any configuration.

[tadfisher]

@Mic92 That's a separate issue, I believe. Even without the nss module, resolved should handle requests to its DBus API and work independently; i.e. resolution can be performed with systemd-resolve <IP or domain>.

In my case, I've forced a symlink from {systemd}/usr/lib/systemd/resolved.conf -> /etc/resolve.conf to use the resolved stub resolver instead. This issue can be reproduced in any of these configurations, as resolved is still listening on 127.0.0.53 and DBus when its service is enabled.

[florianjacob]

This issue of 99-main.network matching everything leads to cjdns being broken when networkd is enabled. ☹️

I see additional options to fix this which have not been mentioned yet:

• Support networkd's Unmanaged option to explicitly take interfaces away from the all-matching 99-main network.
• Reduce the unconditional matching of 99-main to Name = en* like in the man systemd.network example “DHCP on ethernet links“, or Name = en* wl*.

I'll work on a PR for the Unmanaged option as this should be non-controversial, and, as a baseline, would allow all setups with additional configuration on what should be unmanaged.

[fpletz]

You could also manage a black- and/or whitelist like networking.dhcpcd does with the options allowInterfaces and denyInterfaces.

[florianjacob]

@fpletz Thanks for the hint to just look at how it's done in networking.dhcpcd. 😆 I tried googling around how dhcpcd handles that in general, with not much success.

We could add something like allowInterfaces, which essentially would be this in the 99-main.network file:

[Match]
Name = ${cfg.allowInterfaces}

The problem is, this 99-main is created by the config.networking abstraction, so to add that, it would need a new option in the abstraction, not in systemd-networkd options. Would that be an option?

What would work is to interpret networking.interfaces.useDHCP = false; in the way that it just disables 99-main.network completely, which would allow the user to provide their own instead, specifying [Match] like they need it. But this could impose problems with people using networking.interfaces.useDHCP = false; and rely on what's still configured besides DHCP on each interface, that is: networking.interfaces.gateway and networking.inferfaces.domain, and accepting IPv6 Router Advertisements. Assigning link-local unicast IPv6 addresses would still be done, as that's part of the kernel and not of networkd.

PR for linkConfig.Unmanaged option at #27695, my cjdns problems are solved with using that.

[arianvp]

I am in favor of introducing the allowInterfaces option but only if we also implement it for the scripted backend such that it then only touch the interfaces listed in allowInterfaces. This would allow people to run multiple network management tools side by side that don't interfere. E.g. we're thinking of enabling the systemd-networkd.service to manage nixos-container networking even in the case scripted networking is enabled, but that must mean that scripted networking (and systemd networking) ignores any interfaces it wan't explicitly configured to manage. Hence I think the allowInterfaces option is the most elegant solution here.

A more controversial option:

default networking.useDHCP = false; and have nixos-generate-config generate a hardware-configuration.nix with a list of interfaces that should be managed by the networking system during install and have that set networking.interfaces.<name>.useDHCP = true;. That way we know explicitly what parts are managed by networking at what parts aren't

[arianvp]

Came up with an another workaround for you @tadfisher . networkd loads .network files lexographically. So if a user wants to override the match all behaviour in 99-main.network, create a 98-tun.network with a more specific match, and make it explicitly unmanaged.

systemd.networks."98-tun" = {
  matchConfig.Match = "tun*";
  networkConfig.Managed = "no";
};

[alexbakker]

Somehow, this now also causes the loopback interface to lose its configuration when resuming the machine from suspend. Disabling 99-main as mentioned fixes this.

[bendlas]

Somehow, this now also causes the loopback interface to lose its configuration when resuming the machine from suspend.

This is the exact problem, I was having. Worked around it, by configuring it with 127.0.0.1

Also, useDHCP = false doesn't work for dynamically connected USB modems.

[arianvp]

I think wildcards are supported, though not documented. This allows you to opt into DHCP for each interface type, including your USB modem.

networking.interfaces."ens*".useDHCP = true;

[arianvp]

With networkd directly, more deliberate matchConfigs can be created. E.g. match based on device driver.

systemd.networks."20-usb" = {
  matchConfig.Driver = "<USB modem driver name, from udevadm info>"
  networkConfig.DHCP = true;
};

[bendlas]

networking.interfaces."ens*".useDHCP = true;

already tried that, it fails at boot, thus adding 1:30m

I have working wildcards at systemd.network.networks.<name>.matchConfig.Name (*) as well as networking.nat.internalInterfaces (+)

Awesome idea with matching on the USB driver, though. I'll try that.

[flokli]

@tadfisher, @bendlas: We merged #71790, so we shouldn't need any workaround here anymore, and this issue could be closed. Can you have a look?

[tadfisher]

@flokli I'll check it out when it hits nixos-unstable. Thanks!

[bendlas]

@flokli The problem with localhost has disappeared now, thanks! I have created a PR for the DHCP issue #72044

[flokli]

Nice, cool to hear! Will take a look at the PR tomorrow with @fpletz :-)

[flokli]

This was fixed by #71790.