Firewall reload on NixOS upgrade kills networking

[Thesola10]

Describe the bug

I have enabled autoUpgrade on my homelab, which is using the default NixOS firewall module. When an autoUpgrade occurs, or when nixos-rebuild switch --upgrade updates the firewall module, all networking on the server ceases to work when firewall.service is reloaded.

Running iptables -L on a local terminal hangs when it starts listing the nixos-fw chain.

Steps To Reproduce

Steps to reproduce the behavior:

1. TODO: isolate minimal NixOS config and test that causes the issue
2. Activate a NixOS system generation where firewall.service has changed slightly (no meaningful changes to actual behavior)

Expected behavior

The firewall reloads without interrupting the network.

Screenshots

Excerpt from my NixOS firewall config:

networking =
{
  firewall =
  { enable = true;
    allowPing = true;
    logRefusedConnections = false;

    allowedTCPPorts = [ 22 80 443 redacted ];
    allowedUDPPorts = [ 67 68 redacted ]; # NOTE: 67/68 required for proxyDHCP

    extraCommands = ''
      iptables  -A nixos-fw -s 10.111.0.0/16          -j nixos-fw-accept
      iptables  -A nixos-fw -s 10.88.0.0/16           -j nixos-fw-accept
      iptables  -A nixos-fw -s 192.168.1.0/24         -j nixos-fw-accept
      ip6tables -A nixos-fw -s fe80::/64              -j nixos-fw-accept
      ip6tables -A nixos-fw -s fc00:111::/96          -j nixos-fw-accept
      ip6tables -A nixos-fw -s redacted:/64           -j nixos-fw-accept

      iptables  -I FORWARD  -j nixos-fw
      ip6tables -I FORWARD  -j nixos-fw

      iptables  -P FORWARD  DROP
      ip6tables -P FORWARD  DROP
      iptables  -P INPUT    DROP
      ip6tables -P INPUT    DROP
    '';
  };
  nat =
  { enable = true;
    internalInterfaces = [ "ve-+" "vb-+" "br0" ];
    externalInterface = "enp3s0";
    enableIPv6 = true;
    externalIPv6 = "redacted";

    extraCommands = ''
      iptables -t nat -A nixos-nat-post -d 10.111.0.12 -p tcp --dport 7522 -j SNAT --to-source 10.111.0.1
    '';
  };
};

Firewall config added to all NixOS containers:

      networking.firewall.extraCommands = ''
        iptables -A INPUT -s 10.111.0.0/16 -j ACCEPT
        ${if addr6 != ""
        then "ip6tables -A INPUT -s fc00:111::/96 -j ACCEPT"
        else ""}
      '';

Message from firewall.service:

iptables: Chain already exists.
Failed to reload firewall... Stopping

Additional context

I am using both NixOS systemd-nspawn containers, and virtualisation.oci-containers with Podman as the backend. In the config excerpt above, 10.111.0.0/16 is the subnet used by NixOS containers.

Also, I am running dnsmasq on host as proxyDHCP exclusively.

The issue arising appears to correlate with NixOS 21.11, so it might be something exclusive to nftables.

Notify maintainers

Metadata

• system: "x86_64-linux"
• host os: Linux 5.15.78, NixOS, 22.05 (Quokka), 22.05.4250.6474d93e007
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.8.1
• channels(root): "nixos-22.05"
• channels(karim): "git-goodies, nixpkgs-22.05pre363562.30d3d79b7d3"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[Thesola10]

Okay, had a look at the generated nft tables, and found a funny little inconsistency when iptables -L gets stuck:

chain FORWARD {
    type filter hook forward priority filter; policy drop;
     counter packets 1961 bytes 278235 jump CNI-FORWARD
    counter packets 910 bytes 66864 jump nixos-fw
}

Wonder if this has something to do with it? Looks like a conflict between Podman's firewall settings and my own "funnel FORWARD into the NixOS-generated firewall".

Maybe Podman uses nft directly and iptables-nft then walks in and breaks the chain?

[nixinator]

are you sure?, pod-man modifying the systems firewall configuration directly, just willy nilly.

Sounds bizarre,

[Thesola10]

that's the default behavior for podman when setting up port redirects, but I agree, we can't have it running around breaking firewalls.

As a workaround I'll try to move Podman to a NixOS container, where the firewall is much simpler.

[Thesola10]

Found a permanent workaround by systematically flushing the entire firewall on unit stop, ensuring it is regenerated from a clean slate:

extraStopCommands = ''
  iptables -F
  ip6tables -F
'';