Flannel v0.9.1 adds FORWARD rules breaking networkpolicy behaviour

[KashifSaadat]

Expected Behavior

1. Configure the setting FELIX_CHAININSERTMODE to append in the canal manifest file
2. Deploy to a node
3. Inspect iptables: iptables -nv -L FORWARD
4. I would expect only 1 rule to be inserted by calico (at the bottom of the chain, if you have inserted custom iptables rules). For example:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     tcp  --  cali+  *       0.0.0.0/0            169.254.169.254      tcp dpt:80 state NEW 
    0     0 cali-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:wUHhoiAYhphO9Mso */

Current Behavior

Flannel v0.9.1 includes a change to add 2 rules to the FORWARD chain (flannel-io/flannel#872). When Calico is configured with append mode, the Calico rule is added to the bottom of the chain, resulting in the following rule order:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     tcp  --  cali+  *       0.0.0.0/0            169.254.169.254      tcp dpt:80 state NEW reject-with icmp-port-unreachable
    0     0 ACCEPT     all  --  *      *       10.10.0.0/16         0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            10.10.0.0/16
    0     0 cali-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:wUHhoiAYhphO9Mso */

Because of this, the calico rule is never hit and all traffic in/out is accepted by default. This means that k8s ingress/egress network policies will have no effect.

You won't encounter this issue if the chain insert mode isn't set to append (it is insert by default). Flannel will still add 2 rules to the forward chain, but they will never be hit because the calico rule is processed first.

Possible Solution

I'm not sure whether this is a flannel bug (when using as part of a canal deployment) or something that can be fixed by changes to the manifest file.

One suggestion would be to provide a flag to flannel such as "AddIPTablesForwardRules": false and then specify this in the Canal manifest.

Context

I'm performing these deployments using kops v1.8.0, and the relevant manifest file is located here: https://github.com/kubernetes/kops/blob/1.8.0/upup/models/cloudup/resources/addons/networking.projectcalico.org.canal/k8s-1.8.yaml.template

I set FELIX_CHAININSERTMODE to append so that I can insert rules at the top of the INPUT and FORWARD chains to process specific global REJECT rules before the network policies take effect.

A quick fix for kops would be to downgrade flannel to v0.9.0 JUST in the Canal manifest files (3 of them), but of course this isn't ideal.

Your Environment

• Calico version: v2.6.2
• Flannel version: v0.9.1 (the issue was introduced in this version)
• Orchestrator version: kops v1.8.0
• Operating System and version: CoreOS stable v1520.8.0

CC @chrislovecnm @justinsb
Ping @tomdee @caseydavenport

[caseydavenport]

@KashifSaadat just to be clear, this is the correct and expected behavior for Felix when set to use "append" for its rules. By using "append", you're opening up a hole where another application might bypass policy.

That said, it's a bit sad that flannel doesn't play nicely here. There may be room for a change to flannel in order to disable the iptables rules that it installs, but I'll defer to @tomdee or @gunjan5 for that.

I set FELIX_CHAININSERTMODE to append so that I can insert rules at the top of the INPUT and FORWARD chains to process specific global REJECT rules before the network policies take effect.

Wondering, is there a reason you need to hand-write global rules rather than using Calico policy to enforce that behavior?

[KashifSaadat]

Hey @caseydavenport, thanks for the update.

I'm afraid I don't have much experience with using Calico policies, at the time I implemented the custom iptables ruleset there wasn't support for defining these policies when using the K8s datastore (this was back on v2.4.1). Would you be able to point me at the docs for creating them?

[caseydavenport]

@KashifSaadat Ah, yes. I forgot that canal was using the k8s api datastore. Support for Calico policies isn't available yet, but will be in Calico v3.0.0, to be released at the end of the year.

[hosseinsalahi]

I have the same issue with k8s v1.10.2 and Calico v3.1.

[caseydavenport]

Looks like this is just waiting on a flannel release.

Once that's done, we can update the manifests to use a new version of flannel and set the --iptables-forward-rules=false option on flanneld.

Also note that Canal now fully supports Calico global network policies, so the need for writing global reject iptables rules by hand is gone!

[caseydavenport]

We've updated to v0.11 so I think this should be good to go if someone wants to try making the changes?

[KashifSaadat]

Sorry I forgot to update! I have tested and can confirm that using flannel v0.11.0 with the new environment variable / flag set for the deployment (introduced in that release) fixes this issue. I'll close the issue, thanks!