Ramdomize SNAT not working

[InAnimaTe]

Sans me possibly doing this wrong, I've made the variable exist to enable randomization for snat rules, but I'm not seeing this being applied in the iptables rules (when running iptables-save from a aws-node pod via kubectl exec -it aws-node-pvx48 /bin/sh -n kube-system). When I edit the daemonset, the pods re-roll but nothing seems to change (let me know if I'm doing this wrong).

sh-4.2# env | grep CNI
AWS_VPC_K8S_CNI_LOGLEVEL=DEBUG
AWS_VPC_K8S_CNI_RANDOMIZESNAT=hashrandom

sh-4.2# iptables-save | grep random
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-AL73UYBXEM4ASRGB
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-7GVY2H3HQJW2S663
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-JLDNAEVHEO37WMGP

Full iptables-save can be found here. Also tried with prng but then realized iptables in only version 1.4; interested as to if that could be increased or if there's a newer version I can use.

Thanks!

[InAnimaTe]

@taylorb-syd @mogren any insight would be appreciated!

[taylorb-syd]

Hi there, @InAnimaTe, first, can we get your clusters CNI version to confirm if you're running a compatible version:

kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

Next, can you also confirm if you refreshed your workers after adding the environment variable to the DaemonSet? I outlined this limitation in PR #246 here.

[InAnimaTe]

Here's what the daemonset looks like:

      containers:
      - env:
        - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
          value: hashrandom
        - name: AWS_VPC_K8S_CNI_LOGLEVEL
          value: DEBUG
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:v1.3.2

I have not rebooted the worker nodes yet. Let me work on that now and see what I've got after that! In any case, if rebooting is necessary, the docs should be updated to reflect this; I can make that PR.

[InAnimaTe]

Just realized the SNAT change is in 1.4.0 and eksctl 0.1.27 utils update-aws-node only does aws-node 1.3.2.

[StevenACoffman]

ref eksctl-io/eksctl#736 waiting because 1.4.0 is not AMI default.
ref aws/containers-roadmap#149 AMI default waiting on #413

[InAnimaTe]

Ok now it seems to be working closer as to expected:

-A AWS-SNAT-CHAIN-1 -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 192.168.16.254
-A AWS-SNAT-CHAIN-1 -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 192.168.16.254 --random

^This is without a reboot. While new lines are added, those new --random lines won't trigger.

Rebooting the nodes now to see what things look like after the fact: RESULT below:

-A AWS-SNAT-CHAIN-1 -m comment --comment "AWS, SNAT" -m addrtype ! --dst-type LOCAL -j SNAT --to-source 192.168.14.40 --random

Looks like we're all good! (I could have manually pulled out the other rule with a -D command but that's more tedious than just rebooting, especially since I'm doing this at cluster bootstrap).

@taylorb-syd can you weigh here on documentation etc..? Should I just submit a PR or are you working on some updates there? Additionally, any idea when a newer version of iptables will be used? Would really like --random-fully to fully kill the DNS SNAT most clusters exhibit.

[InAnimaTe]

Going to close as the core issue seems to be solved. Using eksctl, I'm able to apply the snat patch and then spawn my nodegroup, so no manual rebooting needed.