Support Pod IRSA for CNI metrics helper

[nithu0115]

What would you like to be added:
Support Pod IRSA feature for CNI metrics helper.

Today, we rely heavily on getting region and EC2 instance ID information from EC2 metadata. However, when using Pod IRSA along with dropping iptables to reach EC2 IMDS (169.254.169.254), we will not be able to get this information and the Pod will fail to establish/create a session.

• For region information: Add new environment variable to list. Something like AWS_REGION. However, starting 1.18+ Pod IRSA inject AWS_REGION variable into the Pod environment which will solve this issue.
• The major problem is with EKS cluster name as cluster names are found by querying EC2 instance ID tags which requires us to get EC2 instance ID from EC2 meta-data.

CC: @jayanthvn

[groodt]

I've been tracking 3 important AWS addons: aws-ebs-csi-driver, aws-efs-csi-driver and amazon-vpc-cni-k8s.

aws-ebs-csi-driver is now operating without hostNetwork, so I imagine for the cni-metrics-helper, similar could be done kubernetes-sigs/aws-ebs-csi-driver#821 (comment)

[achevuru]

@groodt cni-metrics-helper runs in kube-system namespace but doesn't use hostNetwork. Are you referring to something else?

https://github.com/aws/amazon-vpc-cni-k8s/blob/master/config/v1.9/cni-metrics-helper.yaml#L86

[groodt]

We are currently running: https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.7.2/config/v1.7/cni-metrics-helper.yaml

We have had to patch the Deployment through to hostNetwork for it to work. This is on a cluster following AWS EKS Security Best Practice. We have blocked IMDSv1 entirely. IMDSv2 is only permitted for containers that use hostNetwork, which should be limited as far as possible.

The ideal solution here is that cni-metrics-helper (and possibly any other of the amazon-vpc-cni-k8s workloads where it is possible) should be to use IRSA and run on a cluster that blocks IMDSv1 and strictly limits access to IMDSv2 and the instance role of the host node.

[TBBle]

If AWS_REGION is auto-injected as an env-var by IRSA, then for it, all the metrics helper needs to do is skip overriding the default region that the SDK picked up, as the SDK honours the AWS_REGION env-var (older versions needed to be told to honour it, newer versions honour it by default).

However, I can't see any way to get the clusterID except as a user-parameter, as it's a tag on the instance, but not (AFAIK) otherwise exposed inside kubernetes. The AWS Cloud Provider itself relies on IMDS to locate the EC2 Instance data in order to extract the clusterID tags, but does not, for example, add the cluster ID as an annotation on the Node objects (which would be nice for use-cases like this, and might be worth a feature-request).

Given both of these, I guess a reasonable approach is:

• If AWS_REGION env-var is present, assume the SDK consumed it (and make sure the SDK version is recent enough).
• If AWS_CLUSTER_ID env-var is not present, attempt to fetch it from IMDS, and if that fails, fail startup telling the user that AWS_CLUSTER_ID must be injected as an env-var if IMDS access is blocked.

If we can get the AWS instance ID somehow (not sure if that's exposed on a Node object off-hand...) and the IRSA account is given the ec2:DescribeInstances permission (I think?), then we can use the existing code that searches the instance's tags for the cluster ID.

It'd be higher-level nicer if either the IRSA webhook could inject the cluster ID as well as the AWS_REGION (but I don't think it knows the cluster ID...), or if there was an eks API for ListMyCluster, which would automate poking at the tags on various things based on who was asking (instance role, IRSA, etc)

[jayanthvn]

PR 1715 is merged and will be part of 1.10.2 release. EKS documentation will be updated post release. Readme is update in GitHub repo.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.