Creating a NLB load balancer sometimes fails (lacking "ec2:DescribeInternetGateways" permission)

[kkl-acies]

What happened?
I tried creating a NLB load balancer and sometimes it work and sometimes it fails with:

  Normal   EnsuringLoadBalancer    61s (x6 over 3m38s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  60s                  service-controller  Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "AccessDenied: User: arn:aws:sts::3XXXXX:assumed-role/eksctl-core-test-cluster-ServiceRole-SXXXX/12341 is not authorized to perform: ec2:DescribeInternetGateways\n\tstatus code: 403, request id: cXXXX"

What you expected to happen?
EKS should always succeed creating a NLB load balancer.

How to reproduce it?

$ eksctl create cluster --node-volume-size 10 --managed --node-type t3a.small --nodes 2 --name "${NAME}"
eksctl version 0.26.0
[...]
$ eksctl utils associate-iam-oidc-provider --cluster="${NAME}" --approve` # not sure it is related, but I did run it..
$ cat << EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: test-lb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  selector:
    app: test-lb
  ports:
  - port: 80
  type: LoadBalancer
EOF

Anything else we need to know?
eksctl is downloaded from the release page and run inside a GitHub Actions environment (Ubuntu 18.04) with a IAM user with IAMFullAccess and PowerUserAccess . kubectl is run from Ubuntu 20.04 running on WSL2 with a IAM Role mapped to system:masters.

Versions

$ eksctl version
0.26.0
$ kubectl version
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

[kkl-acies]

We probably need to tweak:
https://github.com/weaveworks/eksctl/blob/1cc9746585e6485a128faf150de28d3afcf65906/pkg/cfn/builder/iam.go#L96-L98

According to the AWS doc we need following permissions:

CreateLoadBalancer
  elasticloadbalancing:CreateLoadBalancer
  ec2:DescribeAccountAttributes
  ec2:DescribeAddresses
  ec2:DescribeInternetGateways
  ec2:DescribeSecurityGroups
  ec2:DescribeSubnets
  ec2:DescribeVpcs
  iam:CreateServiceLinkedRole

Some projects (the issue is also worth reading) only adds ec2:DescribeAccountAttributes and ec2:DescribeInternetGateways, so I'm not sure what the correct answer is.

[michaelbeaumont]

Can you confirm this solves the issue reliably? By the logic of terraform-aws-modules/terraform-aws-eks#183 (comment) (linked in #2488 as well) this still isn't sufficient.
EDIT: I see you linked something similar, in the issue I linked is a 3rd permission that is definitely missing from AmazonEKSClusterPolicy)

[kkl-acies]

Can you confirm this solves the issue reliably?

As it only happens sometimes it is very hard to test. Should we just add the 3 permissions mentioned in that comment?