[aws-eks] custom cloudformation resources is unable to execute kubectl in EKS cluster inside of VPC on private subnets

[lucasvieirasilva]

Hi guys, I have a EKS Cluster inside of VPC on private subnets, and when I try to deploy using cdk cli, I received the following error message from CloudFormation stack events:

Failed to update resource. TimeoutError: Connection timed out after 120000ms at ClientRequest.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/http/node.js:86:34) at Object.onceWrapper (events.js:286:20) at ClientRequest.emit (events.js:198:13) at ClientRequest.EventEmitter.emit (domain.js:448:20) at TLSSocket.emitRequestTimeout (_http_client.js:673:40) at Object.onceWrapper (events.js:286:20) at TLSSocket.emit (events.js:198:13) at TLSSocket.EventEmitter.emit (domain.js:448:20) at TLSSocket.Socket._onTimeout (net.js:443:8) at ontimeout (timers.js:436:11)

In my analysis, I noted there is a CloudFormation custom resource (Custom::AWSCDK-EKS-KubernetesResource) on the CFN template that's execute some kubectl commands, but the lambda function isn't inside a VPC, and I believe that is a reason to SDK timeout.

This is the cloudwatch logs from Lambda function:

START RequestId: 91b7de19-612f-4514-bea9-7fbb13c2fa68 Version: $LATEST
{"RequestType": "Update", "ServiceToken": "arn:aws:lambda:us-east-1:XXXXXXX:function:eks-cluster-stack-awscdka-ProviderframeworkonEvent-1V4HUSIACISP8", "ResponseURL": "XXXXXXX
[INFO] 2020-03-09T14:15:51.170Z 91b7de19-612f-4514-bea9-7fbb13c2fa68 {"RequestType": "Update", "ServiceToken": "arn:aws:lambda:us-east-1:XXXXXXXX:function:eks-cluster-stack-awscdka-ProviderframeworkonEvent-1V4HUSIACISP8", "ResponseURL": "XXXXXXX
Added new context arn:aws:eks:us-east-1:XXXXX:cluster/portal-eks-cluster to /tmp/kubeconfig
[INFO] 2020-03-09T14:15:57.211Z 91b7de19-612f-4514-bea9-7fbb13c2fa68 manifest written to: /tmp/manifest.yaml
[INFO] 2020-03-09T14:17:01.576Z 91b7de19-612f-4514-bea9-7fbb13c2fa68 kubectl timed out, retries left: 3
[INFO] 2020-03-09T14:18:04.141Z 91b7de19-612f-4514-bea9-7fbb13c2fa68 kubectl timed out, retries left: 2
[INFO] 2020-03-09T14:19:06.638Z 91b7de19-612f-4514-bea9-7fbb13c2fa68 kubectl timed out, retries left: 1
END RequestId: 91b7de19-612f-4514-bea9-7fbb13c2fa68
REPORT RequestId: 91b7de19-612f-4514-bea9-7fbb13c2fa68 Duration: 195482.89 ms Billed Duration: 195500 ms Memory Size: 256 MB Max Memory Used: 191 MB Init Duration: 127.29 ms

Environment

• CLI Version : aws-cli/2.0.2 Python/3.7.4 Darwin/19.3.0 botocore/2.0.0dev6
• Framework Version: 1.25.0 (build 5ced526)
• OS : macOS Catalina 10.15.3
• Language : English

This is 🐛 Bug Report

[eladb]

Your analysis is probably correct. Just to make sure, can you please share the code the you use to define your VPC and cluster, so we can repro?

[lucasvieirasilva]

Hello @eladb

I tried to setup VPC configuration directly on lambda function via AWS Console to test my theory and when I execute the cdk deploy command, the stack works fine.

This is the code that I used to create my cluster:

const vpc = Vpc.fromLookup(this, 'VPC', {
    vpcName: `my-vpc-name`
});

const cluster = new Cluster(this, 'Cluster', {
    clusterName: `my-cluster-name`,
    defaultCapacity: 0,            
    vpcSubnets: [{
        subnetType: SubnetType.PRIVATE
    }],
    vpc
});

const nodes = cluster.addCapacity('Nodes', {
    instanceType: new InstanceType(`c5.2xlarge`),
    vpcSubnets: {
        subnetType: SubnetType.PRIVATE
    },
    minCapacity: 3,
    maxCapacity: 100
});

Thanks

[eladb]

Duplicate #5220