[EKS] [PrivateLink Support]: Support Kubernetes API server endpoint as AWS PrivateLink Endpoint

[ingshtrom]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
What do you want us to build?

Support for AWS PrivateLink Endpoints for Elastic Kubernetes Service.

Which service(s) is this request for?
Elastic Kubernetes Service. ECR and ECS are already supported as PrivateLink Endpoints.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

We have EKS clusters running in various VPCs for production, pre-production, etc. One of the VPCs is solely devoted to continuous integration/deployment, but of course it needs access to certain resources in our other VPCs in order to do its various jobs. One of those jobs is to run Terraform, which in turn creates Kubernetes resources, so it needs to access the Kubernetes API of our pre-production and production environments, which of course are in separate VPCs and are set up as private EKS endpoints. If we had PrivateLink Endpoint support for EKS, we could create a PrivateLink Endpoint of our pre-prod/prod EKS API Servers into our CI/CD VPC for access by our CI/CD jobs.

Are you currently working around this issue?
How are you currently solving this problem?

Right now, we use a VPC Peering connection to allow access to resources across VPCs. Additionally we attempt to use Security Groups (and soon Security Groups for Pods) to be able to better lock down which resources in the CI/CD VPC can access which resources in the pre-prod/prod VPCs, but it is definitely cumbersome. There are some resources that should never be accessed by our CI/CD VPC, so there is no point in exposing them via the VPC Peering Gateway.

Alternatively, we have thought about creating a load balancer that was registered as a PrivateLink Service and have that point at the IPs behind the EKS Kubernetes API endpoint. Of course that would require a background process that continually queries the EKS enpdoint DNS to get the list of IPs and shove them into a Target Group, but theoretically it sounds possible.

Additional context
Anything else we should know?

PrivateLink Endpoints also make the intention of access more explicit. As I come to look at the set up in a month or two, I might question why we have a whole VPC Peering connection set up as it feels overly permissive. Then I might go on to turn something off not realizing that I broke these other services owned by another team because they relied on some esoteric Security Group setting and the VPC Peering connection. If we used PrivateLink Endpoints, then I know that job a/b/c only need access to the Kubernetes API, and job d/e/f only need access to some other resource. This would be something we inherently get by having explicit endpoints across VPCs for specific resources.

Hopefully this all makes sense, it felt a little "wordy".

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[mikestef9]

Thanks for the feedback. Closing this as a duplicate of #298.

[ingshtrom]

@mikestef9 I do not agree, but it could be that I'm not understanding something correctly.

#298 is talking about the "AWS EKS [management] API", not the Kubernetes API server, which is what I'm requesting here.

So right now we have private endpoints for the Kubernetes API server, but that is limited to a single VPC. I want to have a Private Kubernetes endpoint but then have that accessible across VPCs via PrivateLink (or if there is a better way to accomplish this).

[jhnclvr]

VPC peering is a huge solution to a small problem in the case where the only thing you want to share is an EKS endpoint. Having privatelink work for an EKS endpoint would be great.

[mam8270]

+1 Having this seems like it would be a huge win and reduce complexity for customers. Plus from a security perspective, sometimes you really don't want to peer the whole vpc.

[NeoTheElder]

+1 I have a VPC which has private endpoint enabled, however I don't want to connect another VPC via VPC peering for kubectl based access to the api server, one of the use cases is to have duplicate IPs across VPCs, and still be able to have access to api server. Is this possible?

[presich]

We have the same problem as author says. EKS (Account A, VPC A) is being managed by Terraform (Account B, VPC B) using EKS API server endpoint.
I'd like to have a private connection between, but so far it is only possible either by VPC Peering which provides a lot more security risks with it, or via Public endpoint option.

[rheyer]

+1 I have hashicorp vault in a dedicated vpc and would like to securely use kubernetes authentication for many eks clusters. I can't vpc peer because of overlapping cidrs and I do not want the kubernetes apis exposed over the internet. @mikestef9 do you know of a solution that would work for this?

Can i rig something up with a privatelink -> nlb -> lambda in vpc -> call to kubernetes api?

[pingod]

may this two post could help you:
https://stevehorsfield.wordpress.com/2019/09/20/aws-eks-tunneling-a-private-kube-apiserver/

https://www.freshworks.com/saas/network-access-for-private-clusters-blog/

[wjf3121]

Kindly ping. Do we have plan to support this feature?

The workarounds in the previous comment require either setting up a proxy server + a NLB or a NLB + a lambda function tracking IP changes, which are not ideal for such a small requirement.

Thanks!

Update: kindly ping:) @mikestef9

[jessesanford]

How can Amazon's new VPC Lattice service help with this? https://aws.amazon.com/vpc/lattice/ can we target the EKS master node api server endpoints as services in some way?

[viktoraws]

Any updates on this ask?

[jessesanford]

Is there any chance that Amazon will make the Kubernetes API available via the Amazon API? Already we have read access via the console: https://docs.aws.amazon.com/eks/latest/userguide/view-kubernetes-resources.html, Why not open up the "cluster communication service" to the AWS CLI as well? @mikestef9?

[mikestef9]

While not a native feature of EKS, we have published a blog with a self managed solution

https://aws.amazon.com/blogs/containers/enable-private-access-to-the-amazon-eks-kubernetes-api-with-aws-privatelink/

[jessesanford]

For those who are interested, I have formalized my ask for using VPC lattice to expose the K8s API on Private Endpoint Clusters here: #2249 while it is not the same solution as proposed here, I have tested it, and it does work once the bootstrapping issue is solved by the EKS Addon functionality.

[nitrocode]

FYI official documentation of the NLB workaround https://aws.amazon.com/blogs/containers/enable-private-access-to-the-amazon-eks-kubernetes-api-with-aws-privatelink/

[saedx1]

If you have a TCP service (which the K8s API Server is), then you can expose it as a VPC Endpoint through an NLB, although it would be nice if AWS supports that out of the box for K8s API Server.