-
Notifications
You must be signed in to change notification settings - Fork 320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[EKS] [request]: Support for OIDC endpoint with PrivateLink EKS VPC Endpoint #2038
Comments
Thank you filing the report. Can you also explain the use case for connecting to Generally, when pods requests IAM credentials using the OIDC JWT vended by Kubernetes cluster, STS internally reaches out to Cluster's OIDC endpoint to validate the token. This request to OIDC endpoint is handled between the AWS services and shouldn't be originating from worker VPC. Update: https://github.com/weaveworks/eksctl/blob/main/pkg/iam/oidc/api.go#L156 (looks like eksctl is querying the enspoint to get the CA thumbprint). Called out in their notes section https://eksctl.io/usage/eks-private-cluster/ one possible workaround at the moment for customers facing the issue with eksctl - running the following sample command using IAM VPCE can unblock you in the meantime - |
Similar to eksctl usecase, the Terraform aws_eks_cluster resource documentation recommends retrieving the CA thumbprint via this endpoint which is not accessible within the VPC if the EKS endpoint is enabled. |
This is really a frustrating limitation when using terraform and atlantis from within an AWS VPC. |
Just spend a two days trying to figure out why Terraform being run in an AWS Cluster with Harness IaC Agent was failing with this error .... Thanks @heggenu for filing this ticket! |
This breaks CAPI as well. When running a management cluster for CAPI in a VPC, and it tries to create an EKS cluster with OIDC enabled. |
🌟🌟🌟Workaround for partially Air-Gapped VPCs🌟🌟🌟
With above configuration, |
Thanks for the solution description, @punkwalker ! From the cluster (API) one can apply a similar fix, using the coredns configMap (assuming there is a NAT route to 8.8.8.8 from the control nodes:
|
We have partially solved at least one of the reasons why this request was opened in the first place
EKS has introduced a new dual stack compatible endpoint for OIDC issuer url, with the following format
This is not currently returned as part of describe cluster. We are still debating whether to update the existing field or introduce a new field in the DescribeCluster response. But for now, you can manually construct this URL based the existing value of https://docs.aws.amazon.com/eks/latest/APIReference/API_OIDC.html#AmazonEKS-Type-OIDC-issuer For example if your cluster has a value of
you can instead use
If you enable PrivateLink for the EKS management endpoint, this OIDC url will be accessible, but it's important to note, this URL is still only accessible publicly, not over PrivateLink. But it does solve the original reason the issue being opened, of the OIDC endpoint not being reachable at all when EKS PrivateLink was enabled. This is currently available in commercial and GovCloud regions. |
trying from my air-gapped cluster to reach the oidc:
anything im missing? |
Reopening the issue as I provided incorrect guidance before, please see the updated comment #2038 (comment) |
Community Note
Tell us about your request
What do you want us to build?
Ability to use the EKS VPC Endpoint together with OIDC
Which service(s) is this request for?
EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We want to provide centralized VPC Endpoints for commonly used services, when the EKS endpoint came out, we wanted to offer private connectivity to this endpoint to avoid going out to the internet.
When we deployed the central VPC Endpoint for EKS
com.amazonaws.${region}.eks
this causes the DNS lookup foroidc.eks.${region}.amazonaws.com.
to fail as OIDC is not supported by the VPC Endpoint.Are you currently working around this issue?
How are you currently solving this problem?
Use the public Endpoint. Toggle
Private DNS names enabled
to false for the EKS VPC endpoint. Alternatively, removing the EKS VPC Endpoint if not in use at all.Additional context
Anything else we should know?
Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
The text was updated successfully, but these errors were encountered: