Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EKS] [request]: Support for OIDC endpoint with PrivateLink EKS VPC Endpoint #2038

Open
heggenu opened this issue May 26, 2023 · 10 comments
Open
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue

Comments

@heggenu
Copy link

heggenu commented May 26, 2023

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
What do you want us to build?
Ability to use the EKS VPC Endpoint together with OIDC

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We want to provide centralized VPC Endpoints for commonly used services, when the EKS endpoint came out, we wanted to offer private connectivity to this endpoint to avoid going out to the internet.

When we deployed the central VPC Endpoint for EKS com.amazonaws.${region}.eks this causes the DNS lookup for oidc.eks.${region}.amazonaws.com. to fail as OIDC is not supported by the VPC Endpoint.

Are you currently working around this issue?
How are you currently solving this problem?
Use the public Endpoint. Toggle Private DNS names enabled to false for the EKS VPC endpoint. Alternatively, removing the EKS VPC Endpoint if not in use at all.

Additional context
Anything else we should know?

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

@heggenu heggenu added the Proposed Community submitted issue label May 26, 2023
@heggenu heggenu changed the title [EKS] [request]: Support for OIDC endpoint with the private EKS VPC Endpoint [EKS] [request]: Support for OIDC endpoint with PrivateLink EKS VPC Endpoint May 26, 2023
@mikestef9 mikestef9 added the EKS Amazon Elastic Kubernetes Service label May 26, 2023
@SaranBalaji90
Copy link

SaranBalaji90 commented Jun 16, 2023

Thank you filing the report. Can you also explain the use case for connecting to oidc.eks.${region}.amazonaws.com endpoint from the worker nodes?

Generally, when pods requests IAM credentials using the OIDC JWT vended by Kubernetes cluster, STS internally reaches out to Cluster's OIDC endpoint to validate the token. This request to OIDC endpoint is handled between the AWS services and shouldn't be originating from worker VPC.

Update: https://github.com/weaveworks/eksctl/blob/main/pkg/iam/oidc/api.go#L156 (looks like eksctl is querying the enspoint to get the CA thumbprint). Called out in their notes section https://eksctl.io/usage/eks-private-cluster/

one possible workaround at the moment for customers facing the issue with eksctl - running the following sample command using IAM VPCE can unblock you in the meantime - aws iam create-open-id-connect-provider --url <value_from_describe_cluster_api> --client-id-list sts.amazonaws.com --thumbprint-list 9e99a48a9960b14926bb7f3b02e22da2b0ab7280 to create the provider. [To obtain thumbprint https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html]

@hhsnow
Copy link

hhsnow commented Sep 8, 2023

Similar to eksctl usecase, the Terraform aws_eks_cluster resource documentation recommends retrieving the CA thumbprint via this endpoint which is not accessible within the VPC if the EKS endpoint is enabled.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster.html#enabling-iam-roles-for-service-accounts

@plenti-chris-fowles
Copy link

This is really a frustrating limitation when using terraform and atlantis from within an AWS VPC.

@cydergoth
Copy link

Just spend a two days trying to figure out why Terraform being run in an AWS Cluster with Harness IaC Agent was failing with this error .... Thanks @heggenu for filing this ticket!

@hsmade
Copy link

hsmade commented Jun 4, 2024

This breaks CAPI as well. When running a management cluster for CAPI in a VPC, and it tries to create an EKS cluster with OIDC enabled.

@punkwalker
Copy link

punkwalker commented Jul 10, 2024

🌟🌟🌟Workaround for partially Air-Gapped VPCs🌟🌟🌟

  1. Create Route 53 outbound endpoint in VPC.
  2. Create Route53 Resolver Rule and associated it under above Outbound endpoint and associate with the cluster VPC. Use following values for it:
Rule name: <Some_Name>
Rule type: Forward
VPCs that use this rule: Cluster VPC
Domain name: oidc.eks.<region>.amazonaws.com
Outbound endpoint: <Endpoint from step 1>
Target IP addresses: <Public Nameserver> (e.g 8.8.8.8) 
Note: The Security Group/ NACL of Outbound endpoint (Step 1). Should allow egress traffic to target IP
  1. Create EKS VPC Endpoint

With above configuration, *.eks.<region>.amazonaws.com will resolve to Private VPC IPs except oidc.eks.<region>.amazonaws.com will be resolved by Public Nameserver (Step 2) to Public IP address of OIDC.

@seansummers
Copy link

seansummers commented Jul 15, 2024

Thanks for the solution description, @punkwalker !

From the cluster (API) one can apply a similar fix, using the coredns configMap (assuming there is a NAT route to 8.8.8.8 from the control nodes:

-  forward . /etc/resolv.conf
+  forward . /etc/resolv.conf {
+    except oidc.eks.us-east-1.amazonaws.com
+  }
+  forward oidc.eks.us-east-1.amazonaws.com 8.8.8.8

@mikestef9
Copy link
Contributor

mikestef9 commented Aug 9, 2024

We have partially solved at least one of the reasons why this request was opened in the first place

When we deployed the central VPC Endpoint for EKS com.amazonaws.${region}.eks this causes the DNS lookup for oidc.eks.${region}.amazonaws.com. to fail as OIDC is not supported by the VPC Endpoint.

EKS has introduced a new dual stack compatible endpoint for OIDC issuer url, with the following format

oidc-eks.region.api.aws

This is not currently returned as part of describe cluster. We are still debating whether to update the existing field or introduce a new field in the DescribeCluster response.

But for now, you can manually construct this URL based the existing value of https://docs.aws.amazon.com/eks/latest/APIReference/API_OIDC.html#AmazonEKS-Type-OIDC-issuer

For example if your cluster has a value of

https://oidc.eks.us-west-2.amazonaws.com/id/3187F794DDC03F340D6E663ECE675D9D

you can instead use

https://oidc-eks.us-west-2.api.aws/id/3187F794DDC03F340D6E663ECE675D9D

If you enable PrivateLink for the EKS management endpoint, this OIDC url will be accessible, but it's important to note, this URL is still only accessible publicly, not over PrivateLink. But it does solve the original reason the issue being opened, of the OIDC endpoint not being reachable at all when EKS PrivateLink was enabled.

This is currently available in commercial and GovCloud regions.

@erdeiimre
Copy link

erdeiimre commented Sep 6, 2024

trying from my air-gapped cluster to reach the oidc:

curl -v https://oidc-eks.eu-central-1.api.aws/id/...
trys on public ip:

Trying 3.126.180.18...
despite i have an endpoint for the eks endpoint and seems working (resolves it to private interface):

 curl -v https://eks.eu-central-1.api.aws
* Rebuilt URL to: https://eks.eu-central-1.api.aws/
*   Trying 172.20.74.39...

anything im missing?

@mikestef9
Copy link
Contributor

mikestef9 commented Sep 23, 2024

Reopening the issue as I provided incorrect guidance before, please see the updated comment #2038 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue
Projects
None yet
Development

No branches or pull requests

10 participants