🌱 Support using custom endpoints for AWS services

[Promaethius]

What this PR does / why we need it: Allows custom endpoints for aws services to be specified for the aws provider. Not only would this allow for local stack testing but would pave the road for Cluster API into AWS IC partitions which require FIPS named endpoints.

Which issue(s) this PR fixes:
Fixes #1855

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

Welcome @Promaethius!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @Promaethius. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Promaethius]

I'm thinking that this might have to extend to the cloud.conf that's generated on controller nodes.

[ServiceOverride "IC"]
  Service=ec2
  Region=us-iso-east-1
  URL=https://ec2.us-iso-east-1.c2s.ic.gov
  SigningRegion=us-iso-east-1

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[randomvariable]

Think this one might be tricky if we need to modify the cloud controller manager on the created workload cluster. As @detiber notes, this might be more appropriate as part of the multi-tenancy work in progress. Will get back to you on how to deal with this.

[Promaethius]

@randomvariable did you get direction? I got word back from aws-go-sdk that C2S endpoints should be specified through this custom parser. I'm worried about scope now cause that may mean if I define custom service endpoints here that they should become available to the Python CLI and the kubelet Cloud provider plugin.

[randomvariable]

@Promaethius I didn't get round to it this week, but promise to answer on Monday.

That said, the custom service endpoints won't have downstream impacts on other processes. The main issue we have here is how to make sure the cloud provider also follows the endpoints. It may be the answer is that you use a kubeadmconfigtemplate and KCP with the appropriate settings.

The wider issue is that we're implementing a CRD that will hold different types of AWS credentials, and we might want to extend it to also include the service endpoints. This is being tracked for v0.6.1 in #1753

[Promaethius]

@randomvariable what I'm thinking for configuring endpoints of the aws cli and kubelet cloud provider:

• Kubelet
  • cloud-init arbitrary file: /etc/kubernetes/cloud.conf
  • kubeadm.conf rendered with --cloud-config /etc/kubernetes/cloud.conf flag
• AWS CLI
  • Render cloud-init script with --endpoint-url in AWS CLI calls OR
  • Extend the CAPA image builder with an AWS CLI plugin which allows service endpoint configuration from a file

Now here's where I have an issue with what @detiber pointed out above. In the case of multi-tennant workloads that require different sets of credentials, there must be logic on the templating side that selects a CRD which corresponds to a machine pool and renders the correct service endpoint. However, pools don't have the ability to select credentials; they're only passed a Profile parameter which is separate from the Manager's Credentials. Unsure of how to approach this.

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[Promaethius]

I backtracked the session handler functionality through both EKS and Machine so it should be working for both. As for the resolver, this is pretty much lifted out from the aws resolver example.

…

On Fri, Sep 25, 2020, 7:09 AM Naadir Jeewa ***@***.***> wrote: ***@***.**** commented on this pull request. Thanks for this. Some minor changes. Also, can we please wire this up in the EKS and bootstrap controllers, so we have consistency across them? ------------------------------ In pkg/cloud/scope/session.go <#1858 (comment)> : > + resolver := func(service, region string, optFns ...func(*endpoints.Options)) (endpoints.ResolvedEndpoint, error) { + for _, s := range endpoint { + if service == s.ServiceID { + return endpoints.ResolvedEndpoint{ + URL: s.URL, + SigningRegion: s.SigningRegion, + }, nil + } + } + return endpoints.DefaultResolver().EndpointFor(service, region, optFns...) + } Move this anonymous function out. Might be the former ruby dev in me, but easier to unit test this way. ------------------------------ In pkg/cloud/endpoints/endpoints.go <#1858 (comment)> : > @@ -0,0 +1,122 @@ +/* +Copyright 2018 The Kubernetes Authors. ⬇️ Suggested change -Copyright 2018 The Kubernetes Authors. +Copyright 2020 The Kubernetes Authors. I know it's my copy-pasta, soz. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1858 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFFNN3ABYF7RZE3YTJRFBLSHR3AJANCNFSM4PT6YNEA> .

[randomvariable]

thanks @Promaethius . could you also update the copyright header to the correct year?

[randomvariable]

I backtracked the session handler functionality through both EKS and Machine so it should be working for both.

Need to add the command line flag to the respective main.gos in bootstrap/eks/main.go and controlplane/main.go .

[Promaethius]

I backtracked the session handler functionality through both EKS and Machine so it should be working for both.

Need to add the command line flag to the respective main.gos in bootstrap/eks/main.go and controlplane/main.go .

@randomvariable Added it to controlplane/main.go but couldn't find a place for AWS Service Context in bootstrap/eks/main.go looks like it only operates against Kubernetes objects so service endpoints is useless there.

[randomvariable]

Can you rebase the branch rather than merging master into it?

[randomvariable]

git rebase HEAD~2 should do it

[randomvariable]

/test ?

[k8s-ci-robot]

@randomvariable: The following commands are available to trigger jobs:

• /test pull-cluster-api-provider-aws-test
• /test pull-cluster-api-provider-aws-build
• /test pull-cluster-api-provider-aws-integration
• /test pull-cluster-api-provider-aws-verify
• /test pull-cluster-api-provider-aws-e2e-conformance
• /test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
• /test pull-cluster-api-provider-aws-e2e

Use /test all to run the following jobs:

• pull-cluster-api-provider-aws-test
• pull-cluster-api-provider-aws-build
• pull-cluster-api-provider-aws-integration
• pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[randomvariable]

/test pull-cluster-api-provider-aws-e2e

[randomvariable]

I think there's something wrong with the resultant bootstrap script. Please take a look, and verify with a local run against AWS.

[Promaethius]

I think there's something wrong with the resultant bootstrap script. Please take a look, and verify with a local run against AWS.

the bash if statement didn't evaluate correctly if there weren't spaces around the quotes :P

[Promaethius]

/test pull-cluster-api-provider-aws-e2e

[Promaethius]

/test pull-cluster-api-provider-aws-e2e

[randomvariable]

Ah, the classic [ is actually a program.

[Promaethius]

@randomvariable e2e tests passed :D

[randomvariable]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: randomvariable

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [randomvariable]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment