Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ setup-envtest: download binaries from controller-tools releases #2811

Merged
merged 2 commits into from
May 22, 2024

Conversation

sbueringer
Copy link
Member

@sbueringer sbueringer commented May 3, 2024

setup-envtest today downloads archives with envtest binaries from the kubebuilder GCS bucket. Given kubernetes/k8s.io#2647 we have to ASAP move away from this.

@vincepri and I tried and discussed various options (GitHub packages, k8s staging GCS bucket & registry, k8s prod registry) and we eventually ended up with publishing envtest binaries to controller-tools releases. The reason was that this is the only supported place to host artifacts in the k8s ecosystem for us:

  • GitHub packages: not allowed (see Slack thread / Usage of Github Packages / Container repository kubernetes/org#4514 (comment))
  • k8s staging GCS bucket: retention of 60 days
  • k8s staging image registry: it's just staging, seems wrong to host "prod artifacts" there, not possible to easily consume envtest binary archives with e.g. curl
  • k8s prod image registry: requires additional effort for image promotion, not possible to easily consume envtest binary archives with e.g. curl

So our new envtest binary archive "repository" looks like this:

This PR now extends setup-envtest so that it can download envtest archives from the controller-tools repo. This is also the new default. This PR also deprecates the functionality to download from the kubebuilder bucket or from GCS buckets in general. We anticipate that this will be removed with v0.19 or v0.20. We only keep it around for now as a fallback option.

The UX looks like this:

  • Per default setup-envtest will use the envtest binary archives from controller-tools
  • To download from a custom index --index can be used
  • To use the kubebuilder GCS bucket --use-deprecated-gcs can be used (for now, until either the flag is removed or the bucket goes down, whatever comes first)
  • For more details, see the README.md

TBD how we backport this. My initial idea would be to backport it to release-0.17 with --use-deprecated-gcs enabled per default (I don't want to change the behavior in a patch version of setup-envtest)

I did the following manual testing:

# controller-tools
$ go run ./main.go list
(available)  v1.30.0  darwin/arm64
(available)  v1.29.4  darwin/arm64
(available)  v1.28.0  darwin/arm64

$ go run ./main.go use
Version: 1.30.0
OS/Arch: darwin/arm64
sha512: 48c1fa5555f021f8aa365faaa96b106804ce5e3022142a2a166bc91ecd5b22da79f012d20c1a4963e0236df00a214f3311cdd842de4ab19442ca1c83b3bcb859
Path: /Users/buringerst/Library/Application Support/io.kubebuilder.envtest/k8s/1.30.0-darwin-arm64

$ ls -la "/Users/buringerst/Library/Application Support/io.kubebuilder.envtest/k8s/1.30.0-darwin-arm64"
total 372048
dr-xr-xr-x@ 5 buringerst  staff        160 May  6 19:16 .
drwxr-xr-x@ 3 buringerst  staff         96 May  6 19:16 ..
-r-xr-xr-x@ 1 buringerst  staff   26294882 May  6 19:16 etcd
-r-xr-xr-x@ 1 buringerst  staff  112793586 May  6 19:16 kube-apiserver
-r-xr-xr-x@ 1 buringerst  staff   51395554 May  6 19:16 kubectl

$ go run ./main.go cleanup
removed 1.30.0 (darwin/arm64)

# GCS
$ go run ./main.go list --use-gcs
(available)  v1.30.0  darwin/arm64
(available)  v1.29.3  darwin/arm64
(available)  v1.29.1  darwin/arm64
...

$ go run ./main.go use --use-gcs
Version: 1.30.0
OS/Arch: darwin/arm64
md5: Y8pPj/mvjlDyp8XIiCylDQ==
Path: /Users/buringerst/Library/Application Support/io.kubebuilder.envtest/k8s/1.30.0-darwin-arm64

$ ls -la "/Users/buringerst/Library/Application Support/io.kubebuilder.envtest/k8s/1.30.0-darwin-arm64"
total 371512
dr-xr-xr-x@ 5 buringerst  staff        160 May  6 19:18 .
drwxr-xr-x@ 3 buringerst  staff         96 May  6 19:18 ..
-r-xr-xr-x@ 1 buringerst  staff   26019842 May  6 19:18 etcd
-r-xr-xr-x@ 1 buringerst  staff  112793586 May  6 19:18 kube-apiserver
-r-xr-xr-x@ 1 buringerst  staff   51395554 May  6 19:18 kubectl

$ go run ./main.go cleanup
removed 1.30.0 (darwin/arm64)

Additional infos:

Old

New

Testing this via this PR in CAPI: kubernetes-sigs/cluster-api#10569

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 3, 2024
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 3, 2024
@sbueringer sbueringer changed the title [WIP][DON'T REVIEW] setup-envtest: download binaries from controller-tools releases [WIP][DON'T REVIEW] setup-envtest: Allow downloading binaries from controller-tools releases May 6, 2024
@sbueringer sbueringer changed the title [WIP][DON'T REVIEW] setup-envtest: Allow downloading binaries from controller-tools releases [WIP] ✨ setup-envtest: Allow downloading binaries from controller-tools releases May 6, 2024
@sbueringer sbueringer changed the title [WIP] ✨ setup-envtest: Allow downloading binaries from controller-tools releases ✨ setup-envtest: Allow downloading binaries from controller-tools releases May 6, 2024
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2024
@sbueringer
Copy link
Member Author

/assign @vincepri @alvaroaleman

/cc @chrischdi @fabriziopandini (fyi)

@sbueringer
Copy link
Member Author

cc @camilamacedo86

@sbueringer
Copy link
Member Author

/hold

Want to create a few more envtest binary archives via controller-tools releases after this PR is approved and before it is merged

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 6, 2024
tools/setup-envtest/env/env.go Outdated Show resolved Hide resolved
tools/setup-envtest/env/env.go Outdated Show resolved Hide resolved
tools/setup-envtest/env/env.go Outdated Show resolved Hide resolved
Copy link
Member

@camilamacedo86 camilamacedo86 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you a lot !!!
It seems great

@sbueringer
Copy link
Member Author

Sorry for the push force. Forgot to push a few small improvements before

@sbueringer sbueringer changed the title ✨ setup-envtest: Allow downloading binaries from controller-tools releases ✨ setup-envtest: download binaries from controller-tools releases May 7, 2024
@sbueringer sbueringer deleted the pr-setup-envtest-ct-rel branch May 22, 2024 17:59
damdo added a commit to damdo/cluster-api-provider-aws that referenced this pull request May 30, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
Please enter the commit message for your changes. Lines starting
with '#' will be ignored, and an empty message aborts the commit.
damdo added a commit to damdo/cluster-api-provider-gcp-1 that referenced this pull request May 30, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
k8s-infra-cherrypick-robot pushed a commit to k8s-infra-cherrypick-robot/cluster-api-provider-gcp that referenced this pull request May 30, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
damdo added a commit to damdo/cluster-api-provider-aws that referenced this pull request May 31, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
Please enter the commit message for your changes. Lines starting
with '#' will be ignored, and an empty message aborts the commit.
damdo added a commit to damdo/cluster-api-provider-gcp that referenced this pull request Jun 4, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
r4f4 added a commit to r4f4/installer that referenced this pull request Jun 6, 2024
r4f4 added a commit to r4f4/installer that referenced this pull request Jun 6, 2024
To align with
kubernetes-sigs/controller-runtime#2811.

Also we now have access to the same version we use in the release which
is 1.29.5.
jas-nik pushed a commit to jas-nik/cluster-api-provider-aws that referenced this pull request Jun 26, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
Please enter the commit message for your changes. Lines starting
with '#' will be ignored, and an empty message aborts the commit.
fad3t pushed a commit to fad3t/cluster-api-provider-aws that referenced this pull request Jul 25, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
Please enter the commit message for your changes. Lines starting
with '#' will be ignored, and an empty message aborts the commit.
fad3t pushed a commit to fad3t/cluster-api-provider-aws that referenced this pull request Oct 8, 2024
This is a commit from CR main (22.05.2024).
Intentionally using a commit from main to use a setup-envtest version
that uses binaries from controller-tools, not GCS.
CR PR: kubernetes-sigs/controller-runtime#2811
Please enter the commit message for your changes. Lines starting
with '#' will be ignored, and an empty message aborts the commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants