Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate away from google.com gcp project kubebuilder #3230

Closed
3 of 10 tasks
camilamacedo86 opened this issue Feb 16, 2023 · 11 comments
Closed
3 of 10 tasks

Migrate away from google.com gcp project kubebuilder #3230

camilamacedo86 opened this issue Feb 16, 2023 · 11 comments
Assignees
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@camilamacedo86
Copy link
Member

camilamacedo86 commented Feb 16, 2023

What do you want to happen?

Description

Today we use GCP and have a project, kubebuilder. We need to move this infrastructure as described in this task: kubernetes/k8s.io#2647. Currently, the GCP infra is used to build some artefacts.

kube-rbac-proxy images

These images are used to build a sidecar for the manager, see:

containers:
- name: kube-rbac-proxy
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1

To know how they are built, you can check it here.

kubebuilder-tools

The kubebuilder tools ship the required binaries to test the projects using EnvTest. We scaffold a target to download them in the projects as well:

.PHONY: test
test: manifests generate fmt vet envtest ## Run tests.
KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test ./... -coverprofile cover.out

To know how they are built, you can check it here

Goal

This task aims to migrate the infrastructure used to build and provide them from the new location.

Motivation

To know more about see: https://kubernetes.io/blog/2020/05/27/an-introduction-to-the-k8s-infrastructure-working-group/

Impact

The projects built so far will still need to work on the images scaffold
It is a breaking change and can indeed break the workloads running on the cluster (in the case of the proxy image).

So, we will need to work on providing those from the new places, see if we can also make a copy from all artefacts built and provide so for to the new place, and communicate asap.

IMPORTANT: We should not stop building in the old/current infrastructure until we stop using it. Therefore, that means we need to build in both for a time. I would also suggest that you build these artefacts in the new infrastructure.

What do we need to do

  • - Create the new infrastructure - Create infrastructure for kubebuilder artifacts kubernetes/k8s.io#4586
  • - Add the jobs to start to build and produce the artefacts in the new infra : https://github.com/kubernetes/test-infra/pull/29351/files
  • - Ensure that we will have a target to build the image using GCP and another using the new infra, see; ✨ add new build for new k8s registry infra-prow #3362
  • - Make a copy from all that is provided today in the current locations to the new one
  • - Update go/v3 and go/v4 scaffolds to gather the image from the new locations
  • - Update go/v3 and go/v4 makefile target to download the kubebuilder-tools from the new location
  • - Update kubebuilder docs to point out the new location
  • - Could you please check if the controller runtime will not be impacted and address the required changes? If they are using those artefacts for the tests, then those tests need to update to gather them from the new location.
  • - Communicate broadly to the community and ask for they to update their project to gather them from the new location
  • - disable the jobs that runs in the current infrastructure

References

kubernetes/k8s.io#2647
kubernetes/k8s.io#4586

Extra Labels

No response

@camilamacedo86 camilamacedo86 added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. kind/feature Categorizes issue or PR as related to a new feature. labels Feb 16, 2023
@yashsingh74
Copy link
Member

I can start looking into this.

@camilamacedo86
Copy link
Member Author

HI @yashsingh74,

Thank you for looking into that. But this one is not a good first issue to work on. We will need @rpkatz help to know how to use the new infra and how to promote the assets. You might able to help us out when we have those built, and then we can start to change the docs and scaffolds to use them.

@camilamacedo86
Copy link
Member Author

/assign @rikatz

@rikatz
Copy link
Contributor

rikatz commented Feb 20, 2023

Hi there. I wrote something about it on a huge thread, and copy/pasted to https://docs.google.com/document/d/18EKmym3YJ0Ey3LOrQOeWh6RbnSO-odfKuMNVlHPHDmE/edit?usp=sharing (SORRY!!)

@yashsingh74 we have some subtasks here, as:

We are going to need to tweak some minor stuff on build script, like pushing to the right repository, etc, using proper substitutions on cloudbuild, etc.

@camilamacedo86 If you and @varshaprasad96 I would like to do the "image replication" work here, mostly getting the images from old repo and pushing to new staging one, so we won't start the manifest file promotion from scratch :)

@camilamacedo86
Copy link
Member Author

I wanted to document the most recent status:

From the current outlook, it seems unlikely that we will be able to avoid using GCP, unless there's a proposal to change the aforementioned binaries.

@rikatz
Copy link
Contributor

rikatz commented Sep 1, 2023

On kube-rbac-proxy feature, @sbueringer implemented kubernetes-sigs/controller-runtime#2407 on latest controller-runtime.

I didnt had a chance to look at it yet but have seen some talks about krp not being required anymore with this feature :)

Maybe once kubebuilder supports the new runtime, there's no need to build krp anymore and simply migrate the old images.

@rikatz
Copy link
Contributor

rikatz commented Sep 1, 2023

Ah sorry just saw your latest Pr To bump controller-runtime :))) great! So disregard my comment, but maybe we can not promote krp anymore in favor of the new feature :)

@sbueringer
Copy link
Member

@rikatz There's a lot of related information in this lengthy Slack thread: https://kubernetes.slack.com/archives/CAR30FCJZ/p1693377335373059

@camilamacedo86
Copy link
Member Author

The latest update of this one is:

  • Controller-Runtime maintainers are looking to try to build the envtest binaries in there (in controller-runtime or controller-tools)
  • We could not make the kube-rbac-proxy image rebuild work within the new infra because it is not under the k8s umbreally yet.

Further info: https://kubernetes.slack.com/archives/CCK68P2Q2/p1711913605487359

@BenTheElder
Copy link
Member

If there are any images other than kube-rbac-proxy that are not just being phased out, they need to be moved to registry.k8s.io sooner than later, there are docs at https://registry.k8s.io, specifically https://github.com/kubernetes/k8s.io/tree/main/registry.k8s.io#managing-kubernetes-container-registries

(We should just move them, rather than migrating to AR in a google.com internal project, we have plans in motion to migrate ~everything in Kubernetes to be solely on community controlled infra by EOY and GCR requires action by early next year anyhow. While technically we could move GCR to AR ~in-place, it makes more sense to switch to community controlled resources while we're at it)

@camilamacedo86
Copy link
Member Author

Conclusion and Latest Status

The GCP project must be active until March 18, 2025. After that date then theoretically Kubebuilder GCP project can be shotdown, Google has announced that images under gcr.io/kubebuilder will no longer be available. While Kubebuilder no longer promotes new images or artifacts through this repository, some projects built with older versions may still be using images like gcr.io/kubebuilder/kube-rbac-proxy in production environments.

Therefore, we must keep it running for as long as possible, until these older projects have fully migrated away. However, the goal of this issue was ensure that we migrate away from GCP project and from our side, everything has been done to discontinue the GCP usage.

Below are the details of what was done and the current status:

  • Kubebuilder releases are now managed via GoReleaser and GitHub Actions. For over 1 or 2 years, they no longer rely on GCP to trigger builds.
  • images from kube-rbac-proxy project used to protect metrics endpoints Projects initialized with Kubebuilder release versions >= [v3.15.0](https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v3.15.0) are no longer scaffolded with the kube-rbac-proxy dependency. Instead, they use the built-in features provided by controller-runtime (filters.WithAuthenticationAndAuthorization) by default. You can see an example here. Furthermore, users are given an extra optional option to protect their metrics endpoint with network policies, as shown here. This change was communicated via the mailing list, Slack channels, release notes, and a discussion topic. For more details, see the discussion: 🚨 Action Required: Ensure that you no longer use gcr.io/kubebuilder images #3907
  • kubebuilder-tools tarballs which provides the EnvTest binaries The artifacts provided via https://storage.googleapis.com/kubebuilder-tools, which contain binaries to allow users use the Controller-Runtime ENVTEST library to test their controllers, are now built and released via controller-tools. You can find the releases:https://github.com/kubernetes-sigs/controller-tools/releases. Since the release of controller-runtime v0.19, the setup-envtest downloads tarballs from the new location. Any projects generated with Kubebuilder >= [v4.2.0](https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v4.2.0)—which adds support for controller-runtime v0.19 and Kubernetes 1.31—will use this new location. Projects using Kubernetes 1.31 for testing will need to adopt the new location, which ensures a smooth transition. This was communicated via the mailing list, Slack channels, release notes, and a discussion topic. For further information, see the discussion: 🚨 Action Required: Ensure that you no longer use https://storage.googleapis.com/kubebuilder-tools #4082.
  • The image used in the GitHub Action to validate the PR title (from kubebuilder-release-tools) will also break once GCP images are unavailable. This project is not actively maintained, but Kubebuilder has implemented a simple solution to achieve the same functionality with GitHub Actions and shell scripts. The PR to update the documentation is here. Once this PR is merged, I am commit to share the update via the mailing list and channels to ensure everyone is informed.

Therefore, from our side, everything is ready. We just need to give projects enough time to migrate before we fully discontinue the GCP usage and I am closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

No branches or pull requests

6 participants