Migrate existing KB project to v1 CRDs and Webhooks with minimal user effort

[erictune]

Facts:

• 1.16 introduced CRD and Webhooks v1 APIs.
• CRD and Webhooks v1beta1 are deprecated and scheduled for removal in v1.19.
• Kubernetes users use wide range of Kubernetes minor versions.

Consequences:

• In about 12 months, expect clusters running versions 1.15 through 1.19.
• If KB only generates v1beta1 or only generates v1 CRDs/WHs, then my controller can't be installed on that range of clusters.

Request for KubeBuilder:

• Develop CRD and Webhook generation that can emit both v1beta1 and v1 CRDs/WHs.
• Document how a developer can have a single project and a single release artifact and have it work on v1beta1-only, both, and v1-only clusters.
• Don't decide that KubeBuilder v2.x.0 (or likewise for controller-tools, etc) is going to do a hard switch to generating v1 CRDs/WHs. I want to get new features and fixes for KB while still releasing controllers work with 1.15 and earlier Kubernetes versions.

Related issues:

• CRD GA support #903 is about using the new features of v1 CRDs and webhooks. This one is about expressing existing KB projects using v1 types.
• API deprecations on 1.16 #944 is about APIs being deprecated in 1.16. This one is about APIs being deprecated in 1.1

Additional Data about Versions:

• From this slide of the Kubernetes Apps Survey, it appears that most (~95%) of users are running a cluster from one of the 4 most recent minor versions.
• From the detailed data of the same survey in column AN, many users have 3 or even 4 minor versions in use concurrently.
• I'd expect this distribution to broaden over time, so having users spread across 5 versions in 2020 seems likely.

/kind feature

[DirectXMan12]

/priority important-soon

[DirectXMan12]

Don't decide that KubeBuilder v2.x.0 (or likewise for controller-tools, etc) is going to do a hard switch to generating v1 CRDs/WHs. I want to get new features and fixes for KB while still releasing controllers work with 1.15 and earlier Kubernetes versions.

Agreed, that seems untenable. We'll want to continue to generate v1beta1 CRDs in new versions of CT fo quite some time.

[erictune]

In that case, v2 needs to still get periodic updates for at least a year, IMO. Might be less work for the KB maintainers if you stay at v2 and let the user select whether to generate v1beta1 or v1 CRDs.

…

On Tue, Nov 5, 2019 at 3:31 AM Camila Macedo ***@***.***> wrote: Hi @DirectXMan12 <https://github.com/DirectXMan12>, WDYT about if we keep the v2 using the no-structural schemas and the next v3 generating it with apiextensions.k8s.io/v1? Would it be a good approach? c/c @joelanford <https://github.com/joelanford> @hasbro17 <https://github.com/hasbro17> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1065?email_source=notifications&email_token=AB5245QMAERRVZCN7LRDSWLQSFKQJA5CNFSM4I4ZYVD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDCQW7Y#issuecomment-549784447>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5245SQERQTBTGUN5AXZVLQSFKQJANCNFSM4I4ZYVDQ> .

[DirectXMan12]

Yep, that's what we're doing in CT now

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[liggitt]

/remove-lifecycle rotten
/lifecycle frozen

[liggitt]

Is there a plan/timeframe in place for this?

[DirectXMan12]

@liggitt can you clarify what, specifically, you're looking for here?

[DirectXMan12]

AFAIK, all the pieces are in place. There's a couple of switches that need to be flipped manually atm to use GA vs beta, but otherwise this is pretty much done

[liggitt]

I couldn't tell whether kubebuilder could generate new v1 CRDs, or regenerate existing APIs as v1 CRDs. If that's mostly done, that's good news.

[DirectXMan12]

yep -- controller-tools crd:crdVersion=v1 will do it for you :-)

[DirectXMan12]

IIRC, we don't generate it by default because most folks still support (or supported at the time) kubernetes versions that didn't have v1 yet.

[liggitt]

makes sense. is there a plan to default to v1, once 1.15 is out of support?

[camilamacedo86]

@DirectXMan12 in SDK we solve it by adding a flag in order to allow to keep the backwards compatibility. So, it means that by default the resources are scaffold using the latest version but users still able to use the flag in the commands to choose the old one. Do you think that it would be required here? Also, I think this would be a good one to get done now in the V3. WDYT?

c/c @joelanford @estroz

[estroz]

The SDK needs a flag because it calls controller-gen libraries directly, i.e. you cannot set crdVersion=<version> manually. Since controller-gen is being called via make recipe in kubebuilder, docs should be sufficient. I didn't see anything discussing CRD API versions in the docs, so I can take this on if we agree it's just a docs issue.

[camilamacedo86]

Hi @estroz,

IHMO we need to address here:

• Change the project to the scaffold CRD's and Webhooks using the V1 by default
• update the docs and etc

Then, we will have to address the backwards compatibility scenario:

a) if possible, we could doc the steps to manually gen the CRDs and Webhooks in the old version. (I am not sure if it is 100% possible get done just by changing the makefile since the resources are also scaffolded by the tool)

OR

b) solve it as SDK by creating a flag which allows users to choose if would like to still gen in the old version.

[DirectXMan12]

This is totally something we could do in a new layout version in v3, and then switch that to the default once 1.15 is out of support.

[estroz]

v1 CRDs and webhooks generated by default in controller-gen, respectively: kubernetes-sigs/controller-tools#468, kubernetes-sigs/controller-tools#469

[erictune]

As the OP, I'm not sure this original issue is closed yet.

Users may be running a wide range of clusters. From what I gather, 1.14, 1.15, 1.16, 1.17, 1.18 and 1.19 are all moderately widespread at present. How should a KubeBuilder user produce an release that runs on that range of cluster version?

[estroz]

@erictune given the list of 3 things you wanted out of kubebuilder, the only one that doesn't appear to be resolved is

Document how a developer can have a single project and a single release artifact and have it work on v1beta1-only, both, and v1-only clusters.

Is that correct? Was there something else that needs doing?

To that effect: you can target the v1.14-v1.19 cluster range if you use v1beta1 CRDs/webhooks:

kubebuilder init --plugins go/v3-alpha
kubebuilder create api --crd-version v1beta1 ...
kubebuilder create webhook --webhook-version v1beta1 ...

For v1.14 clusters, you need to remove crd:preserveUnknownFields=false from your Makefile.

To target v1.16+ clusters, use defaults:

kubebuilder init --plugins go/v3-alpha
kubebuilder create api ...
kubebuilder create webhook ...

[erictune]

Awesome. Can the text of your last comment, starting after To that effect:, go into the docs somewhere?

I searched but could not find something like it.

[estroz]

Definitely! I'll whip something up now.