Create CRDs V1

[sebgl]

Bump the current v1beta1 CRD to a separate v1 CRD with the exact same content. As such, it's fine to rely on the OpenAPI validation schema of the v1 version to validate the v1beta1 resources.

I tried to make sure every single reference we had to the v1beta1 API is updated to use the v1 API now. Along with that change I tried to homogenize the imports so we always import esv1, kbv1, etc. instead of using a mix of estype, esv1beta1, elasticsearchv1beta1, v1beta1, etc.

Commit messages show high-level changes.

I made sure the webhook for v1 works fine with the all-in-one version, but I have some doubts on:

• the v1beta1 webhook (alongside the v1 webhook)
• webhooks for the global + namespace operators
  I'll investigate and maybe fix them in a follow-up PR.

Fixes #2117.

[thbkrkr]

I went relatively quickly on all the files, played a little with grep -r, generated and applied all-crds.yml and everything seems ok.

I've the same concerns about the webhook but let's merge this PR and fix them if needed after to avoid too many "conflicts fix".

LGTM!

[sebgl]

API docs generation does not produce a consistent order, which messes up with CI:

Error: dirty local changes
12:37:05   M docs/api-docs.asciidoc

I'll see what I can do here.

[charith-elastic]

API docs generation does not produce a consistent order

That's odd. It should produce output sorted by the API name and version. Let me know if you need any help.

[barkbay]

I did the following:

• Apply all-in-one with v1beta1
• Deploy an Elasticsearch instance
• Apply all-in-one with v1

I have the following error:

"error":"Object default/elasticsearch-sample-es-http-certs-internal is already owned by another Elasticsearch controller elasticsearch-sample"

Is it related to the one you got yesterday ?

[sebgl]

elastic/gen-crd-api-reference-docs#1 should fix the inconsistent api docs rendering.

[sebgl]

@barkbay I get the same error you get, probably because the operator with CRD v1 is trying to add elasticsearch/v1 as ownerReference to the secret that already has elasticsearch/v1beta1 as ownerReference 😕

The corresponding code in controller-runtime does a strict comparison between both resource versions:

for i, r := range existingRefs {
		if referSameObject(ref, r) {
			fi = i
		} else if r.Controller != nil && *r.Controller {
			return newAlreadyOwnedError(object, r)
		}
	}

func referSameObject(a, b metav1.OwnerReference) bool {
	aGV, err := schema.ParseGroupVersion(a.APIVersion)
	if err != nil {
		return false
	}

	bGV, err := schema.ParseGroupVersion(b.APIVersion)
	if err != nil {
		return false
	}

	return aGV == bGV && a.Kind == b.Kind && a.Name == b.Name
}

I created an issue in the controller-runtime repository: kubernetes-sigs/controller-runtime#699.
And an issue in our own: #2191

kubernetes-sigs/controller-runtime#699.

[sebgl]

I found another blocker for this: #2192

[sebgl]

#2211 should fix the owner reference problem.

[anyasabo]

I tried to homogenize the imports so we always import esv1, kbv1, etc. instead of using a mix of estype, esv1beta1, elasticsearchv1beta1, v1beta1, etc.

👏👏👏 should make refactors easier in the future

[sebgl]

#2217 should fix the StatefulSet volumeClaimTemplates problem.

[sebgl]

The PR is ready for another round of reviews.
I did some manual testing following the procedure outlined in #2218 (comment).
Everything seems alright as far as I can tell.

The only thing I'm not sure about is the webhook. I created a follow-up issue so it does not delay this PR.

[pebrc]

Maybe I am doing something silly here, that I am not aware of, but when I run the operator based on this PR I see errors on my existing v1beta1 resources.

2019-12-06T11:18:47.656+0100	ERROR	controller-runtime.controller	Reconciler error	{"ver": "1.0.0-beta1-620892a3", "controller": "elasticsearch-controller", "request": "default/es-apm-sample", "error": "Elasticsearch.elasticsearch.k8s.elastic.co \"es-apm-sample\" is invalid: apiVersion: Invalid value: \"elasticsearch.k8s.elastic.co/v1beta1\": must be elasticsearch.k8s.elastic.co/v1", "errorCauses": [{"error": "Elasticsearch.elasticsearch.k8s.elastic.co \"es-apm-sample\" is invalid: apiVersion: Invalid value: \"elasticsearch.k8s.elastic.co/v1beta1\": must be elasticsearch.k8s.elastic.co/v1"}]}
github.com/go-logr/zapr.(*zapLogger).Error

Update: so it looks like it is failing on the status subresource update (we are sending v1 I checked, but presumable the subresource update tries to patch only the status bit and we still have v1beta1 stored on the server)

[sebgl]

@pebrc hm right I did not retest APM. Will look into it, thanks for testing!

[sebgl]

We discussed the bug offline with @pebrc:

• it happens when Status subresources are updated on a resource whose storedVersion is v1beta1
• since the migration from v1beta1 to v1 includes a rewrite of the resource to remove finalizers, I could not reproduce it easily. But @pebrc had resources with finalizers removed already, hence could reproduce it.
• enabling debug logs at the client-go level seems to indicate everything is fine on our side, but there's a bug on the apiserver
• spotted an issue that seems to exactly match this: TestValidateOnlyStatus flakes with apiVersion: Invalid value kubernetes/kubernetes#72493
• it seems resolved in k8s 1.15
• we'll probably need to stop updating the subresource API and update the entire resource directly to avoid the error on pre-1.15 clusters

I'm extracting this to its own issue.

[sebgl]

The status subresource validation error is now fixed with #2229.

[pebrc]

LGTM! (I did not retest but assume we are good now)

[sebgl]

Jenkins test this please.