Ignore resources created in 0.8.x

[pebrc]

We are not (yet) offering backwards compatibility in ECK. To avoid disruptive operator upgrades we should ignore any cluster created via an 0.8.x operator.

Needs design how to actually do that. Some ideas:

• leverage Move controller revision to annotations #1234 controller revisions annotated on Elasticsearch/Kibana/APM resources in combination with
• some other attribute to identify a resource that has not just been created

Why do we need this 'other' attribute?
Because we would ignore newly created resources otherwise in 0.9 that also don't have a controller revision annotation

• What other attribute could we use?
  We could look at common.k8s.elastic.co/type. So if a resource has common.k8s.elastic.co/type but does not have a controller revision ignore it because it is not created by this operator version.

[sebgl]

We could look at common.k8s.elastic.co/type

Which resource do you have in mind here? I guess any pod/secret/configmap could work, but not the Elasticsearch resource itself?

[pebrc]

I think I did not realise that we are not applying the label I used in the example to the top level resources. So we have to look that the sub resources created by the controller that have those annotations.

[sebgl]

Summary of a discussion with @pebrc:

• The ES resource is not labeled at all when handled by ECK 0.8 (btw it would be a bit strange to label an ES resource with type=elasticsearch since the resource Kind is Elasticsearch already).
• Once we have an annotation in place on the ES resource, it's easy to identify which ECK version handled it. But the annotation is not there when the resource is initially created by the user.
• So we'd like to keep the annotation mechanism to tag versions to resources, but we also need a way to identify resources that were just created and don't have this annotation set (yet).

Proposal:

• At the beginning of each reconciliation, retrieve the current resource version from annotations
  • if the annotation is set, use it (eg. return 0.9)
  • if the annotation isn't set, 2 possibilities:
    a. the resource was just created and does not have the annotation yet
    b. the resource was created with ECK 0.8
  • To detect if we are in case b, list pods and secrets labeled with the ES cluster name (or APM or Kibana): if there are some, then the resource was created with ECK 0.8 (return 0.8). If there are none (length 0), return noVersion. Listing pods and secrets from the cache should be a very cheap operation.
• Check compatibility of the resource version with the operator version. Each ECK version comes with its own comparison logic (but in most cases, it should probably just check the annotated resource version is in the backward compatibility range of the operator version).
• If not compatible, log and return. Don't reconcile.
• If compatible:
  • Make sure the resource is annotated with the current operator version. If it isn't, annotate it and requeue. What we want to avoid here is moving on with creating pods/secrets while the resource isn't yet annotated in the cache (we would end up in the 0.8 scenario if newly created pods and secrets appear in the cache before the annotated resource appears).
  • Move on to normal reconciliation