[deploy/eck-operator] Don't deploy CA bundle if manageCerts is true

[maartengo]

Bug Report

What did you do?
Try to deploy eck-operator chart v2.7.0 through ArgoCD.

What did you expect to see?
Everything in sync. I expect the caBundle of the webhook to not be set when it is not configured, especially not when manageCerts is true. This behavior was changed in #6489

What did you see instead? Under which circumstances?
Argo keeps going out-of-sync because the webhook caBundle desired state is Cg==, but the eck operator also sets the value.
This happens when using the defaults for the webhook config.

Environment

• ECK version:

  https://github.com/elastic/cloud-on-k8s/releases/tag/v2.7.0

• Kubernetes information:
  Irrelevant

• Values:

eck-operator:
  imagePullSecrets:
    - name: registry-image-pull-secret
  image:
    repository: $privateRegistryUrl/docker.elastic.co/eck/eck-operator
  replicaCount: 2

• Logs:

{"log.level":"info","@timestamp":"2023-04-03T14:38:18.673Z","log.logger":"webhook-certificates-controller","message":"Starting reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2476","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:18.690Z","log.logger":"webhook-certificates-controller","message":"Creating new webhook certificates","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2476","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co","webhook":"elastic-operator.eckoperator.k8s.elastic.co","secret_namespace":"eckoperator","secret_name":"elastic-operator-webhook-cert"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:19.036Z","log.logger":"webhook-certificates-controller","message":"Ending reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2476","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co","took":0.362917211}
{"log.level":"info","@timestamp":"2023-04-03T14:38:19.036Z","log.logger":"webhook-certificates-controller","message":"Starting reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2477","namespace":"eckoperator","validating_webhook_configuration":"elastic-operator-webhook-cert"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:19.065Z","log.logger":"webhook-certificates-controller","message":"Ending reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2477","namespace":"eckoperator","validating_webhook_configuration":"elastic-operator-webhook-cert","took":0.028636967}
{"log.level":"info","@timestamp":"2023-04-03T14:38:19.065Z","log.logger":"webhook-certificates-controller","message":"Starting reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2478","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:19.094Z","log.logger":"webhook-certificates-controller","message":"Ending reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2478","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co","took":0.028348574}
{"log.level":"info","@timestamp":"2023-04-03T14:38:34.204Z","log.logger":"webhook-certificates-controller","message":"Starting reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2479","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:34.223Z","log.logger":"webhook-certificates-controller","message":"Creating new webhook certificates","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2479","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co","webhook":"elastic-operator.eckoperator.k8s.elastic.co","secret_namespace":"eckoperator","secret_name":"elastic-operator-webhook-cert"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:34.482Z","log.logger":"webhook-certificates-controller","message":"Ending reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2479","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co","took":0.277682406}
{"log.level":"info","@timestamp":"2023-04-03T14:38:34.482Z","log.logger":"webhook-certificates-controller","message":"Starting reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2480","namespace":"eckoperator","validating_webhook_configuration":"elastic-operator-webhook-cert"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:34.512Z","log.logger":"webhook-certificates-controller","message":"Ending reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2480","namespace":"eckoperator","validating_webhook_configuration":"elastic-operator-webhook-cert","took":0.029903424}
{"log.level":"info","@timestamp":"2023-04-03T14:38:34.512Z","log.logger":"webhook-certificates-controller","message":"Starting reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2481","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co"}
{"log.level":"info","@timestamp":"2023-04-03T14:38:34.547Z","log.logger":"webhook-certificates-controller","message":"Ending reconciliation run","service.version":"2.7.0+0ef8d5e3","service.type":"eck","ecs.version":"1.4.0","iteration":"2481","namespace":"","validating_webhook_configuration":"elastic-operator.eckoperator.k8s.elastic.co","took":0.034575581}

[maartengo]

Example of out-of-sync state:

[thbkrkr]

I see the problem. We should be able to improve this by including the webhook client configuration CA only when certificates are not managed by the operator.

As an immediate workaround, to support objects which are immediately 'out-of-sync' after creation ArgoCD has the ignoreDifferences setting: https://argoproj.github.io/argo-cd/user-guide/diffing/.