The renewal process with custom CA does not work

[scholzj]

Discussed in #5437

^{Originally posted by FrankWang1108 August 17, 2021}
Describe the bug
This happens when we try to install our own CA when the strimzi is running and the strimzi ca is in use. It could also happen when updating the CA.

It happens after uploading new CA and certs into the *-cluster-ca-certs.
Following the renew own CA document, we have both ca.crt and ca-dates.crt in the cluster-ca-certs, and because the keystore and old ca-dates.crt can trust each other, there should still have connection. This is what is expected and the connection between zk and broker are stable because of this.

However the ClusterOperator only uses ca.crt to check for connection between zk and broker. Therefore after uoloading the new CA and certs. The ClusterOperator loses connection to zk and broker, it tries to connect to it multiple times until eventually it will perform a rolling update for zk and broker.
Once the zk updated, it will pickup the new keystore, then it will be able to connect to ClusterOperator again.

To Reproduce
Steps to reproduce the behavior:

1. bring up a cluster
2. change the cluster-ca and cluster-ca-cert to your own ca
3. check the log of ClusterOperator to see it trying to connect to zk and broker.
4. See error

Expected behavior
I would expect when the ClusterOperator trying to connect to zk and broker, it will use all .crt files in the cluster-ca-cert to check for connection, instead of just using ca.crt.
I would expect it to stay connected and trigger a rolling update because it sees new certs updates, not because it was unable to connect.

Environment (please complete the following information):

• Strimzi version: 0.25.1 (but I think the bug is still there for newer versions)
• Installation method: YAML files

[scholzj]

The renewal process with custom CA currently does not work as described in the docs and needs to be fixed.

[FrankWang1108]

based on the slack discussion, the title doesn't represent the actual issue anymore. It should be something like "The renewal process with customer CA does not work"

Describe the bug
Following the current renew own CA document. after the own CA being renewed, It doesn't pod trigger update and secret regeneration. It should be fixed.

Expected behavior
After the custom CA being renewed, the broker and zk cert should be regenerated using the new CA and it should trigger a pod update.
based on discussion https://cloud-native.slack.com/archives/CMH3Q3SNP/p1637277974083600?thread_ts=1637276658.082400&cid=CMH3Q3SNP

[scholzj]

@FrankWang1108 good point, I renamed it.