[cert-manager-setup] allow for multiple clusterissuers

[alejandroEsc]

This change allows one to define multiple certificates, issuers and clusterissuers. Note that it maintains backwards compatibility.

• fixes a job issue where SA is not defined and so task cannot complete, this is particularly harmful for upgrades
• removes job that contains a sleep, that isnt necessary.

tested out the chart as mentioned below

[joejulian]

Worked for me, setting up LE and LE-staging issuers on my home cluster.

[jimmidyson]

This is a breaking change to the chart though right? Do we want to handle upgrade without need to change values on upgrade, as in still support clusterissuer if set and clusterissuers is unset?

[alejandroEsc]

This is a breaking change to the chart though right? Do we want to handle upgrade without need to change values on upgrade, as in still support clusterissuer if set and clusterissuers is unset?

this is absolutely a breaking change. we have upgrade strategies in place, and we can mitigate things as you mention here. But still trying to achieve the right set of features we want from this.

[joejulian]

To rephrase what I think I hear Jimmi suggesting:

If you retain the old clusterissuer template and value, but also add the clusterissuers template (as you have), it won't be breaking but will still support the new functionality. Perhaps add something to NOTES.txt warning of a deprecation (if we're deprecating clusterissuer).

[alejandroEsc]

To rephrase what I think I hear Jimmi suggesting:

If you retain the old clusterissuer template and value, but also add the clusterissuers template (as you have), it won't be breaking but will still support the new functionality. Perhaps add something to NOTES.txt warning of a deprecation (if we're deprecating clusterissuer).

yes, i agree. What i am saying is i am still figuring out what is missing. As you mention, we can handle the breaking change in this way you suggest.

[GoelDeepak]

Good work!! lgtm

[joejulian]

With a previous install of 0.1.5 with the values of:

clusterissuer:
  name: kubernetes-ca
  spec:
    ca:
      secretName: kubernetes-intermediate-ca

and an upgrade to this version with the values:

---
clusterissuers:
  - name: kubernetes-ca
    spec:
      ca:
        secretName: kubernetes-intermediate-ca
  - name: letsencrypt
    spec:
      acme:
        email: me@joejulian.name
        privateKeySecretRef:
          name: acme-issuer-account-key
        server: https://acme-v02.api.letsencrypt.org/directory
        solvers:
          - http01:
              ingress:
                class: traefik
  - name: letsencrypt-staging
    spec:
      acme:
        email: me@joejulian.name
        privateKeySecretRef:
          name: acme-staging-issuer-account-key
        server: https://acme-staging-v02.api.letsencrypt.org/directory
        solvers:
          - http01:
              ingress:
                class: traefik
installCertManager: true

Job name: cert-manager-kubeaddons-cert-manager-setup

Error from server (Forbidden): apiservices.apiregistration.k8s.io "v1beta1.webhook.certmanager.k8s.io" is forbidden: User "system:serviceaccount:cert-manager:default" cannot get resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

[alejandroEsc]

With a previous install of 0.1.5 with the values of:

clusterissuer:
  name: kubernetes-ca
  spec:
    ca:
      secretName: kubernetes-intermediate-ca

and an upgrade to this version with the values:

---
clusterissuers:
  - name: kubernetes-ca
    spec:
      ca:
        secretName: kubernetes-intermediate-ca
  - name: letsencrypt
    spec:
      acme:
        email: me@joejulian.name
        privateKeySecretRef:
          name: acme-issuer-account-key
        server: https://acme-v02.api.letsencrypt.org/directory
        solvers:
          - http01:
              ingress:
                class: traefik
  - name: letsencrypt-staging
    spec:
      acme:
        email: me@joejulian.name
        privateKeySecretRef:
          name: acme-staging-issuer-account-key
        server: https://acme-staging-v02.api.letsencrypt.org/directory
        solvers:
          - http01:
              ingress:
                class: traefik
installCertManager: true

Job name: cert-manager-kubeaddons-cert-manager-setup

Error from server (Forbidden): apiservices.apiregistration.k8s.io "v1beta1.webhook.certmanager.k8s.io" is forbidden: User "system:serviceaccount:cert-manager:default" cannot get resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope

That may in fact be an issue with the chart itself. Would have to investigate a bit, also did you create the issuers and certificates that you needed? I would like to get a copy of your values chart and test out the upgrade. Thanks for checking in!

[alejandroEsc]

So after some investigation, i got things to work out, turned out that there was no pointer to the SA for the job to follow. Regardless, here is the output of an upgrade:

A new konvoy cluster was created using master kubernetes-base-addons

I used the following values file

issuers:
  - name: kubernetes-root-issuer
    secretName: kubernetes-root-ca

certificates:
  - name: kubernetes-intermediate-ca
    secretName: kubernetes-intermediate-ca
    issuerRef:
      name: kubernetes-root-issuer
      kind: Issuer
      # These are the default usages for reference
      usages:
        - "digital signature"
        - "key encipherment"
    commonName: cert-manager
    duration: 87600h
    dnsNames: []

clusterissuers:
  - name: kubernetes-ca
    spec:
      ca:
        secretName: kubernetes-intermediate-ca

# DEPRECATED, please use the above issuers, certificates and clusterissuers
clusterissuer: {}

then i ran the upgrade commands

$ helm upgrade --install cert-manager-kubeaddons --values myvalues.yaml --namespace cert-manager ./
Release "cert-manager-kubeaddons" has been upgraded.
LAST DEPLOYED: Thu Jan 30 12:38:12 2020
NAMESPACE: cert-manager
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                                               AGE
cert-manager-kubeaddons-edit                       4m57s
cert-manager-kubeaddons-view                       4m57s
cert-manager-kubeaddons-webhook:webhook-requester  4m57s

==> v1/Deployment
NAME                                AGE
cert-manager-kubeaddons             4m57s
cert-manager-kubeaddons-cainjector  4m57s
cert-manager-kubeaddons-webhook     4m57s

==> v1/Pod(related)
NAME                                                 AGE
cert-manager-kubeaddons-7d7f98fbc6-c94g5             4m57s
cert-manager-kubeaddons-cainjector-6dcd94769b-mnhqc  4m57s
cert-manager-kubeaddons-webhook-77fbc6d59b-9t5kd     4m57s

==> v1/Service
NAME                             AGE
cert-manager-kubeaddons          4m57s
cert-manager-kubeaddons-webhook  4m57s

==> v1/ServiceAccount
NAME                                AGE
cert-manager-kubeaddons             4m57s
cert-manager-kubeaddons-cainjector  4m57s
cert-manager-kubeaddons-webhook     4m57s

==> v1beta1/APIService
NAME                                AGE
v1beta1.webhook.certmanager.k8s.io  4m57s

==> v1beta1/ClusterRole
NAME                                               AGE
cert-manager-kubeaddons-cainjector                 4m57s
cert-manager-kubeaddons-controller-certificates    4m57s
cert-manager-kubeaddons-controller-challenges      4m57s
cert-manager-kubeaddons-controller-clusterissuers  4m57s
cert-manager-kubeaddons-controller-ingress-shim    4m57s
cert-manager-kubeaddons-controller-issuers         4m57s
cert-manager-kubeaddons-controller-orders          4m57s
cert-manager-kubeaddons-leaderelection             4m57s

==> v1beta1/ClusterRoleBinding
NAME                                               AGE
cert-manager-kubeaddons-cainjector                 4m57s
cert-manager-kubeaddons-controller-certificates    4m57s
cert-manager-kubeaddons-controller-challenges      4m57s
cert-manager-kubeaddons-controller-clusterissuers  4m57s
cert-manager-kubeaddons-controller-ingress-shim    4m57s
cert-manager-kubeaddons-controller-issuers         4m57s
cert-manager-kubeaddons-controller-orders          4m57s
cert-manager-kubeaddons-leaderelection             4m57s
cert-manager-kubeaddons-webhook:auth-delegator     4m57s

==> v1beta1/MutatingWebhookConfiguration
NAME                             AGE
cert-manager-kubeaddons-webhook  4m57s

==> v1beta1/RoleBinding
NAME                                                           AGE
cert-manager-kubeaddons-webhook:webhook-authentication-reader  4m57s

==> v1beta1/ValidatingWebhookConfiguration
NAME                             AGE
cert-manager-kubeaddons-webhook  4m57s


NOTES:
Deploys cert-manager and submits the kubernetes-ca ClusterIssuer post installation.

with helm list

$ helm list
NAME                           	REVISION	UPDATED                 	STATUS  	CHART                      	APP VERSION	NAMESPACE
cert-manager-kubeaddons        	2       	Thu Jan 30 12:38:12 2020	DEPLOYED	cert-manager-setup-0.1.8   	0.10.1     	cert-manager

The upgrade process succeeded and did not get any of the issues you saw, the issuers, certificatets, and clusterissuers were as expected. I was able to log into the cluster with no perceivable issues.

[joejulian]

Thanks for fixing the other stuff this exposed.