Kubernetes Change Tracking: Add reloader to roll deployments after env or secret changes #6

obriensystems · 2024-12-11T17:41:17Z

Options for deployment rollout on config change

obriensystems · 2024-12-12T16:38:45Z

clone/fork reloader

 git clone git@github.com:stakater/Reloader.git

follow
https://github.com/stakater/Reloader/blob/master/README.md#vanilla-manifests

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl apply -f reloader.yaml 
serviceaccount/reloader-reloader created
clusterrole.rbac.authorization.k8s.io/reloader-reloader-role created
clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding created
deployment.apps/reloader-reloader created

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-cbc7c88b9-b56lq             1/1     Running   0          46m
reloader-reloader-5d85b9bbd9-pfdng   1/1     Running   0          3m12s

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get deployment                        
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
helloweb            1/1     1            1           51m
reloader-reloader   1/1     1            1           8m14s

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get clusterrole | grep reloader
reloader-reloader-role                                                 2024-12-12T16:38:15Z

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get clusterrolebinding | grep reloader
reloader-reloader-role-binding                                  ClusterRole/reloader-reloader-role                                          7m51s

test tracking of env vairables via reloader

https://github.com/stakater/Reloader/blob/master/README.md#how-to-use-reloader


diff --git a/src/kubernetes/eks/deployment.sh b/src/kubernetes/eks/deployment.sh
index 7b8aae9..d22c46a 100755
--- a/src/kubernetes/eks/deployment.sh
+++ b/src/kubernetes/eks/deployment.sh
@@ -147,5 +147,5 @@ if [[ -z $UNIQUE ]]; then
 fi
 
 #deployment "$CREATE_PROJ" "$DELETE_PROJ" "$PROVISION_PROJ" "$BOOT_PROJECT_ID" "$STREAM_PROJECT_ID"
-deployApps
+deployReloader
 printf "**** Done ****\n"
\ No newline at end of file
diff --git a/src/kubernetes/eks/deployment.yaml b/src/kubernetes/eks/deployment.yaml
index b589c6f..85ca077 100644
--- a/src/kubernetes/eks/deployment.yaml
+++ b/src/kubernetes/eks/deployment.yaml
@@ -5,6 +5,9 @@ metadata:
   name: helloweb
   labels:
     app: hello
+  # reloader
+  #annotations:
+  #  reloader.stakater.com/auto: "true"
 spec:
   selector:
     matchLabels:
@@ -27,6 +30,8 @@ spec:
             mountPath: /etc/tls
             readOnly: true
         env:
+          - name: dummy-env-name
+            value: dummy-env-value
           - name: TLS_CERT
             value: /etc/tls/tls.crt
           - name: TLS_KEY
(venv-metal) michaelobrien@Michaels-MBP eks % 

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl apply -f deployment.yaml 
deployment.apps/helloweb configured
(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-d6548f95b-m7l5b             1/1     Running   0          11s
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          2m40s
(venv-metal) michaelobrien@Michaels-MBP eks % kubectl describe pod helloweb-d6548f95b-m7l5b
Name:             helloweb-d6548f95b-m7l5b

    Environment:
      dummy-env-name:  dummy-env-value
      TLS_CERT:        /etc/tls/tls.crt
      TLS_KEY:         /etc/tls/tls.key



enable reloader tracking

(venv-metal) michaelobrien@Michaels-MBP eks % git diff                                       
diff --git a/src/kubernetes/eks/deployment.yaml b/src/kubernetes/eks/deployment.yaml
index 85ca077..330ef10 100644
--- a/src/kubernetes/eks/deployment.yaml
+++ b/src/kubernetes/eks/deployment.yaml
@@ -6,8 +6,8 @@ metadata:
   labels:
     app: hello
   # reloader
-  #annotations:
-  #  reloader.stakater.com/auto: "true"
+  annotations:
+    reloader.stakater.com/auto: "true"
 spec:
   selector:
     matchLabels:


(venv-metal) michaelobrien@Michaels-MBP eks % kubectl apply -f deployment.yaml
deployment.apps/helloweb configured

obriensystems · 2024-12-12T17:18:02Z

Simulating change to force a deployment redeploy

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl describe pod  reloader-reloader-5d85b9bbd9-sqmkc 
Name:             reloader-reloader-5d85b9bbd9-sqmkc
Namespace:        default
...
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  13m   default-scheduler  Successfully assigned default/reloader-reloader-5d85b9bbd9-sqmkc to ip-172-31-80-165.ec2.internal
  Normal  Pulled     13m   kubelet            Container image "ghcr.io/stakater/reloader:latest" already present on machine
  Normal  Created    13m   kubelet            Created container reloader-reloader
  Normal  Started    13m   kubelet            Started container reloader-reloader
(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get nodes
NAME                            STATUS   ROLES    AGE   VERSION
ip-172-31-42-248.ec2.internal   Ready    <none>   48d   v1.30.4-eks-a737599
ip-172-31-80-165.ec2.internal   Ready    <none>   48d   v1.30.4-eks-a737599


bounce the certificate
(venv-metal) michaelobrien@Michaels-MBP eks % kubectl delete -f clusterissuer-selfsigned.yaml               
clusterissuer.cert-manager.io "selfsigned" deleted
(venv-metal) michaelobrien@Michaels-MBP eks % kubectl create -f clusterissuer-selfsigned.yaml
clusterissuer.cert-manager.io/selfsigned created

no rotation yet
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  19m   default-scheduler  Successfully assigned default/helloweb-d6548f95b-m7l5b to ip-172-31-42-248.ec2.internal
  Normal  Pulling    19m   kubelet            Pulling image "us-docker.pkg.dev/google-samples/containers/gke/hello-app-tls:1.0"
  Normal  Pulled     19m   kubelet            Successfully pulled image "us-docker.pkg.dev/google-samples/containers/gke/hello-app-tls:1.0" in 418ms (418ms including waiting). Image size: 14359445 bytes.
  Normal  Created    19m   kubelet            Created container hello-app
  Normal  Started    19m   kubelet            Started container hello-app
(venv-metal) michaelobrien@Michaels-MBP eks % kubectl describe pod  helloweb-d6548f95b-m7l5b



got it - reloader is on one node - app is on the other - I need to expand the ResplicaSet

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  53m   default-scheduler  Successfully assigned default/reloader-reloader-5d85b9bbd9-sqmkc to ip-172-31-80-165.ec2.internal
  Normal  Pulled     53m   kubelet            Container image "ghcr.io/stakater/reloader:latest" already present on machine
  Normal  Created    53m   kubelet            Created container reloader-reloader
  Normal  Started    53m   kubelet            Started container reloader-reloader
(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get pods --all-namespaces -o wide               
NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE     IP              NODE                            NOMINATED NODE   READINESS GATES
cert-manager   cert-manager-b6fd485d9-cp9pg              1/1     Running   0          6d23h   172.31.95.107   ip-172-31-80-165.ec2.internal   <none>           <none>
cert-manager   cert-manager-cainjector-dcc5966bc-jgrxp   1/1     Running   0          6d23h   172.31.34.155   ip-172-31-42-248.ec2.internal   <none>           <none>
cert-manager   cert-manager-webhook-dfb76c7bd-hwq8d      1/1     Running   0          6d23h   172.31.32.165   ip-172-31-42-248.ec2.internal   <none>           <none>
default        helloweb-6b6c7545db-dsx5z                 1/1     Running   0          83s     172.31.34.56    ip-172-31-42-248.ec2.internal   <none>           <none>
default        reloader-reloader-5d85b9bbd9-sqmkc        1/1     Running   0          53m     172.31.95.132   ip-172-31-80-165.ec2.internal   <none>           <none>

(venv-metal) michaelobrien@Michaels-MBP eks % git diff                                 
diff --git a/src/kubernetes/eks/deployment.yaml b/src/kubernetes/eks/deployment.yaml
index 85ca077..b411702 100644
--- a/src/kubernetes/eks/deployment.yaml
+++ b/src/kubernetes/eks/deployment.yaml
@@ -6,8 +6,8 @@ metadata:
   labels:
     app: hello
   # reloader
-  #annotations:
-  #  reloader.stakater.com/auto: "true"
+  annotations:
+    reloader.stakater.com/auto: "true"
 spec:
   selector:
     matchLabels:
@@ -31,7 +31,7 @@ spec:
             readOnly: true
         env:
           - name: dummy-env-name
-            value: dummy-env-value
+            value: dummy-env-value2
           - name: TLS_CERT
             value: /etc/tls/tls.crt
           - name: TLS_KEY
diff --git a/src/kubernetes/eks/reloader.yaml b/src/kubernetes/eks/reloader.yaml
index 986cdb4..5f1c5c3 100644
--- a/src/kubernetes/eks/reloader.yaml
+++ b/src/kubernetes/eks/reloader.yaml
@@ -81,7 +81,7 @@ metadata:
   name: reloader-reloader
   namespace: default
 spec:
-  replicas: 1
+  replicas: 2
   revisionHistoryLimit: 2
   selector:
     matchLabels:

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl apply -f reloader.yaml  
serviceaccount/reloader-reloader unchanged
clusterrole.rbac.authorization.k8s.io/reloader-reloader-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding unchanged
deployment.apps/reloader-reloader configured

(venv-metal) michaelobrien@Michaels-MBP eks % kubectl get pods --all-namespaces -o wide
NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE     IP              NODE                            NOMINATED NODE   READINESS GATES
cert-manager   cert-manager-b6fd485d9-cp9pg              1/1     Running   0          6d23h   172.31.95.107   ip-172-31-80-165.ec2.internal   <none>           <none>
cert-manager   cert-manager-cainjector-dcc5966bc-jgrxp   1/1     Running   0          6d23h   172.31.34.155   ip-172-31-42-248.ec2.internal   <none>           <none>
cert-manager   cert-manager-webhook-dfb76c7bd-hwq8d      1/1     Running   0          6d23h   172.31.32.165   ip-172-31-42-248.ec2.internal   <none>           <none>
default        helloweb-6b6c7545db-dsx5z                 1/1     Running   0          4m31s   172.31.34.56    ip-172-31-42-248.ec2.internal   <none>           <none>
default        reloader-reloader-5d85b9bbd9-447sg        1/1     Running   0          24s     172.31.45.140   ip-172-31-42-248.ec2.internal   <none>           <none>
default        reloader-reloader-5d85b9bbd9-sqmkc        1/1     Running   0          56m     172.31.95.132   ip-172-31-80-165.ec2.internal   <none>           <none>

obriensystems · 2024-12-16T17:02:14Z

2 - Simulating change to force a deployment redeploy - delete the certificate to force a regenerate

michaelobrien@mbp8 eks % kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-6b6c7545db-dsx5z            1/1     Running   0          3d21h
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          3d21h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          3d22h
michaelobrien@mbp8 eks % kubectl get certificates
NAME   READY   SECRET    AGE
www    True    www-tls   3d22h
michaelobrien@mbp8 eks % kubectl get certificates --all-namespaces
NAMESPACE   NAME   READY   SECRET    AGE
default     www    True    www-tls   3d22h
michaelobrien@mbp8 eks % kubectl get pods --all-namespaces        
NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE
cert-manager   cert-manager-b6fd485d9-cp9pg              1/1     Running   0          10d
cert-manager   cert-manager-cainjector-dcc5966bc-jgrxp   1/1     Running   0          10d
cert-manager   cert-manager-webhook-dfb76c7bd-hwq8d      1/1     Running   0          10d
default        helloweb-6b6c7545db-dsx5z                 1/1     Running   0          3d22h
default        reloader-reloader-5d85b9bbd9-447sg        1/1     Running   0          3d22h
default        reloader-reloader-5d85b9bbd9-sqmkc        1/1     Running   0          3d23h
kube-system    aws-node-8bwzk                            2/2     Running   0          52d
kube-system    aws-node-f9mz4                            2/2     Running   0          52d
kube-system    coredns-586b798467-fdvwr                  1/1     Running   0          63d
kube-system    coredns-586b798467-gff7r                  1/1     Running   0          63d
kube-system    eks-pod-identity-agent-cq8nn              1/1     Running   0          52d
kube-system    eks-pod-identity-agent-g4wxv              1/1     Running   0          52d
kube-system    kube-proxy-b8v4z                          1/1     Running   0          52d
kube-system    kube-proxy-h2ttb                          1/1     Running   0          52d
michaelobrien@mbp8 eks % kubectl get deployments --all-namespaces
NAMESPACE      NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager   cert-manager              1/1     1            1           10d
cert-manager   cert-manager-cainjector   1/1     1            1           10d
cert-manager   cert-manager-webhook      1/1     1            1           10d
default        helloweb                  1/1     1            1           4d
default        reloader-reloader         2/2     2            2           3d23h
kube-system    coredns                   2/2     2            2           63d
michaelobrien@mbp8 eks % kubectl get services --all-namespaces
NAMESPACE      NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                  AGE
cert-manager   cert-manager                ClusterIP      10.100.175.92    <none>                                                                   9402/TCP                 10d
cert-manager   cert-manager-cainjector     ClusterIP      10.100.130.103   <none>                                                                   9402/TCP                 10d
cert-manager   cert-manager-webhook        ClusterIP      10.100.20.33     <none>                                                                   443/TCP,9402/TCP         10d
default        helloweb                    LoadBalancer   10.100.44.187    ae19feae055414514b7f4536facfb51a-629444322.us-east-1.elb.amazonaws.com   443:30474/TCP            4d
default        kubernetes                  ClusterIP      10.100.0.1       <none>                                                                   443/TCP                  63d
kube-system    eks-extension-metrics-api   ClusterIP      10.100.127.149   <none>                                                                   443/TCP                  22d
kube-system    kube-dns                    ClusterIP      10.100.0.10      <none>                                                                   53/UDP,53/TCP,9153/TCP   63d
michaelobrien@mbp8 eks % echo $DOMAIN_NAME

michaelobrien@mbp8 eks % export AWS_DEFAULT_OUTPUT=json
michaelobrien@mbp8 eks % export AWS_DEFAULT_REGION=us-east-1
michaelobrien@mbp8 eks % export DOMAIN_NAME=eventfield.io
michaelobrien@mbp8 eks % HOSTED_ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name $DOMAIN_NAME --query "HostedZones[0].Id" --output text)
michaelobrien@mbp8 eks % echo $DOMAIN_NAME                                                                                                        
eventfield.io
michaelobrien@mbp8 eks % echo $HOSTED_ZONE_ID
/hostedzone/Z1005

michaelobrien@mbp8 eks % kubectl get secrets --all-namespaces
NAMESPACE      NAME                                 TYPE                 DATA   AGE
cert-manager   cert-manager-webhook-ca              Opaque               3      18d
cert-manager   sh.helm.release.v1.cert-manager.v1   helm.sh/release.v1   1      10d
default        www-tls                              kubernetes.io/tls    3      10d

verify current dates
michaelobrien@mbp8 eks % cmctl inspect secret www-tls | grep 2024
	Not Before: Thu, 05 Dec 2024 19:19:16 UTC
michaelobrien@mbp8 eks % cmctl inspect secret www-tls | grep 2025
	Not After: Wed, 05 Mar 2025 19:19:16 UTC


check dns and elb
michaelobrien@mbp8 eks % curl --insecure -v https://www.$DOMAIN_NAME 
* Could not resolve host: www.eventfield.io
* Closing connection
curl: (6) Could not resolve host: www.eventfield.io
michaelobrien@mbp8 eks % dig www.$DOMAIN_NAME A

; <<>> DiG 9.10.6 <<>> www.eventfield.io A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39875
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.eventfield.io.		IN	A

;; Query time: 1 msec
;; SERVER: 2607:fea8:e25d:7b60:688f:2eff:fe18:cd81#53(2607:fea8:e25d:7b60:688f:2eff:fe18:cd81)
;; WHEN: Mon Dec 16 11:43:43 EST 2024
;; MSG SIZE  rcvd: 46




DNS record not found because ELB was refreshed on the 12th
ae19feae055414514b7f4536facfb51a-629444322.us-east-1.elb.amazonaws.com
from
a14e2cd253b504a59ba28c5dd984808c-1311392204.us-east-1.elb.amazonaws.com.
in the zone

michaelobrien@mbp8 eks % dig www.$DOMAIN_NAME A                     

; <<>> DiG 9.10.6 <<>> www.eventfield.io A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33509
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1410
;; QUESTION SECTION:
;www.eventfield.io.		IN	A

;; ANSWER SECTION:
www.eventfield.io.	60	IN	A	52.3.127.204
www.eventfield.io.	60	IN	A	18.235.183.185

;; Query time: 78 msec
;; SERVER: 2607:fea8:e25d:7b60:688f:2eff:fe18:cd81#53(2607:fea8:e25d:7b60:688f:2eff:fe18:cd81)
;; WHEN: Mon Dec 16 11:50:35 EST 2024
;; MSG SIZE  rcvd: 78

flushing cache - no effect
michaelobrien@mbp8 eks % sudo killall -HUP mDSNResponder

delete secret

michaelobrien@mbp8 eks % kubectl get secrets --all-namespaces              
NAMESPACE      NAME                                 TYPE                 DATA   AGE
cert-manager   cert-manager-webhook-ca              Opaque               3      18d
cert-manager   sh.helm.release.v1.cert-manager.v1   helm.sh/release.v1   1      10d
default        www-tls                              kubernetes.io/tls    3      10s
michaelobrien@mbp8 eks % cmctl inspect secret www-tls | grep 2024          
	Not Before: Mon, 16 Dec 2024 16:54:11 UTC
michaelobrien@mbp8 eks % cmctl inspect secret www-tls | grep 2025
	Not After: Sun, 16 Mar 2025 16:54:11 UTC


check deployment
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-6b6c7545db-dsx5z            1/1     Running   0          3d23h
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          3d23h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d

obriensystems · 2024-12-16T17:22:26Z

deleted/recreated deployment/service

michaelobrien@mbp8 eks % kubectl apply -f deployment.yaml
deployment.apps/helloweb-tls created
michaelobrien@mbp8 eks % kubectl apply -f service.yaml   
service/helloweb-tls created
michaelobrien@mbp8 eks % kubectl get services --all-namespaces        
NAMESPACE      NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                  AGE
cert-manager   cert-manager                ClusterIP      10.100.175.92    <none>                                                                   9402/TCP                 10d
cert-manager   cert-manager-cainjector     ClusterIP      10.100.130.103   <none>                                                                   9402/TCP                 10d
cert-manager   cert-manager-webhook        ClusterIP      10.100.20.33     <none>                                                                   443/TCP,9402/TCP         10d
default        helloweb-tls                LoadBalancer   10.100.57.93     a89a0ecda2413499eb2f341448e31acf-950405761.us-east-1.elb.amazonaws.com   443:31804/TCP            13s
default        kubernetes                  ClusterIP      10.100.0.1       <none>                                                                   443/TCP                  63d
kube-system    eks-extension-metrics-api   ClusterIP      10.100.127.149   <none>                                                                   443/TCP                  22d
kube-system    kube-dns                    ClusterIP      10.100.0.10      <none>                                                                   53/UDP,53/TCP,9153/TCP   63d


delete secret
michaelobrien@mbp8 eks % kubectl delete secret www-tls -n default
secret "www-tls" deleted

increase replicas
michaelobrien@mbp8 eks % kubectl apply -f deployment.yaml 
deployment.apps/helloweb-tls configured
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-5789f85f96-9v7sm        1/1     Running   0          2m53s
helloweb-tls-5789f85f96-gvlj5        1/1     Running   0          7s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          3d23h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d

michaelobrien@mbp8 eks % kubectl delete secret www-tls -n default
secret "www-tls" deleted
michaelobrien@mbp8 eks % kubectl get secrets                    
NAME      TYPE                DATA   AGE
www-tls   kubernetes.io/tls   3      9s
michaelobrien@mbp8 eks % kubectl get pods                        
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-5789f85f96-9v7sm        1/1     Running   0          10m
helloweb-tls-5789f85f96-gvlj5        1/1     Running   0          8m13s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          3d23h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d

obriensystems · 2024-12-16T17:33:40Z

Try certificate modification - dns

michaelobrien@mbp8 eks % kubectl get pods                        
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-5789f85f96-9v7sm        1/1     Running   0          10m
helloweb-tls-5789f85f96-gvlj5        1/1     Running   0          8m13s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          3d23h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d

michaelobrien@mbp8 eks % git diff            
diff --git a/src/kubernetes/eks/certificate.yaml b/src/kubernetes/eks/certificate.yaml
index 07e9bcf..3ce8d67 100644
--- a/src/kubernetes/eks/certificate.yaml
+++ b/src/kubernetes/eks/certificate.yaml
@@ -10,7 +10,7 @@ spec:
     rotationPolicy: Always
   commonName: www.$DOMAIN_NAME
   dnsNames:
-    - www.$DOMAIN_NAME
+    - www2.$DOMAIN_NAME
   usages:
     - digital signature
     - key encipherment
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml 
certificate.cert-manager.io/www configured

yes - change kicked in
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-78d46588c9-gr4sn        1/1     Running   0          26s
helloweb-tls-78d46588c9-qx6k2        1/1     Running   0          24s

obriensystems · 2024-12-16T23:35:24Z

attempt to verify that the reloader is picking up the www to www2 dns change

change www2 back to www - picked up change

michaelobrien@mbp8 eks % git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
michaelobrien@mbp8 eks % kubectl get pods      
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-78d46588c9-gr4sn        1/1     Running   0          5h56m
helloweb-tls-78d46588c9-qx6k2        1/1     Running   0          5h56m
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml                      
certificate.cert-manager.io/www configured
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS              RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        0/1     ContainerCreating   0          2s
helloweb-tls-566f7c5466-r969f        1/1     Running             0          4s
helloweb-tls-78d46588c9-qx6k2        1/1     Running             0          5h56m
reloader-reloader-5d85b9bbd9-447sg   1/1     Running             0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running             0          4d6h
michaelobrien@mbp8 eks % kubectl get pods 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          24s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          26s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h


turn off reloader annotation - no change
michaelobrien@mbp8 eks % git diff                         
diff --git a/src/kubernetes/eks/deployment.yaml b/src/kubernetes/eks/deployment.yaml
index 90d3116..f261cf3 100644
--- a/src/kubernetes/eks/deployment.yaml
+++ b/src/kubernetes/eks/deployment.yaml
@@ -6,8 +6,8 @@ metadata:
   labels:
     app: helloweb-tls
   # reloader
-  annotations:
-    reloader.stakater.com/auto: "true"
+  #annotations:
+  #  reloader.stakater.com/auto: "true"
 spec:
   replicas: 2
   selector:
michaelobrien@mbp8 eks % kubectl apply -f deployment.yaml 
deployment.apps/helloweb-tls configured
michaelobrien@mbp8 eks % kubectl get pods                
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          51s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          53s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % git diff                         
diff --git a/src/kubernetes/eks/certificate.yaml b/src/kubernetes/eks/certificate.yaml
index 07e9bcf..3ce8d67 100644
--- a/src/kubernetes/eks/certificate.yaml
+++ b/src/kubernetes/eks/certificate.yaml
@@ -10,7 +10,7 @@ spec:
     rotationPolicy: Always
   commonName: www.$DOMAIN_NAME
   dnsNames:
-    - www.$DOMAIN_NAME
+    - www2.$DOMAIN_NAME
   usages:
     - digital signature
     - key encipherment
diff --git a/src/kubernetes/eks/deployment.yaml b/src/kubernetes/eks/deployment.yaml
index 90d3116..f261cf3 100644
--- a/src/kubernetes/eks/deployment.yaml
+++ b/src/kubernetes/eks/deployment.yaml
@@ -6,8 +6,8 @@ metadata:
   labels:
     app: helloweb-tls
   # reloader
-  annotations:
-    reloader.stakater.com/auto: "true"
+  #annotations:
+  #  reloader.stakater.com/auto: "true"
 spec:
   replicas: 2
   selector:
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www configured
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          86s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          88s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % kubectl get pods 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          89s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          91s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % kubectl get pods 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          92s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          94s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % git diff                         
diff --git a/src/kubernetes/eks/certificate.yaml b/src/kubernetes/eks/certificate.yaml
index 07e9bcf..3ce8d67 100644
--- a/src/kubernetes/eks/certificate.yaml
+++ b/src/kubernetes/eks/certificate.yaml
@@ -10,7 +10,7 @@ spec:
     rotationPolicy: Always
   commonName: www.$DOMAIN_NAME
   dnsNames:
-    - www.$DOMAIN_NAME
+    - www2.$DOMAIN_NAME
   usages:
     - digital signature
     - key encipherment
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          2m21s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          2m23s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % git diff         
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www configured
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          2m58s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          3m
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % kubectl get pods 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          3m2s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          3m4s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % git diff                         
diff --git a/src/kubernetes/eks/certificate.yaml b/src/kubernetes/eks/certificate.yaml
index 07e9bcf..3ce8d67 100644
--- a/src/kubernetes/eks/certificate.yaml
+++ b/src/kubernetes/eks/certificate.yaml
@@ -10,7 +10,7 @@ spec:
     rotationPolicy: Always
   commonName: www.$DOMAIN_NAME
   dnsNames:
-    - www.$DOMAIN_NAME
+    - www2.$DOMAIN_NAME
   usages:
     - digital signature
     - key encipherment


turned reloader back on - should have picked up the change

michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www configured
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          3m33s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          3m35s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % kubectl get pods 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          3m35s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          3m37s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h

should have picked it up

reapplied deployment again - now ok

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloweb-tls
  labels:
    app: helloweb-tls
  # reloader
  annotations:
    reloader.stakater.com/auto: "true"

www3 to www4
michaelobrien@mbp8 eks % git diff
diff --git a/src/kubernetes/eks/certificate.yaml b/src/kubernetes/eks/certificate.yaml
index 07e9bcf..90d5fb4 100644
--- a/src/kubernetes/eks/certificate.yaml
+++ b/src/kubernetes/eks/certificate.yaml
@@ -10,7 +10,7 @@ spec:
     rotationPolicy: Always
   commonName: www.$DOMAIN_NAME
   dnsNames:
-    - www.$DOMAIN_NAME
+    - www4.$DOMAIN_NAME


michaelobrien@mbp8 eks % kubectl apply -f deployment.yaml 
deployment.apps/helloweb-tls configured
michaelobrien@mbp8 eks % kubectl get pods                
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-566f7c5466-fdbfb        1/1     Running   0          8m43s
helloweb-tls-566f7c5466-r969f        1/1     Running   0          8m45s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www configured
michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-6ffdfccffc-b5zm2        1/1     Running   0          4s
helloweb-tls-6ffdfccffc-fxv27        1/1     Running   0          3s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d5h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d6h

should have checked logs
michaelobrien@mbp8 eks % kubectl logs -f reloader-reloader-5d85b9bbd9-447sg
time="2024-12-12T17:51:18Z" level=info msg="Starting Controller to watch resource type: secrets"
time="2024-12-16T17:31:29Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"
time="2024-12-16T23:28:11Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"
time="2024-12-16T23:37:18Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"

try one more time - turn off reloader on deployment, attempt multiple changes on the certificate dns name

obriensystems · 2024-12-17T16:09:17Z

Use case: variant: warning on deployment roll - rolls anyway on the other replica

     rotationPolicy: Always
   commonName: www.$DOMAIN_NAME
   dnsNames:
-    - www.$DOMAIN_NAME
+    - www2.$DOMAIN_NAME

michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml

michaelobrien@mbp8 src % kubectl logs -f reloader-reloader-5d85b9bbd9-sqmkc
time="2024-12-17T16:06:10Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"

michaelobrien@mbp8 src % kubectl logs -f reloader-reloader-5d85b9bbd9-447sg
time="2024-12-17T16:06:10Z" level=error msg="Update for 'helloweb-tls' of type 'Deployment' in namespace 'default' failed with error Operation cannot be fulfilled on deployments.apps \"helloweb-tls\": the object has been modified; please apply your changes to the latest version and try again"
time="2024-12-17T16:06:10Z" level=error msg="Rolling upgrade for 'www-tls' failed with error = Operation cannot be fulfilled on deployments.apps \"helloweb-tls\": the object has been modified; please apply your changes to the latest version and try again"
time="2024-12-17T16:06:10Z" level=error msg="Error syncing events: Operation cannot be fulfilled on deployments.apps \"helloweb-tls\": the object has been modified; please apply your changes to the latest version and try again"

michaelobrien@mbp8 eks % kubectl get pods                 
NAME                                 READY   STATUS    RESTARTS   AGE
helloweb-tls-cb74dd48-6q8g6          1/1     Running   0          23s
helloweb-tls-cb74dd48-z5znt          1/1     Running   0          22s
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d22h
reloader-reloader-5d85b9bbd9-sqmkc   1/1     Running   0          4d23h

scale down to 1 replica both the app and the reloader - we are ok on a cert change

michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www configured
michaelobrien@mbp8 eks % kubectl get pods -o wide         
NAME                                 READY   STATUS              RESTARTS   AGE     IP              NODE                            NOMINATED NODE   READINESS GATES
helloweb-tls-b764889c-mrdr7          0/1     ContainerCreating   0          1s      <none>          ip-172-31-42-248.ec2.internal   <none>           <none>
helloweb-tls-b99d8fff8-rq4h7         1/1     Running             0          47s     172.31.84.250   ip-172-31-80-165.ec2.internal   <none>           <none>
reloader-reloader-5d85b9bbd9-447sg   1/1     Running             0          4d22h   172.31.45.140   ip-172-31-42-248.ec2.internal   <none>           <none>
michaelobrien@mbp8 eks % kubectl get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE                            NOMINATED NODE   READINESS GATES
helloweb-tls-b764889c-mrdr7          1/1     Running   0          11s     172.31.34.56    ip-172-31-42-248.ec2.internal   <none>           <none>
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d22h   172.31.45.140   ip-172-31-42-248.ec2.internal   <none>           <none>

testing cert delete - not working

kind: Deployment
metadata:
  name: helloweb-tls
  labels:
    app: helloweb-tls
  # reloader
  annotations:
    reloader.stakater.com/auto: "true"

michaelobrien@mbp8 eks % kubectl delete secret www-tls -n default
secret "www-tls" deleted
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml       
certificate.cert-manager.io/www unchanged
michaelobrien@mbp8 eks % kubectl get pods -o wide                
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE                            NOMINATED NODE   READINESS GATES
helloweb-tls-5d7dc89df7-c7vlq        1/1     Running   0          37s     172.31.84.250   ip-172-31-80-165.ec2.internal   <none>           <none>
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d22h   172.31.45.140   ip-172-31-42-248.ec2.internal   <none>           <none>

checked namespace - ok

checked rbac - clusterrole - ok

checking that cert contexts changed

michaelobrien@mbp8 eks % kubectl get secret www-tls -o yaml      
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFakNDQWZxZ0F3SUJBZ0lRRDYzbnNkZ2Y1YUI0MjN2QndaTExZVEFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWUVFEREJCM2QzY3VKRVJQVFVGSlRsOU9RVTFGTUI0WERUSTBNVEl4TnpFMk16TXpNbG9YRFRJMQpNRE14TnpFMk16TXpNbG93R3pFWk1CY0dBMVVFQXd3UWQzZDNMaVJFVDAxQlNVNWZUa0ZOUlRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU0rRmIxV1V3YjEwVE85UjFSOGhGREs3ME5yUC96N3QKWWtZeTVMbGFsak5Oa1ByeHFLN2hPbGttUElRN1MwNm9MY1JDekNDR3NpSVRRaWhvcmRuYkJ6dnFuaitmNVB4OQptSTdJV0F2NjJDb0xOVWlORzRuc3FNbzN2OGcxWWtwbWRJZUFrekRmN3RzTWhJakw1c2I2QVhlcHZyR3hYdnM2CnRQeVNtVjRNQWF1L09iSng5UjRCanY2Ny8wRnI0VXo0VFVRczB2UXhReW9HSENyOVlUazkyS1JoZEFvcnhaVTEKR3BVNWxab1ZtcC9ONG1taVg1UTN2YjdWK3A5aFVzM3FiZWU1L3pBdFF2QUlpdDU5aVJVclV5ZnovWkxvanpCQQo2Z3NkT0FPVGlTVk0xZ0xZcVJDYWRvdHFxTzl4bnFnQ0hodjhyQklka1NKQ1pYTzdhWUplaUZjQ0F3RUFBYU5TCk1GQXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUJNQXdHQTFVZEV3RUIKL3dRQ01BQXdHd1lEVlIwUkJCUXdFb0lRZDNkM0xpUkVUMDFCU1U1ZlRrRk5SVEFOQmdrcWhraUc5dzBCQVFzRgpBQU9DQVFFQU1NNkQxY1Y1TUtzSXpWZUVJNW5RWU9teCtobnVHMUVWT09SZ2hQS1UzSXZNMjNMSkcxSEM3VDNaCm9FSVZXS2krUWVpaTJBNW1hQmZPaHF0ZEdPa0xScUF4NWcreUNYMnkzWU83NDkwQUtoc2dyVXFCNmVMRHpQTEgKRXl0VE9tWlRCWW9yNFQrMnNEZ1JQZWp1ZVNkVUZLRjhDN2JKUWRNRU5UcmpETURkTlUzQ3J5YlF3Q0ZlRFB6bgp5Q2dZT0k3aStxQUhWdWRPb2diWm5TTG1UK1JiTXVTQjVCU3h6SWZsMzB2MHg3R2NNU2gwa0xsTkczdVkwajlWClNuL3N3VGJ2eVBaVEMxbEFIYWt0MTA3Vko5elVuMjg4c3VndFc0SlRhZWxaQWJweG4wWUZyUmFrTUhGUlVZeEIKK1hHWmJWRnZmRklISWxMM2JvYmdkOEdoNHBucG9nPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFakNDQWZxZ0F3SUJBZ0lRRDYzbnNkZ2Y1YUI0MjN2QndaTExZVEFOQmdrcWhraUc5dzBCQVFzRkFEQWIKTVJrd0Z3WURWUVFEREJCM2QzY3VKRVJQVFVGSlRsOU9RVTFGTUI0WERUSTBNVEl4TnpFMk16TXpNbG9YRFRJMQpNRE14TnpFMk16TXpNbG93R3pFWk1CY0dBMVVFQXd3UWQzZDNMaVJFVDAxQlNVNWZUa0ZOUlRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU0rRmIxV1V3YjEwVE85UjFSOGhGREs3ME5yUC96N3QKWWtZeTVMbGFsak5Oa1ByeHFLN2hPbGttUElRN1MwNm9MY1JDekNDR3NpSVRRaWhvcmRuYkJ6dnFuaitmNVB4OQptSTdJV0F2NjJDb0xOVWlORzRuc3FNbzN2OGcxWWtwbWRJZUFrekRmN3RzTWhJakw1c2I2QVhlcHZyR3hYdnM2CnRQeVNtVjRNQWF1L09iSng5UjRCanY2Ny8wRnI0VXo0VFVRczB2UXhReW9HSENyOVlUazkyS1JoZEFvcnhaVTEKR3BVNWxab1ZtcC9ONG1taVg1UTN2YjdWK3A5aFVzM3FiZWU1L3pBdFF2QUlpdDU5aVJVclV5ZnovWkxvanpCQQo2Z3NkT0FPVGlTVk0xZ0xZcVJDYWRvdHFxTzl4bnFnQ0hodjhyQklka1NKQ1pYTzdhWUplaUZjQ0F3RUFBYU5TCk1GQXdEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CTUdBMVVkSlFRTU1Bb0dDQ3NHQVFVRkJ3TUJNQXdHQTFVZEV3RUIKL3dRQ01BQXdHd1lEVlIwUkJCUXdFb0lRZDNkM0xpUkVUMDFCU1U1ZlRrRk5SVEFOQmdrcWhraUc5dzBCQVFzRgpBQU9DQVFFQU1NNkQxY1Y1TUtzSXpWZUVJNW5RWU9teCtobnVHMUVWT09SZ2hQS1UzSXZNMjNMSkcxSEM3VDNaCm9FSVZXS2krUWVpaTJBNW1hQmZPaHF0ZEdPa0xScUF4NWcreUNYMnkzWU83NDkwQUtoc2dyVXFCNmVMRHpQTEgKRXl0VE9tWlRCWW9yNFQrMnNEZ1JQZWp1ZVNkVUZLRjhDN2JKUWRNRU5UcmpETURkTlUzQ3J5YlF3Q0ZlRFB6bgp5Q2dZT0k3aStxQUhWdWRPb2diWm5TTG1UK1JiTXVTQjVCU3h6SWZsMzB2MHg3R2NNU2gwa0xsTkczdVkwajlWClNuL3N3VGJ2eVBaVEMxbEFIYWt0MTA3Vko5elVuMjg4c3VndFc0SlRhZWxaQWJweG4wWUZyUmFrTUhGUlVZeEIKK1hHWmJWRnZmRklISWxMM2JvYmdkOEdoNHBucG9nPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBejRWdlZaVEJ2WFJNNzFIVkh5RVVNcnZRMnMvL1B1MWlSakxrdVZxV00wMlErdkdvCnJ1RTZXU1k4aER0TFRxZ3R4RUxNSUlheUloTkNLR2l0MmRzSE8rcWVQNS9rL0gyWWpzaFlDL3JZS2dzMVNJMGIKaWV5b3lqZS95RFZpU21aMGg0Q1RNTi91Mnd5RWlNdm14dm9CZDZtK3NiRmUrenEwL0pLWlhnd0JxNzg1c25IMQpIZ0dPL3J2L1FXdmhUUGhOUkN6UzlERkRLZ1ljS3YxaE9UM1lwR0YwQ2l2RmxUVWFsVG1WbWhXYW44M2lhYUpmCmxEZTl2dFg2bjJGU3plcHQ1N24vTUMxQzhBaUszbjJKRlN0VEovUDlrdWlQTUVEcUN4MDRBNU9KSlV6V0F0aXAKRUpwMmkycW83M0dlcUFJZUcveXNFaDJSSWtKbGM3dHBnbDZJVndJREFRQUJBb0lCQUJvRUNHa2FOUStqUE9Iagp5R3dXb1ltYXNodzhaeXRrWnlXVnk4YlE3SE9TRGRMbjZOWER0cVRtZmorMWowTlZJaWV3bUJhYnpFa3BVT252CjladVNNZDM1RTgyMkplYmhkL2F1ZHh0SlgwQW1GNC80NExMeDBGZEpsZWxyaDk3NzZTL3psb0JTRW9lemtnbjAKUmdJNzVXTXpESTBLUjNZSGJoa2I1ODQ2Z3hzY3JEZCt6c2Q1MVEyVFlTSjV4SDZxdExGVzNCNEs2dmNyMUpQNwoyRkNVMnMvY2U1dTBLeDV4c0JqYjhhY3N4SW1DNFBURWVhQkdGclp0alB5NFNrOGdhbW81alp3dTBoYkw2dGE3CjBtM2RDSFBRc2duQSsrZjc0L2FpR3c1bFVBSkIrYjlhNnJrMW8vMnlIN3V6TFBwQVFlZk9icjF1N2w2NGZoRVQKUG9uM08wRUNnWUVBNEpOVDlHaXd6OG5QOEFwZEFhZFN1a2pMdmNKcTRLUlA3ZVRTbEpXYlliOEtBeHdXWWRQLwo3VENVM05jNHAzOWRUay9jaHhSN2wybEZYSDlJN0pZS0NkQk1vMkUyOHc4TUhLS0JHaDRBVW12blVDR09sMk1YCm1GZW41dXU4SGF1THFQQlVJay9XUjZOcVJDUjZPL0NVNzNMb3lZY3p6SkdIalBJVVJhRGlUdjBDZ1lFQTdJOHgKdjdQTS9jcTRRR05XY2pSQkE0YWNQU3lTQWIyVjZIdi9DeEZteUlKblNVcFIvN2RXSUZvTVZ2TkZSeThxTlNQUwoyNWRzcWc2anRIMG9ZZiswdUlMODhYRFpjMWx5Q1BLK0ZHMldmWmVlMFFNVWZxQk1sNnJMUXg1UG5oYzdNWVNKCnNqOGlmcEdRVFczbDZtWkFMeWd4MTdRZGYxZHU2eWNnaStuSDF1TUNnWUJTTVZtKzBEbzJNcFNxTkFES1JDVWgKRE5lVXdmMC83cU5hWEM4K05tdENJUnRXL0FhdEUxeE53VVFhZDZMQ212alpXcDFKeXhOWTNTeUhxQVpXb0t0WAo0Q2JNZlI0UlkyWjViSGgzeUpjeGsxQjdvV3lBdDhIQS9nUHpuaitoVkZHdktSYUI0RnA2cWJLUThoRFNzMEN2ClNvMHNoUkZPUTgwcXBzY0luTk4zT1FLQmdDKzdsU1pCZVpPY1hDRmg2Z2VSVTYwS2RHeERtaEtQMXJmYWpWUUcKWnQ3K3NTaXd2bXNDNjNZTHRZRzJCTHlnWDd2cFJwcllxeXlGVjdNTHJpSllIbmM4dHgxZ3ovL2VoSXdacnhFZwpLOFdiMm9nQ1B4aHQ4QWFLaUFrN0RJV3gybGZiam8xUS8vemFFbXdOU045TFVwZ3hYQjFTVXNGbk94Sm9Dc0V3Cm93cFJBb0dCQU4vYVBtdG1HdDJlbkZPQUp6dExOSzdPaHp5RUZ3cGFZLytYMG95L1FyRnloMXZIcm0wTUpCT3UKL1FFeGVGRUlQZ2JlSC9mM2JHc3pFa3FxUXpUcVV4ZjZKSFhDZzRjQnRwZzdrZnlDVEdCV3Q1U0FuSDZESm9BVwo5TXc1b3B6M0krVy93N1NzbVBKTnNhT2dEUlI3bnJOektMTTVwMmVkSThhWGl3RmRaQm1iCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: www.$DOMAIN_NAME
    cert-manager.io/certificate-name: www
    cert-manager.io/common-name: www.$DOMAIN_NAME
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: selfsigned
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2024-12-17T16:33:32Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: www-tls
  namespace: default
  resourceVersion: "14982447"
  uid: 39114b96-fbb1-46e9-b5c0-8d7b223edca6
type: kubernetes.io/tls


michaelobrien@mbp8 eks % kubectl delete secret www-tls -n default
secret "www-tls" deleted
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml 
certificate.cert-manager.io/www unchanged
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www unchanged
michaelobrien@mbp8 eks % kubectl get secret www-tls -o yaml      
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lSQVBFSnRLVXdXRlJMM05iYXY5VG9FOGt3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXd3UWQzZDNMaVJFVDAxQlNVNWZUa0ZOUlRBZUZ3MHlOREV5TVRjeE5qTTRNelZhRncweQpOVEF6TVRjeE5qTTRNelZhTUJzeEdUQVhCZ05WQkFNTUVIZDNkeTRrUkU5TlFVbE9YMDVCVFVVd2dnRWlNQTBHCkNTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFER0dCV1owU1dqWDlFaU8rSE9lMlhJZXdITDRtb0cKbUoydFNCcEQxZ094dXJvNTVSeHF4eVE4aVNLRG5ibDVzazdGRTRST3FQa04vZjdQM2ExbXcwamZsUWZ2SDV5Lwpmc2NWUFZTRm05ZnRvTkpqS3ExY3RwalQyQjJCQm5QQ1BkMFJJOHUvenFxamtFL09yWmVMdXpEUU9vNE5JUWszCkZST1czdVNqYTlxd25CNE1mL0tqMEhiemRaSExPQzF5Mnp1RVhvNyt4bkVyTVVBVHU4M0pnKzVXMDYybm9XaC8KcG9wUjVvd1Q1SzczQUtlS1pWeXZaRGllVWZKZkZlK3doNEdWSDVTQUN4NkE1VmtZSEgxTEFUbHFsZXJ1SWdYQQp6djlxSlB5UEZCbVlxTkJHSU10c2lPY2VGNU1OYXZrR1pTOTI4dVFYQ3h6NW1JVWl0Z0RTMyt6MUFnTUJBQUdqClVqQlFNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBVEFNQmdOVkhSTUIKQWY4RUFqQUFNQnNHQTFVZEVRUVVNQktDRUhkM2R5NGtSRTlOUVVsT1gwNUJUVVV3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFHem14cDF4T0g1NGJ4K0NybHZ6UFFydGJaRU1Qd0d5T0N4OEY5QUp3UlNib3Fza1FSZXNBdDZ0CkQwL0pxTmtmc0NEcWlKYThiZUtnUjhOeHVBSjRGUytEY2NCWTVkK1RlZFEwa2o1Z1BoWmtydE10UkFQdTEra1IKbkFFZlArRFhjSmFOc29iWkZ5SUlwR09WaittY1J1WHc3SFAvYnM2cm5ndTgzWjNXSEdBRWhuYnBUT2phemN0bwplS0tWcURITE5xWXZKczh1MGZvSHMxNGRHaUY1ckt5VXVUSytWVkl4RlB3d3pONlFxbEYvelhIN3ptMkQ4bitECkM2cXNDMVZRSTJ4TjU0RS8xVlduc1c5NUxQSm8rWXlQazBPaWd5YTNnQno1aU56SGc1eHNYVkYxcUxFZldUUGMKQUI0MHdWUWJIWEtoYVRCNWo5cVlGSkpoUlM2Vzlnbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lSQVBFSnRLVXdXRlJMM05iYXY5VG9FOGt3RFFZSktvWklodmNOQVFFTEJRQXcKR3pFWk1CY0dBMVVFQXd3UWQzZDNMaVJFVDAxQlNVNWZUa0ZOUlRBZUZ3MHlOREV5TVRjeE5qTTRNelZhRncweQpOVEF6TVRjeE5qTTRNelZhTUJzeEdUQVhCZ05WQkFNTUVIZDNkeTRrUkU5TlFVbE9YMDVCVFVVd2dnRWlNQTBHCkNTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFER0dCV1owU1dqWDlFaU8rSE9lMlhJZXdITDRtb0cKbUoydFNCcEQxZ094dXJvNTVSeHF4eVE4aVNLRG5ibDVzazdGRTRST3FQa04vZjdQM2ExbXcwamZsUWZ2SDV5Lwpmc2NWUFZTRm05ZnRvTkpqS3ExY3RwalQyQjJCQm5QQ1BkMFJJOHUvenFxamtFL09yWmVMdXpEUU9vNE5JUWszCkZST1czdVNqYTlxd25CNE1mL0tqMEhiemRaSExPQzF5Mnp1RVhvNyt4bkVyTVVBVHU4M0pnKzVXMDYybm9XaC8KcG9wUjVvd1Q1SzczQUtlS1pWeXZaRGllVWZKZkZlK3doNEdWSDVTQUN4NkE1VmtZSEgxTEFUbHFsZXJ1SWdYQQp6djlxSlB5UEZCbVlxTkJHSU10c2lPY2VGNU1OYXZrR1pTOTI4dVFYQ3h6NW1JVWl0Z0RTMyt6MUFnTUJBQUdqClVqQlFNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBVEFNQmdOVkhSTUIKQWY4RUFqQUFNQnNHQTFVZEVRUVVNQktDRUhkM2R5NGtSRTlOUVVsT1gwNUJUVVV3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFHem14cDF4T0g1NGJ4K0NybHZ6UFFydGJaRU1Qd0d5T0N4OEY5QUp3UlNib3Fza1FSZXNBdDZ0CkQwL0pxTmtmc0NEcWlKYThiZUtnUjhOeHVBSjRGUytEY2NCWTVkK1RlZFEwa2o1Z1BoWmtydE10UkFQdTEra1IKbkFFZlArRFhjSmFOc29iWkZ5SUlwR09WaittY1J1WHc3SFAvYnM2cm5ndTgzWjNXSEdBRWhuYnBUT2phemN0bwplS0tWcURITE5xWXZKczh1MGZvSHMxNGRHaUY1ckt5VXVUSytWVkl4RlB3d3pONlFxbEYvelhIN3ptMkQ4bitECkM2cXNDMVZRSTJ4TjU0RS8xVlduc1c5NUxQSm8rWXlQazBPaWd5YTNnQno1aU56SGc1eHNYVkYxcUxFZldUUGMKQUI0MHdWUWJIWEtoYVRCNWo5cVlGSkpoUlM2Vzlnbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcGdJQkFBS0NBUUVBeGhnVm1kRWxvMS9SSWp2aHpudGx5SHNCeStKcUJwaWRyVWdhUTlZRHNicTZPZVVjCmFzY2tQSWtpZzUyNWViSk94Uk9FVHFqNURmMyt6OTJ0WnNOSTM1VUg3eCtjdjM3SEZUMVVoWnZYN2FEU1l5cXQKWExhWTA5Z2RnUVp6d2ozZEVTUEx2ODZxbzVCUHpxMlhpN3N3MERxT0RTRUpOeFVUbHQ3a28ydmFzSndlREgveQpvOUIyODNXUnl6Z3RjdHM3aEY2Ty9zWnhLekZBRTd2TnlZUHVWdE90cDZGb2Y2YUtVZWFNRStTdTl3Q25pbVZjCnIyUTRubEh5WHhYdnNJZUJsUitVZ0FzZWdPVlpHQng5U3dFNWFwWHE3aUlGd003L2FpVDhqeFFabUtqUVJpREwKYklqbkhoZVREV3I1Qm1VdmR2TGtGd3NjK1ppRklyWUEwdC9zOVFJREFRQUJBb0lCQVFDb09YWEduV1Z3TzRPWApwZkRaNzlMOUVQRHltd0ZkUHFmUXF3V2JEQmt4Q3BqeFo0N2RZdVp6L2lWd1RqWmZrYnZ4QXVUK0ErTVZMZGtCCjM0aE5WNVFBUTBoc21wVDd6b2IwazVPK3J4ME5NT0RaMURsTEhjbmxSWWV3dGJFQ2lKUys1SDQ0RWlLNUttbUwKNDhGcHFYc0hxbUpONWZzMGF0S3E4ZmlLaTZJNURhczhZdmgrMXBuQVRBRXcvR2MyMGJKWHZWOG1EN3d4Y1NHbApvNy9OTU5ERVNQZDNHd2xNU2xJdy9Xb2paaE1ZZVhFemVrM05PcjRWb1dhcTZiQXRzRDAwVUpSZjlpbW1pYjFoCnlEcGQ5OWFZMzRaQVRHb0xSN0lIakY3bm1zM1RaeGM5dGE1N0QrU3QvVGJNcnFUZE45MjF3Uk1yYlRpR1lmbHMKZ09NMFQ4c2hBb0dCQVBZallCOGc4aTVnOEpHditpaWhnL3Q4UXdtS0JxbHpoOG5OVmYyM1dud2JJZzh6L3ZITAo5dUxuSTZwelBYV0w5ZjlIV2x1QTFwQUZDR05YRCtHWlZXVzl3QzZGNXV1dS9vVnJBMjh1UW5rK2w2OXdDVDdtCkRFWlpMZlR6YThwRC9LWFBQeUdCd0tWTktBVDlGanlvODgyTGVEc2FoNUJ3NnhacVZadFN1WVF0QW9HQkFNNEgKN0dSbFFTcWpOV2RnTlZ0R0greGpNOTBoVlRteTVsTmgyT2YyWUdCMzZrb0tnbDZIVStlaUNLc2QyQlhZZWc5OQpmcGRDdFArNi9nQm5xanEvZlhWQTdLcjFFeGdCTWRYckgzZHRlWVJaUHJyUW5qMzFLMG01SThpS2JzUEx2ejlTCkZXN29hNzBUbm5yOU1hVng3QnYwWEw1YXhtdXJucDRYVE8xUkpTRHBBb0dCQUtJSGE1RlNXTnJiQ2NKOSs4bWYKZVlpbHBtT3pMdXVCbUJQcVRmODFJaVdzbnVFdk0xYnRFOXNPbVZYdGdybk1wdVdqalFITGNITmlQTUZZWUtYOApFRkZuZGdtcUwvZGFQVWFLeWJuNDA5MklyTVhhbE51SkNPK09hVWhrcmNKNnV1aFQ0NVVsR3pJMTluQkNRczh2CjBUNnYzemZaSVFDZXRNTU93TWY4enc4SkFvR0JBSiszcXIwRmRhaE5ENWhORCsvckE5Sm44UXZLekZ2SFlTQmcKcVJyaGQwTXhrdEhGSmQyTUFUSisvVU5TQm5LZzk3Yk1iWGRyUWc5QmNhajB0QldJM3dRbnlaQ2FIaXdzdU9xQgp0RW9MOXZqdmlsNmVYUjNnMHFiN1BFODVhYmFsVzJRR0NSR0tZZ01ZazNINm1DTU9scC94Sm5oMnZIMDVkT1JQCllkWlRZbm9KQW9HQkFPem4vNCtlaC9ZbDJxQUU2Y1B5RVFqZlA4Ym1FTThyWDdNL25sNmh1K3F4UDYzM2pxMDAKOUxYTDdHUUlBbGlJem5ldEJoYVpnaTlTZlMrTy9SV241TGZNZFZxa092djBoR2NEWnVLR0grUXNqT0swL0V3SApFY2FzR3FUalJJeWt2N2llekh2OWcwa2dDOHg3QzE1N204cVhzTitSbHdTSm9yZGk5U01LQit4YwotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: www.$DOMAIN_NAME
    cert-manager.io/certificate-name: www
    cert-manager.io/common-name: www.$DOMAIN_NAME
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: selfsigned
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2024-12-17T16:38:35Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: www-tls
  namespace: default
  resourceVersion: "14983328"
  uid: 4e1709da-f788-4bcf-adf7-b121192aeb08
type: kubernetes.io/tls

i will try adding fields for hashing in the deployment derived from the cert

obriensystems · 2024-12-17T17:06:07Z

retrying delete of certificate

michaelobrien@mbp8 eks % kubectl get certificate -o wide
NAME   READY   SECRET    ISSUER       STATUS                                          AGE
www    True    www-tls   selfsigned   Certificate is up to date and has not expired   32m
michaelobrien@mbp8 eks % kubectl delete -f certificate.yaml 
certificate.cert-manager.io "www" deleted
michaelobrien@mbp8 eks % kubectl get certificate -o wide   
No resources found in default namespace.
michaelobrien@mbp8 eks % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www created
michaelobrien@mbp8 eks % kubectl get certificate -o wide  
NAME   READY   SECRET    ISSUER       STATUS                                          AGE
www    True    www-tls   selfsigned   Certificate is up to date and has not expired   5s
michaelobrien@mbp8 eks % kubectl get pods -o wide         
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE                            NOMINATED NODE   READINESS GATES
helloweb-tls-5d7dc89df7-c7vlq        1/1     Running   0          20m     172.31.84.250   ip-172-31-80-165.ec2.internal   <none>           <none>
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d23h   172.31.45.140   ip-172-31-42-248.ec2.internal   <none>           <none>

no logs or roll

it looks like I should be deleting the secret - this also does not trigger reloader

michaelobrien@mbp8 eks % kubectl get secret -o wide      
NAME      TYPE                DATA   AGE
www-tls   kubernetes.io/tls   3      26m
michaelobrien@mbp8 eks % kubectl delete secret www-tls
secret "www-tls" deleted
michaelobrien@mbp8 eks % kubectl get secret -o wide   
NAME      TYPE                DATA   AGE
www-tls   kubernetes.io/tls   3      4s
michaelobrien@mbp8 eks % kubectl get pods -o wide        
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE                            NOMINATED NODE   READINESS GATES
helloweb-tls-5c7c5dc7c-bnd8p         1/1     Running   0          5m3s    172.31.95.132   ip-172-31-80-165.ec2.internal   <none>           <none>
reloader-reloader-5d85b9bbd9-447sg   1/1     Running   0          4d23h   172.31.45.140   ip-172-31-42-248.ec2.internal   <none>           <none>

secret in the pod is unchanged

michaelobrien@mbp8 eks % kubectl describe pod helloweb-tls-5c7c5dc7c-bnd8p
Name:             helloweb-tls-5c7c5dc7c-bnd8p
...
    Environment:
      dummy-env-name:           dummy-env-value2
      TLS_CERT:                 /etc/tls/tls.crt
      TLS_KEY:                  /etc/tls/tls.key
      STAKATER_WWW_TLS_SECRET:  c94f58ebc4942ee95afb3b1cfb2d8dbdec17fcf1
    Mounts:
      /etc/tls from tls (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4tz8g (ro)
...
Volumes:
  tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  www-tls
    Optional:    false
...
michaelobrien@mbp8 eks % kubectl delete secret www-tls                    
secret "www-tls" deleted
michaelobrien@mbp8 eks % kubectl get secret -o wide                       
NAME      TYPE                DATA   AGE
...
    Environment:
      dummy-env-name:           dummy-env-value2
      TLS_CERT:                 /etc/tls/tls.crt
      TLS_KEY:                  /etc/tls/tls.key
      STAKATER_WWW_TLS_SECRET:  c94f58ebc4942ee95afb3b1cfb2d8dbdec17fcf1
    Mounts:
      /etc/tls from tls (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4tz8g (ro)
... 
Volumes:
  tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  www-tls
    Optional:    false

michaelobrien@mbp8 eks % kubectl describe certificate www

unchanged STAKATER_WWW_TLS_SECRET: c94f58ebc4942ee95afb3b1cfb2d8dbdec17fcf1
but
the certificate events show the recreation

  Normal  Issuing    2s (x15 over 157m)   cert-manager-certificates-issuing          The certificate has been successfully issued
  Normal  Issuing    2s (x11 over 152m)   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  2s                   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "www-hlln5"
  Normal  Requested  2s                   cert-manager-certificates-request-manager  Created new CertificateRequest resource "www-15"

obriensystems · 2024-12-17T19:50:50Z

the uid changes on secret deletion

kubectl delete secret www-tls 
 kubectl get secret www-tls -o yaml
uid: 227a908b-4511-4cc1-b0e4-ad6d5408ea33


rechecking stakater env vars
michaelobrien@mbp8 eks % kubectl get pod helloweb-tls-778d5dcd9-l4pm2 -o yaml | grep -A1 STAK
    - name: STAKATER_WWW_TLS_SECRET
      value: cc0663c05a93d019e80a273ec793c013b2f7fdc2

delete again
michaelobrien@mbp8 eks % kubectl delete secret www-tls                                            
secret "www-tls" deleted
michaelobrien@mbp8 eks % kubectl get pod helloweb-tls-778d5dcd9-l4pm2 -o yaml | grep -A1 STAK     
    - name: STAKATER_WWW_TLS_SECRET
      value: cc0663c05a93d019e80a273ec793c013b2f7fdc2

unchanged

However, if the domain name on the cert is changed - this triggers a rollover as expected - only in this case the secret value is changed
michaelobrien@mbp8 eks % kubectl get pod helloweb-tls-59d77678cf-6pfl6 -o yaml | grep -A1 STAK
    - name: STAKATER_WWW_TLS_SECRET
      value: c13ac60ddb8a6fe9aebfa1a7461ca23a7312ab2c

obriensystems · 2024-12-18T17:01:10Z

On M1 Max - verify helm charts as opposed to kubernetes raw yamls
https://github.com/stakater/Reloader?tab=readme-ov-file#helm-charts

michaelobrien@mbp7 eks % helm version   
version.BuildInfo{Version:"v3.16.3", GitCommit:"cfd07493f46efc9debd9cc1b02a0961186df7fdf", GitTreeState:"dirty", GoVersion:"go1.23.3"}
michaelobrien@mbp7 eks % helm repo add stakater https://stakater.github.io/stakater-charts
"stakater" has been added to your repositories
michaelobrien@mbp7 eks % helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "stakater" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
michaelobrien@mbp7 eks % helm install stakater/reloader --generate-name
NAME: reloader-1734540518
LAST DEPLOYED: Wed Dec 18 11:48:38 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
- For a `Deployment` called `foo` have a `ConfigMap` called `foo-configmap`. Then add this annotation to main metadata of your `Deployment`
  configmap.reloader.stakater.com/reload: "foo-configmap"

- For a `Deployment` called `foo` have a `Secret` called `foo-secret`. Then add this annotation to main metadata of your `Deployment`
  secret.reloader.stakater.com/reload: "foo-secret"

- After successful installation, your pods will get rolling updates when a change in data of configmap or secret will happen.
michaelobrien@mbp7 eks % kubectl get namespaces
NAME              STATUS   AGE
default           Active   66d
kube-node-lease   Active   66d
kube-public       Active   66d
kube-system       Active   66d
michaelobrien@mbp7 eks % kubectl create namespace reloader
namespace/reloader created
michaelobrien@mbp7 eks % helm list                                                                                          
NAME               	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART         	APP VERSION
reloader-1734540518	default  	1       	2024-12-18 11:48:38.678981 -0500 EST	deployed	reloader-1.2.0	v1.2.0     
michaelobrien@mbp7 eks % kubectl get pods
NAME                                            READY   STATUS    RESTARTS   AGE
reloader-1734540518-reloader-7d44c89899-pgrl4   1/1     Running   0          2m22s

michaelobrien@mbp7 eks % kubectl logs -f reloader-1734540518-reloader-7d44c89899-pgrl4
time="2024-12-18T16:48:42Z" level=info msg="Environment: Kubernetes"
time="2024-12-18T16:48:42Z" level=info msg="Starting Reloader"
time="2024-12-18T16:48:42Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2024-12-18T16:48:42Z" level=info msg="created controller for: configMaps"
time="2024-12-18T16:48:42Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2024-12-18T16:48:42Z" level=info msg="created controller for: secrets"
time="2024-12-18T16:48:42Z" level=info msg="Starting Controller to watch resource type: secrets"

add cert-manager - follow #3

michaelobrien@mbp7 dockerdesktop % helm install cert-manager cert-manager \
  --repo https://charts.jetstack.io \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true
NAME: cert-manager
LAST DEPLOYED: Wed Dec 18 12:07:29 2024
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.16.2 has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:

https://cert-manager.io/docs/configuration/

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/


michaelobrien@mbp7 dockerdesktop % kubectl get pods --all-namespaces -o wide
NAMESPACE      NAME                                            READY   STATUS    RESTARTS       AGE   IP             NODE             NOMINATED NODE   READINESS GATES
cert-manager   cert-manager-974688dfd-4rrcl                    1/1     Running   0              98s   10.1.0.105     docker-desktop   <none>           <none>
cert-manager   cert-manager-cainjector-6687cc685b-wqzkb        1/1     Running   0              98s   10.1.0.106     docker-desktop   <none>           <none>
cert-manager   cert-manager-webhook-b8cdf84f-8v95h             1/1     Running   0              98s   10.1.0.104     docker-desktop   <none>           <none>
default        reloader-1734540518-reloader-7d44c89899-pgrl4   1/1     Running   0              20m   10.1.0.103     docker-desktop   <none>           <none>
kube-system    coredns-7db6d8ff4d-s9lwf                        1/1     Running   8 (18d ago)    66d   10.1.0.100     docker-desktop   <none>           <none>
kube-system    coredns-7db6d8ff4d-zt89m                        1/1     Running   8 (18d ago)    66d   10.1.0.99      docker-desktop   <none>           <none>
kube-system    etcd-docker-desktop                             1/1     Running   8 (18d ago)    66d   192.168.65.3   docker-desktop   <none>           <none>
kube-system    kube-apiserver-docker-desktop                   1/1     Running   8 (26m ago)    66d   192.168.65.3   docker-desktop   <none>           <none>
kube-system    kube-controller-manager-docker-desktop          1/1     Running   8 (18d ago)    66d   192.168.65.3   docker-desktop   <none>           <none>
kube-system    kube-proxy-zvblg                                1/1     Running   8 (18d ago)    66d   192.168.65.3   docker-desktop   <none>           <none>
kube-system    kube-scheduler-docker-desktop                   1/1     Running   9 (18d ago)    66d   192.168.65.3   docker-desktop   <none>           <none>
kube-system    storage-provisioner                             1/1     Running   17 (25m ago)   66d   10.1.0.101     docker-desktop   <none>           <none>
kube-system    vpnkit-controller                               1/1     Running   8 (18d ago)    66d   10.1.0.102     docker-desktop   <none>           <none>


michaelobrien@mbp7 dockerdesktop % export DOMAIN_NAME=eventfield.io
michaelobrien@mbp7 dockerdesktop % kubectl apply -f clusterissuer-selfsigned.yaml
clusterissuer.cert-manager.io/selfsigned created

michaelobrien@mbp7 dockerdesktop % curl -L https://github.com/a8m/envsubst/releases/download/v1.2.0/envsubst-`uname -s`-`uname -m` -o envsubst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 2080k  100 2080k    0     0  3832k      0 --:--:-- --:--:-- --:--:-- 3832k
michaelobrien@mbp7 dockerdesktop % chmod +x envsubst
michaelobrien@mbp7 dockerdesktop % sudo mv envsubst /usr/local/bin


michaelobrien@mbp7 dockerdesktop % envsubst < certificate.yaml | kubectl apply -f -
certificate.cert-manager.io/www created

michaelobrien@mbp7 dockerdesktop % kubectl get certificate
NAME   READY   SECRET    AGE
www    True    www-tls   44s
michaelobrien@mbp7 dockerdesktop % kubectl get secret
NAME                                        TYPE                 DATA   AGE
sh.helm.release.v1.reloader-1734540518.v1   helm.sh/release.v1   1      25m
www-tls                                     kubernetes.io/tls    3      52s

test - pre env and app change

michaelobrien@mbp7 dockerdesktop % kubectl delete secret www-tls
secret "www-tls" deleted
michaelobrien@mbp7 dockerdesktop % kubectl get secret                                           
NAME                                        TYPE                 DATA   AGE
sh.helm.release.v1.reloader-1734540518.v1   helm.sh/release.v1   1      27m
www-tls                                     kubernetes.io/tls    3      3s
michaelobrien@mbp7 dockerdesktop % kubectl logs -f reloader-1734540518-reloader-7d44c89899-pgrl4
time="2024-12-18T16:48:42Z" level=info msg="Environment: Kubernetes"
time="2024-12-18T16:48:42Z" level=info msg="Starting Reloader"
time="2024-12-18T16:48:42Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2024-12-18T16:48:42Z" level=info msg="created controller for: configMaps"
time="2024-12-18T16:48:42Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2024-12-18T16:48:42Z" level=info msg="created controller for: secrets"
time="2024-12-18T16:48:42Z" level=info msg="Starting Controller to watch resource type: secrets"


set reloadOnCreate=true

michaelobrien@mbp7 dockerdesktop % helm install stakater/reloader --generate-name --set reloader.reloadOnDelete=true                       
NAME: reloader-1734542855
LAST DEPLOYED: Wed Dec 18 12:27:35 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
- For a `Deployment` called `foo` have a `ConfigMap` called `foo-configmap`. Then add this annotation to main metadata of your `Deployment`
  configmap.reloader.stakater.com/reload: "foo-configmap"

- For a `Deployment` called `foo` have a `Secret` called `foo-secret`. Then add this annotation to main metadata of your `Deployment`
  secret.reloader.stakater.com/reload: "foo-secret"

- After successful installation, your pods will get rolling updates when a change in data of configmap or secret will happen.


delete old release
michaelobrien@mbp7 dockerdesktop % helm list
NAME               	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART         	APP VERSION
reloader-1734540518	default  	1       	2024-12-18 11:48:38.678981 -0500 EST	deployed	reloader-1.2.0	v1.2.0     
reloader-1734542855	default  	1       	2024-12-18 12:27:35.548978 -0500 EST	deployed	reloader-1.2.0	v1.2.0     
michaelobrien@mbp7 dockerdesktop % helm uninstall reloader-1734540518 -n default
release "reloader-1734540518" uninstalled
michaelobrien@mbp7 dockerdesktop % helm list                                    
NAME               	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART         	APP VERSION
reloader-1734542855	default  	1       	2024-12-18 12:27:35.548978 -0500 EST	deployed	reloader-1.2.0	v1.2.0     

check logs
michaelobrien@mbp7 dockerdesktop % kubectl logs  reloader-1734542855-reloader-7c7669459c-8g9xt                      
time="2024-12-18T17:27:36Z" level=info msg="Environment: Kubernetes"
time="2024-12-18T17:27:36Z" level=info msg="Starting Reloader"
time="2024-12-18T17:27:36Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2024-12-18T17:27:36Z" level=info msg="created controller for: configMaps"
time="2024-12-18T17:27:36Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2024-12-18T17:27:36Z" level=info msg="created controller for: secrets"
time="2024-12-18T17:27:36Z" level=info msg="Starting Controller to watch resource type: secrets"

add app
michaelobrien@mbp7 dockerdesktop % kubectl apply -f deployment.yaml 
deployment.apps/helloweb-tls created
michaelobrien@mbp7 dockerdesktop % kubectl get pods                                           
NAME                                            READY   STATUS    RESTARTS   AGE
helloweb-tls-9fbffcdbf-4j5xp                    1/1     Running   0          10s
helloweb-tls-9fbffcdbf-h29c4                    1/1     Running   0          10s
reloader-1734542855-reloader-7c7669459c-8g9xt   1/1     Running   0          5m41s
michaelobrien@mbp7 dockerdesktop % kubectl get deployment
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
helloweb-tls                   2/2     2            2           22s
reloader-1734542855-reloader   1/1     1            1           5m53s

test full force dns change = ok
michaelobrien@mbp7 dockerdesktop % kubectl apply -f certificate.yaml                          
certificate.cert-manager.io/www configured
michaelobrien@mbp7 dockerdesktop % kubectl logs  reloader-1734542855-reloader-7c7669459c-8g9xt
time="2024-12-18T17:34:05Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"


check secret deletion
michaelobrien@mbp7 dockerdesktop % kubectl delete secret www-tls                              
secret "www-tls" deleted

time="2024-12-18T17:35:01Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"

rollover happened
michaelobrien@mbp7 dockerdesktop % kubectl get deployment                                     
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
helloweb-tls                   2/2     2            2           2m30s
reloader-1734542855-reloader   1/1     1            1           8m1s
michaelobrien@mbp7 dockerdesktop % kubectl get pods                                           
NAME                                            READY   STATUS    RESTARTS   AGE
helloweb-tls-9fbffcdbf-6ppxx                    1/1     Running   0          37s
helloweb-tls-9fbffcdbf-bgg4q                    1/1     Running   0          39s
reloader-1734542855-reloader-7c7669459c-8g9xt   1/1     Running   0          8m5s


2nd one not
michaelobrien@mbp7 dockerdesktop % kubectl delete secret www-tls                              
secret "www-tls" deleted
michaelobrien@mbp7 dockerdesktop % kubectl get pods             
NAME                                            READY   STATUS    RESTARTS   AGE
helloweb-tls-9fbffcdbf-6ppxx                    1/1     Running   0          63s
helloweb-tls-9fbffcdbf-bgg4q                    1/1     Running   0          65s
reloader-1734542855-reloader-7c7669459c-8g9xt   1/1     Running   0          8m31s
michaelobrien@mbp7 dockerdesktop % kubectl get pods
NAME                                            READY   STATUS    RESTARTS   AGE
helloweb-tls-9fbffcdbf-6ppxx                    1/1     Running   0          65s
helloweb-tls-9fbffcdbf-bgg4q                    1/1     Running   0          67s
reloader-1734542855-reloader-7c7669459c-8g9xt   1/1     Running   0          8m33s
michaelobrien@mbp7 dockerdesktop % kubectl get pods
NAME                                            READY   STATUS    RESTARTS   AGE
helloweb-tls-9fbffcdbf-6ppxx                    1/1     Running   0          77s
helloweb-tls-9fbffcdbf-bgg4q                    1/1     Running   0          79s
reloader-1734542855-reloader-7c7669459c-8g9xt   1/1     Running   0          8m45s
michaelobrien@mbp7 dockerdesktop % kubectl logs  reloader-1734542855-reloader-7c7669459c-8g9xt
time="2024-12-18T17:27:36Z" level=info msg="Environment: Kubernetes"
time="2024-12-18T17:27:36Z" level=info msg="Starting Reloader"
time="2024-12-18T17:27:36Z" level=warning msg="KUBERNETES_NAMESPACE is unset, will detect changes in all namespaces."
time="2024-12-18T17:27:36Z" level=info msg="created controller for: configMaps"
time="2024-12-18T17:27:36Z" level=info msg="Starting Controller to watch resource type: configMaps"
time="2024-12-18T17:27:36Z" level=info msg="created controller for: secrets"
time="2024-12-18T17:27:36Z" level=info msg="Starting Controller to watch resource type: secrets"
time="2024-12-18T17:34:05Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"
time="2024-12-18T17:35:01Z" level=info msg="Changes detected in 'www-tls' of type 'SECRET' in namespace 'default'; updated 'helloweb-tls' of type 'Deployment' in namespace 'default'"

obriensystems · 2024-12-19T17:13:50Z

reloader kicks in a 2nd time on a secret delete after a certificate change (dns)

helm install stakater/reloader --generate-name --set reloader.reloadOnDelete=true

michaelobrien@mbp7 dockerdesktop % kubectl describe pod reloader-1734542855-reloader-7c7669459c-8g9xt | grep -A3 Args
    Args:
      --log-level=info
      --reload-on-delete=true
    State:          Running

use case: change dns on cert - apply, then delete old secret from previous - triggers reloader
A regenerate of the secret for an existing aligned certificate-secret has no change effect for the reloader

example

cert updated - secret not
michaelobrien@mbp7 dockerdesktop % kubectl apply -f certificate.yaml
certificate.cert-manager.io/www configured

older secret
michaelobrien@mbp7 dockerdesktop % kubectl describe secret www-tls
Name:         www-tls
Namespace:    default
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: local7.$DOMAIN_NAME
              cert-manager.io/certificate-name: www
              cert-manager.io/common-name: local.$DOMAIN_NAME
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: 
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: selfsigned
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
tls.crt:  1135 bytes
tls.key:  1675 bytes
ca.crt:   1135 bytes
michaelobrien@mbp7 dockerdesktop % kubectl delete secret www-tls    
secret "www-tls" deleted
michaelobrien@mbp7 dockerdesktop % kubectl describe secret www-tls
Name:         www-tls
Namespace:    default
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: local7.$DOMAIN_NAME
              cert-manager.io/certificate-name: www
              cert-manager.io/common-name: local.$DOMAIN_NAME
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: 
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: selfsigned
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
ca.crt:   1135 bytes
tls.crt:  1135 bytes
tls.key:  1679 bytes

obriensystems self-assigned this Dec 11, 2024

obriensystems mentioned this issue Dec 12, 2024

Experimentation: cert-manager end to end canary - EKS #3

Open

obriensystems added a commit that referenced this issue Dec 12, 2024

#6 - add reloader

0c05feb

obriensystems added a commit that referenced this issue Dec 12, 2024

#6 - add dummy env for tracking

6a1bd1e

obriensystems added a commit that referenced this issue Dec 15, 2024

#6 - enable reloader

3dbc7a7

obriensystems added a commit that referenced this issue Dec 16, 2024

#6 - standardize selectors

7078758

obriensystems added a commit that referenced this issue Dec 16, 2024

#6 - replicaset 2

5e32156

obriensystems mentioned this issue Dec 16, 2024

Kubernetes ConfigMap or secret volume change tracking - with stakater/Reloader ObrienlabsDev/blog#81

Open

obriensystems added a commit that referenced this issue Dec 19, 2024

#6 - docker desktop helm

0945b81

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes Change Tracking: Add reloader to roll deployments after env or secret changes #6

Kubernetes Change Tracking: Add reloader to roll deployments after env or secret changes #6

obriensystems commented Dec 11, 2024 •

edited

Loading

obriensystems commented Dec 12, 2024 •

edited

Loading

obriensystems commented Dec 12, 2024 •

edited

Loading

obriensystems commented Dec 16, 2024 •

edited

Loading

obriensystems commented Dec 16, 2024 •

edited

Loading

obriensystems commented Dec 16, 2024

obriensystems commented Dec 16, 2024 •

edited

Loading

obriensystems commented Dec 17, 2024 •

edited

Loading

obriensystems commented Dec 17, 2024 •

edited

Loading

obriensystems commented Dec 17, 2024 •

edited

Loading

obriensystems commented Dec 18, 2024 •

edited

Loading

obriensystems commented Dec 19, 2024 •

edited

Loading

Kubernetes Change Tracking: Add reloader to roll deployments after env or secret changes #6

Kubernetes Change Tracking: Add reloader to roll deployments after env or secret changes #6

Comments

obriensystems commented Dec 11, 2024 • edited Loading

obriensystems commented Dec 12, 2024 • edited Loading

obriensystems commented Dec 12, 2024 • edited Loading

obriensystems commented Dec 16, 2024 • edited Loading

obriensystems commented Dec 16, 2024 • edited Loading

obriensystems commented Dec 16, 2024

obriensystems commented Dec 16, 2024 • edited Loading

obriensystems commented Dec 17, 2024 • edited Loading

obriensystems commented Dec 17, 2024 • edited Loading

obriensystems commented Dec 17, 2024 • edited Loading

obriensystems commented Dec 18, 2024 • edited Loading

obriensystems commented Dec 19, 2024 • edited Loading

obriensystems commented Dec 11, 2024 •

edited

Loading

obriensystems commented Dec 12, 2024 •

edited

Loading

obriensystems commented Dec 12, 2024 •

edited

Loading

obriensystems commented Dec 16, 2024 •

edited

Loading

obriensystems commented Dec 16, 2024 •

edited

Loading

obriensystems commented Dec 16, 2024 •

edited

Loading

obriensystems commented Dec 17, 2024 •

edited

Loading

obriensystems commented Dec 17, 2024 •

edited

Loading

obriensystems commented Dec 17, 2024 •

edited

Loading

obriensystems commented Dec 18, 2024 •

edited

Loading

obriensystems commented Dec 19, 2024 •

edited

Loading