not picking up TLS secret updates

[jmhodges]

I'm on GKE k8s 1.11.7-gke.12

My secret with my tls certificate in it was updated by my automated systems on 2019-04-13 00:58:40 but my GLB ingress has still not picked it up.

There is one entry in my gcloud compute ssl-certificates list:

NAME CREATION_TIMESTAMP
k8s-ssl-0861999af802e3cc-8526cda76e62be92--bb860f3113bd5eae 2019-02-16T18:39:46.060-08:00

It's old. It also has a suspicious "--" in it's name. In the past, I had to delete my cluster whole cloth to fix a similar bug because my ingress-uid configmap had no data under provider-uid and uid. (See #311).

However, this time, the ingress-uid configmap looks correct.

There seems to be some regression of some kind?

[rramkumar1]

@jmhodges Can you provide me with your project name, cluster name and cluster location?

I can take a closer look.

[jmhodges]

personal-sites-1295, dg, and us-east1-c

Thanks!

[rramkumar1]

@jmhodges I'm not seeing anything in your logs that indicates an error. Granted, I only just got to looking at this so the logs from the day this occurred for you have been rotated out.

Is it possible for you to reproduce the issue?

[jmhodges]

I'll give it a shot! This seems like prior tickets which were stochastic in their failure, so I don't have high hopes.

[jmhodges]

Ah, good! I just refreshed it, it's been 10 minutes, and it's still not updated.

[jmhodges]

Gentle ping!

[rramkumar1]

@jmhodges Sorry for some reason did not get notified on your previous message. Taking a look now.

[rramkumar1]

@jmhodges Are you sure the cert has not been updated? In your logs I see that your Target HTTPS proxy was updated with a brand new cert (this is what happens when the contents of your secret change since GCP does not allow cert objects to be updated)

I0422 18:54:42.342515       1 certificates.go:98] Creating new sslCertificate "k8s-ssl-0861999af802e3cc-d2d772c3c9170584--bb860f3113bd5eae" for LB "prod-apps--bb860f3113bd5eae"
I0422 18:54:45.741912       1 target_proxies.go:103] Https proxy "k8s-tps-prod-apps--bb860f3113bd5eae" has the wrong ssl certs, setting [k8s-ssl-0861999af802e3cc-d2d772c3c9170584--bb860f3113bd5eae] overwriting [https://www.googleapis.com/compute/v1/projects/personal-sites-1295/global/sslCertificates/k8s-ssl-0861999af802e3cc-8526cda76e62be92--bb860f3113bd5eae]
I0422 18:54:50.068682       1 certificates.go:273] Cleaning up old SSL Certificate k8s-ssl-0861999af802e3cc-8526cda76e62be92--bb860f3113bd5eae

[jmhodges]

Hunh, what was the date for that? It seems like it took 4 or 5 hours to happen if that was the same date (if that's in UTC) and had previously not updated. That seems outside of an expected SLA?

[jmhodges]

(And yes it has updated now. But that time difference seems very large and had not updated previously! This seems to be stochastic like my previous bugs)

[jmhodges]

You know, what distinguishes this last working but very slow update is that I added a domain to the SAN in the cert.

[rramkumar1]

It seems like it took 4 or 5 hours to happen if that was the same date (if that's in UTC) and had previously not updated

Do you know when exactly you updated the secret? Not sure if that's easy for you to find out but that would help.

[jmhodges]

Ah, looks like I got the time wrong. It was 18:45 UTC on the 22nd. I did toss it through OpenSSL s_client after 10 or 15 minutes, though. Is it possible it didn’t get distributed for a while after that log line? Is it also possible that this update only worked eventually because I updated the domain list?

…

On Thu, Apr 25, 2019 at 2:30 PM Rohit Ramkumar ***@***.***> wrote: It seems like it took 4 or 5 hours to happen if that was the same date (if that's in UTC) and had previously not updated Do you know when exactly you updated the secret? Not sure if that's easy for you to find out but that would help. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#724 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAAEVX56BGY4YIWSNK3C5LPSIPG5ANCNFSM4HFZVJEQ> .

[rramkumar1]

Ok, 18:45 UTC makes perfect sense then. Since we do not watch updates to secrets, your Ingress would only be requeued periodically (which is every 10 mins). Once it was requeued, the cert was updated.

[rramkumar1]

@jmhodges I'm going to go ahead and close this because its WAI. If you have any further questions, feel free to reopen.