Agent/auto_encrypt "leaf certificate watch fired" erroneously firing every couple seconds

[exo-cedric]

Overview of the Issue

With Agents configured with auto_encrypt along Server configured with Vault provider along Connect CA, Agents renew their leaf certificates every couple seconds or so, despite the notAfter timestamp being 72h in the future.

Reproduction Steps

Steps to reproduce this issue, eg:

1. Provision the Vault's Consul IntermediateCA with max. 2 weeks (336h) TTL
2. Create a cluster with Vault provider enabled as Connect CA
3. Configure the Consul Connect CA with IntermediateCertTTL=336h and RotationPeriod=168h

# consul connect ca get-config
{
	"Provider": "vault",
	"Config": {
		"Address": "https://vault:8200",
		"CAFile": "/etc/consul.d/ssl/vault-cas.pem",
		"IntermediateCertTTL": "336h",
		"IntermediatePKIPath": "pki/consul-connect",
		"LeafCertTTL": "72h",
		"RootPKIPath": "pki/consul",
		"RotationPeriod": "168h",
		"Token": "[[ REDACTED ]]"
	},
	"State": null,
	"CreateIndex": 7051964,
	"ModifyIndex": 10644873
}

(Vault's RootCertTTL - pki/consul CA - is several years)

Consul info for both Client and Server

version 1.8.8 on both Server and Client

Operating system and Environment details

N/A (I believe)

Log Fragments

# journalcrt -u consul -f | grep leaf
Apr 20 09:44:01 consul[1060141]:     {"@level":"debug","@message":"handling a cache update event","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:01.631557Z","correlation_id":"leaf"}
Apr 20 09:44:01 consul[1060141]:     {"@level":"debug","@message":"leaf certificate watch fired - updating TLS certificate","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:01.631676Z"}
Apr 20 09:44:15 consul[1060141]:     {"@level":"debug","@message":"handling a cache update event","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:15.994876Z","correlation_id":"leaf"}
Apr 20 09:44:15 consul[1060141]:     {"@level":"debug","@message":"leaf certificate watch fired - updating TLS certificate","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:15.994970Z"}
Apr 20 09:44:28 consul[1060141]:     {"@level":"debug","@message":"handling a cache update event","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:28.337489Z","correlation_id":"leaf"}
Apr 20 09:44:28 consul[1060141]:     {"@level":"debug","@message":"leaf certificate watch fired - updating TLS certificate","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:28.338255Z"}
Apr 20 09:44:42 consul[1060141]:     {"@level":"debug","@message":"handling a cache update event","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:42.218133Z","correlation_id":"leaf"}
Apr 20 09:44:42 consul[1060141]:     {"@level":"debug","@message":"leaf certificate watch fired - updating TLS certificate","@module":"agent.auto_config","@timestamp":"2021-04-20T09:44:42.218224Z"}
^C
@ 2021-04-20 09:44:46 +0000

# openssl s_client 127.0.0.1:8501 </dev/null 2>/dev/null | openssl x509 -noout -enddate
notAfter=Apr 23 09:44:42 2021 GMT
@ 2021-04-20 09:44:47 +0000

(mark the notAfter timestamp correctly being 72h in the future)

[rboyer]

This was fixed via #9428 in versions 1.9.4, 1.8.9, and 1.7.13

[jsosulska]

@exo-cedric - Please attempt to replicate with 1.8.9, as @rboyer mentioned. I'll hold this issue open for a week in the meantime.

[exo-cedric]

Thanks all for the quick feedback. Currently on to deploying and testing latest 1.8.10. I'll come back to you asap.

[exo-cedric]

I can now confirm issue is fixed using 1.8.10. Thanks for the pointer.