New puppetserver CA certs causes backup failures

[jflorian]

I've had a long stretch of success with this module but I believe I've uncovered a bug.So what changed? I rebuilt my Puppet Server, going from puppetserver-6.16.1-1.el7 on CentOS 7 to puppetserver-7.4.1-1.el8 on CentOS 8 Stream. I did not try to preserve my Puppet DB content nor any of the old certificates. New DB, new CA. That meant wiping the ssldir of each puppet client, new CSRs and signing. Nice and clean and not too much work because I only have a couple dozen clients. The module rebuilt all the expected configs from the exported resources like normal. I was therefore a bit surprised to see this morning that all my backup jobs failed overnight since this migration. Each job failed with:

15-Oct 03:05 scooby-f34.doubledog.org-dir JobId 2619: Start Backup JobId 2619, Job=Backup_home.2021-10-15_03.05.00_04
15-Oct 03:05 scooby-f34.doubledog.org-dir JobId 2619: Error: tls.c:87 CA certificate is self signed. With OpenSSL 1.1, enforce basicConstraints = CA:true in the certificate creation to avoid this issue
15-Oct 03:05 scooby-f34.doubledog.org-dir JobId 2619: Error: tls.c:92 Error with certificate at depth: 2, issuer = /CN=Puppet Root CA: ebfc05a5e5ff83, subject = /CN=Puppet Root CA: ebfc05a5e5ff83, ERR=19:self signed certificate in certificate chain
15-Oct 03:05 scooby-f34.doubledog.org-dir JobId 2619: Error: openssl.c:81 Connect failure: ERR=error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
15-Oct 03:05 scooby-f34.doubledog.org-dir JobId 2619: Fatal error: TLS negotiation failed with SD at "scooby-f34.doubledog.org:9103"

Examining the certificates in /etc/bacula/ssl on scooby-f34 (my director, running on Fedora 34 with puppet-7.9.0-1.fc34 and bacula-director-11.0.5-1.fc34) didn't reveal anything and the following also looked good (as run on scooby-f34):

cd /etc/bacula/ssl/
openssl verify -CAfile ca.pem scooby-f34.doubledog.org_cert.pem
scooby-f34.doubledog.org_cert.pem: OK

Digging further, I discovered the bacula-fd.service did not get restarted on the clients, nor did the bacula-sd.service.

[jflorian]

I was going to put together a PR for this and couldn't figure out how/where these certs/key were getting deployed by the module so that I could add the appropriate notifiy. Since I found nothing here, I went back to my code where I bring in this module and that's when I confirmed that this was a bug in my code as it's apparently my responsibility to deploy these. So... I don't believe there's any bug here at all. I guess that's the mark of a great module when you completely forget how it works because it just works.

[smortex]

Since I found nothing here, I went back to my code where I bring in this module and that's when I confirmed that this was a bug in my code as it's apparently my responsibility to deploy these.

I think it was half automatic at some point, but fragile so we had to rework it extensively 😉

Changing the transport certs should not hurt, but if you encrypted the data, beware! Changing the so called "PKI" certs will silently prevent you from restoring 😨 Testing advised!

https://github.com/xaque208/puppet-bacula/wiki/PKI-Setup

[jflorian]

@smortex Thanks for that heads up. This was only transport, but I tested a restore of a backup before and after the switchover just to be sure.