Docker daemon needs restart to handle system ca change (i.e. internal registry)

[cblomart]

Description

We have setup a internal registry which uses a private key infrastucture: the certificate authority is internal.

We have installed the CA in the linux client and reloaded docker configuration with systemd but still get x509: certificate signed by unknown authority

After rebooting the full linux (bazooka solution) this works.

So i guess docker doesn't reload system certificates on reload

Steps to reproduce the issue:

I am on debian.
The hostname and port of the internal registry have been edited.

1. insert your CA into the trusted cert authorities

> sudo cp ca.crt /usr/local/share/ca-certificates/
> sudo update-ca-certificates
> # Check that it added a certificate on the output

2. try a docker login to your favorite internal repository

> docker login registry.local:5000
Username: cblomart
Password:
Error response from daemon: Get https://registry.local:5000/v2/: x509: certificate signed by unknown authority

3. reload docker config

> sudo systemctl reload docker

4. try again a docker login to your favorite internal repository

> docker login registry.local:5000
Username: cblomart
Password:
Error response from daemon: Get https://registry.local:5000/v2/: x509: certificate signed by unknown authority

5. restart docker

> sudo systemctl restart docker

6. try again a docker login to your favorite internal repository

> docker login registry.local:5000
Username: cblomart
Password:

Describe the results you received:

CA is taken into account only after a restart of docker.
Otherwise: x509: certificate signed by unknown authority

Describe the results you expected:

I would have expected that docker takes into account the system certs without reload or restart.
I would have expected that a reload was suffisient to reload the system certificate.
I have an issue restarting my docker daemons to reload certificates for the internal registry

Output of docker version:

Client:
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.6
 Git commit:        74b1e89e8ac6
 Built:             Fri Jul 26 12:00:00 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.6
  Git commit:       74b1e89e8ac6
  Built:            Fri Jul 26 12:00:00 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.1.3_catatonit
  GitCommit:

[AkihiroSuda]

/etc/docker/certs.d should work for you

[cblomart]

Most probably but this require an additional configuration to the docket daemon. We would like to avoid this and rationalize that system certificate (where our CA is needed for other purposes) should be equivalent.

[ddl-denis-parnovskiy]

FYI sudo systemctl reload docker reloads docker.service unit file into systemd. It doesn't do anything with the docker process.

Not a bug: systemctl reload is not supposed to cause docker process to reload its configuration

[mkhpalm]

Ignoring wrappers like systemctl reload that typically send SIGHUP. What about sending signals directly at dockerd like kill -HUP <dockerd pid> to reload system trust stores?

[mkhpalm]

Most probably but this require an additional configuration to the docket daemon. We would like to avoid this and rationalize that system certificate (where our CA is needed for other purposes) should be equivalent.

You can do something like this to get around the inconsistencies in dockerd

mkdir /etc/docker/certs.d/registry.local:5000
ln -s /etc/ssl/certs/ca-certificates.crt /etc/docker/certs.d/registry.local:5000/ca.crt

(good and ugly, but works...)

[thaJeztah]

systemctl reload is not supposed to cause docker process to reload its configuration

Note that (since #22446), systemctl reload docker should send a kill -s HUP dockerd to reload configuration, but I must admit that I don't know if that would reload system certs (if that's handled by the Go runtime itself, it's possible it caches those, and doesn't reload them)

[mkhpalm]

systemctl reload is not supposed to cause docker process to reload its configuration

Note that (since #22446), systemctl reload docker should send a kill -s HUP dockerd to reload configuration, but I must admit that I don't know if that would reload system certs (if that's handled by the Go runtime itself, it's possible it caches those, and doesn't reload them)

It does not reload them. You can reload certs in certs.d but the code treats both differently. The problem with that is neither are "static" so-to-speak files.

[cpuguy83]

Go will not reload the system cert pool since it is populated through a sync.Once.

[beornf]

The TLS server config in Docker daemon is returned with static Certificates and ClientCAs.

moby/cmd/dockerd/daemon.go

Lines 577 to 588 in 76016b8

	tlsOptions := tlsconfig.Options{
	CAFile: cli.Config.CommonTLSOptions.CAFile,
	CertFile: cli.Config.CommonTLSOptions.CertFile,
	KeyFile: cli.Config.CommonTLSOptions.KeyFile,
	ExclusiveRootPools: true,
	}
	if cli.Config.TLSVerify == nil || *cli.Config.TLSVerify {
	// server requires and verifies client's certificate
	tlsOptions.ClientAuth = tls.RequireAndVerifyClientCert
	}
	tlsConfig, err := tlsconfig.Server(tlsOptions)

If the TLS config was extended with dynamic GetCertificate and GetClientCertificate methods it would be possible to reload the certificate files.

[cpuguy83]

That config is not related to registries, it is for auth over a TCP socket for admins.

[thaJeztah]

Yes, so this is the Go runtime that doesn't support it (as it's using a sync.Once, which means it will only load one, but won't refresh after that); https://github.com/golang/go/blob/1868f8296eb15d11c45c2c7c1d373b211f745d84/src/crypto/x509/root.go#L21-L31

I see there's a ticket there with a proposal to provide an API for reloading; golang/go#41888, and a pull request / patch, but it's not been accepted / reviewed yet; https://golang.org/cl/327069