Etcd serving TLS is not supported

[scottyeager]

I have a VDC with etcd deployed inside. Locally, I've built the latest 0-stor from source. I can interact with the VDC etcd using etcdctl and I can use 0-stor with a local etcd. However, 0-stor and the VDC etcd will not work together.

I've downloaded the 0-stor config and added the endpoint for my VDC etcd. Attempting to add or check files gives Error: "could not save value: response failed" and Error: "could not load value: response failed", respectively.

Etcd endpoint: "https://scott-etcd-etc.webg1test.grid.tf:2379"

[LeeSmet]

Examining the error with a proper debug build shows:

WARN  zstor_v2::etcd               > Response(Status { code: Internal, message: "Unexpected compression flag: 52, while receiving response with status: 404 Not Found" })

The cause seems to be related to the https proxy in between, we will need to investigate if it is related to webgateway or https proxy in general (e.g. also a problem when just using caddy)

[scottyeager]

Thanks for looking into this, Lee. I wonder if there's any chance of a workaround in the meantime, as I see indications in #2262 that this may have been working under an older configuration of the etcd deployment where manually adding port forwarding was required.

[xmonader]

@scottyeager running a local etcd on your machine and configuring that in the zstor config should help :)

[scottyeager]

Indeed it does, @xmonader, but I'm hoping to see a fully grid based solution working :)

[LeeSmet]

After some debugging, it seems that the problem is that the current library does not automatically use TLS, even when the etcd endpoint is https. After hacking around in the code a bit, I did manage to get it to work by manually providing a TLS Config to the etcd lib, with a downloaded ca_certificate.pem file containing the full certificate chain.

Since etcdctl does work out of the box, we should probably parse the endpoint string for an https endpoint, and if one is found, first connect and download the servers certificate chain.

[LeeSmet]

Some work has been done to hopefully support tls, however I don't really have an env to test this on. I tried to use the config from #17, but that one errors on the unexpected compression flag again. However it seems that even etcdctl has issues with the etcd endpoint in that config, so that's not really a good test case.

[LeeSmet]

Using "https://scott-etcd-etc.webg1test.grid.tf:2379" as etcd endpoint does seem to work out of the box now (I can upload a file), but there seems to be an error downloading the file again. But the tls works now.

[LeeSmet]

Automatic TLS for https endpoints for etcd is now available on master. Note that this relies on the running systems root certificates. In other words, self signed certificates for etcd will not work unless the certificate root is added to the systems trusted certs.