Add/remove certificates using API

[strarsis]

From what I understand, new certificates can be only set up using the fabio properties file.
When adding/removing apps or updating the PEM files, I would find it very helpful when fabio supports management of certificates using its API or even consul tags.

[magiconair]

Managing the certificates via the config file is a stop gap until the entire configuration moves to the backend to have a full cluster configuration. Also, hashicorp has Vault which is built for managing secrets. Both things are on my list of things to work on.

[strarsis]

If I want to use fabio in production now, how can I prevent issues when adding/removing certificates for existing apps?
Spawning up a second fabio container and shutting down the old one (blue/green deployment)?
Is there a config reload command for fabio?

[magiconair]

Interesting use case. Didn't take that into account. Yes, with the current setup you'd just update the configuration and restart which would interrupt existing connections. How often do the certificates change in your setup?

[magiconair]

No, there is no 'reload' command since by design this shouldn't be necessary... We'll find a solution.

[strarsis]

Thank you for your response.

For example: Let's Encrypt (LE) requires certificate renewal in intervals (90 days).
Also apps may be just added, new domains assigned to them,
in any of these cases there would be a small but existing downtime/disruption.

The vault PKI Secret Backend could be used to retrieve the PKI files:
https://www.vaultproject.io/docs/secrets/pki/index.html

For fabio authenticating itself against vault, using an Auth Backend,
the most straightforward for containers/apps would be App ID:
https://www.vaultproject.io/docs/auth/app-id.html

The main concern would be how to offer the configuration
for all these different Auth Backends in the fabio configuration.

As fabio is already opinionated insofar that it uses Consul and its protocol for service discovery,
adding vault as definite backend for obtaining the certificate keys/data would of course force the fabio operator to implement and maintain a vault server, too.

As kind of compromise, one could offer a pure HTTP API, similar to the existing one for adding routes to fabio ("overriding"), to allow assigning certificates to routes on-the-fly.
Then a kind of Vault-HTTP-adapter could be plugged in when vault should be used.

[magiconair]

Fabio will support different backends soon. The one for Google Compute Platform is in the works so there is definitely a tradeoff in using either Vault or some other configuration mechanism. Storing the certificates in the backend is simple at least for consul.

How would you store the certificates in consul? As PEM files without password or as P12 files with password? If you want password protection how does fabio get the password?

[strarsis]

Thank you for your answer.

As vault already handles the authentication of apps against itself in order to retrieve secrets,
wouldn't it be already sufficient to store the actual PEM files without passphrase?

Instad of a cert passphrase, fabio itself would auth against vault with a Auth Backend,
then vault will or will not give out the requested PEM files according to the permissions set for fabio in vault.

[magiconair]

Yes, that is correct if I use Vault for storing secrets. If you don't want that additional dependency then you get that unlock problem.

[strarsis]

So in nginx configuration for example, there is often an extra *_key file provided
which contains the passphrase
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_password_file

For haproxy config, the passphrase is always stripped first, otherwise it will ask for it at each restart.
http://blog.armbruster-it.de/2010/03/remove-the-passphrase-from-a-pkcs12-certificate/
The "password server" mentioned in this thread comes close to describing vault: http://comments.gmane.org/gmane.comp.web.haproxy/14943

It is often recommended that the passphrase is stripped from the cert when it is used in an automated way (like restarting a web server that uses it).
Shouldn't rather vault PKI backend offer an option to automatically strip + offer the unencrypted PEM to the authenticated vault client?

So for the non-vault API, there could be an extra parameter to the passphrase file, if needed,
when vault is used, a cert without passphrase is expected to be given to fabio.

Concerning the Auth Backend for fabio against vault:
With an app authenticating against vault, a token is generated by vault
and given back that then can be used for retrieving secrets from vault.

One way could be offering some kind of pre-auth script
so the admin is free to any use authentication backend of vault
which has to return a token to be used by fabio, each time when a cert has to be loaded or updated.
That pre-auth script should be ran as same user / in some system/container as vault itself.

[magiconair]

Storing the passphrase next to the cert isn't any benefit at all. You might as well just use the unlocked files. So the question then becomes how secure is your backend registry or the servers you're deploying to. I'm leaning towards this:

• fetch PEM files from URL (registry or otherwise). Retrieval via HTTPS would be nice but that's not how consul is usually deployed. least secure option. For dev or in trusted networks.
• fetch PKCS#12 files from URL (registry or otherwise) and provide passphrase during fabio startup either via properties file or on the command line. All certs would then require the same passphrase. Since someone who has access to the machine could also dump the memory to get to the passphrase properties should be good enough.
• use Vault which most likely also requires a key for fabio to unlock the vault

[strarsis]

I am really excited by this feature, together with HTTP2 support by Go 1.6 seeming soon coming out,
I can finally use fabio as elastic HTTP2/1 reverse proxy solution.

There is a similar reverse HTTP proxy, https://github.com/EmileVauge/traefik,
which from what I can find out about in its documentation,
only offers assigning certificate files via its config file,
it doesn't seem to be planned to implement dynamic addition/update during runtime,
neither vault backend support - so fabio would really have its edge here.

[strarsis]

Further thoughts:
Certs are assigned to common names which in turn are used by routes (route-prefixes).
domain1.com -> cert1.pem.
domain2.com -> cert2.pem

route1: domain1.com/whatever -> uses cert for domain1.com
route2: domain1.com/abc -> uses cert for domain1.com
route3: domain2.com/123 -> uses cert for domain2.com

Setting a cert that for an existing common name would override it (=reload),
this is necessary for certificate renewal.

1. Fabio API: Set certificate for CN domain1.com
2. Consul informs fabio about new service with prefix which uses CN domain1.com
   The other way around would mean that fabio would first serve only HTTP,
   when the certificate becomes available would start serving HTTPS, too.

[strarsis]

OK, so Traefik reads the CN of each passed certificate file
to determine automatically to what domains (and their routes) it applies:
https://github.com/emilevauge/traefik/blob/4e9ff45747f9172809065fb63c6d0baca782dad1/integration/fixtures/https/https_sni.toml

Something similar with HTTP API would be great for fabio.

[magiconair]

ok, so I can add multiple certificates to a single listener which are either statically or dynamically configured. This isn't really a code issue but a config issue. I need to think about this a bit more how I express this in the configuration. Once I've figured that out the implementation should be straightforward.

[strarsis]

Addition: It may be a good starting point when for the beginning a folder of directories can be passed
and fabio determines the right cert/key/chain files to use by CNAME (SNI).
This would allow using letsencrypt-auto with fabio.

[magiconair]

@strarsis I've started working on this.

[magiconair]

Current plan is to support a configurable certificate store per listener. A certificate store is either a fixed set of certificates, a directory with a list of certificates which is reloaded whenever something changes, consul which is again updated when something changes and vault. Since consul already offers a secured HTTP API lets first get this working and then see whether another API is necessary.

[magiconair]

@strarsis fabio will support multiple different certificate sources like file, path, http, consul and vault. You can test this in the cert-source branch. SNI is supported.

Since they all have different provisioning mechanisms I find it difficult to come up with an API that will work in all cases. Also, some of the tools already provide HTTP APIs which can be used for provisioning. So I chose not to provide an API in fabio but to make the integration with different types of certificate sources easy.

file   - path to a cert/key file like in the current configuration
path   - path to a directory with cert/key files, refreshed periodically (provision with puppet, salt, ansible)
http   - url to a directory with cert/key files, refreshed periodically (provision with puppet, git, ...)
consul - url to KV store, refreshed automatically 
vault  - url to secret list, refreshed periodically

[rokka-n]

@magiconair Are you considering letsencrypt integration?

[magiconair]

Yes Frank Schröder

…

On 2. Apr 2017, at 01:39, Roman Naumenko ***@***.***> wrote: @magiconair Are you considering letsencrypt integration? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tino]

Is there any work being done on letsencrypt integration? I've started it as my first Go project this weekend, in the form of a separate binary that implements copies the registry code (and a lot more code ☺️) from here and generates certs for each domain if they don't exist yet. I envision a file and Vault store, that fabio then can use.

Once I get it roughly working I'll push it to a repo. Still got to think about reusing the urlprefix-mydomain.com/ tag, or going with a new letsencrypt-mydomain.com tag though. Think the latter would be cleaner.

[magiconair]

@tino Nobody has been picking this up yet and if you are working on a PR then this would be very welcome. You can integrate that directly with fabio. @pschultz has done preliminary work for these types of cert stores in #135/#315. He has provided a proof-of-concept implementation for ACME which you can see here: https://gist.github.com/pschultz/80a13aa3488f4e9afa1e440bea3c3d6e

This still needs config, wiring and testing but it is a start. I suggest you open an issue for that and submit a PR. This would also prevent morphing this ticket into something else.

[magiconair]

@tino There should be no need to add a separate tag name for that. The only thing that is required is to fetch or generate a certificate on the fly. Keep in mind that fabio can be run in clusters so you need a place to store the certs, e.g. consul.

[ont]

@magiconair what is current state of fabio and letsencrypt integration? mholt/caddy has letsencrypt integration but miss consul auto configuration. Fabio has autoconfiguration but miss letsencrypt auto-certificates. And traefik has very strange configs and bugs...

[aaronhurt]

@ont it's not currently supported IN fabio but there are ways of getting it done. Vault is supported as a certificate store for fabio and there are third party projects that will handle renew of lets encrypt and push to vault (https://github.com/ketchoop/letsencrypt-to-vault). There is also support for reading/refreshing certificates directly from the filesystem builtin to fabio so any system that writes letsencrypt certs to disk should work as well.

[ont]

@leprechau yes, I understand it. But main benefit from integrating letsencrypt into fabio is to avoid management hell with webroot/.well-known dirs for each service which need SSL support. And some services expose no webroot at all, only their api routes at http endpoint.

[aaronhurt]

@ont Understood. I was just trying to offer options until someone has time to work on full integration of letsencrypt with fabio.