v2: Implement 'pki' app powered by Smallstep for localhost certificates #3125

mholt · 2020-03-07T20:07:18Z

This pull request adds support for localhost certificates. It creates a new App module called pki and a new Issuer module called internal.

In Caddy v1, users would write tls self_signed in the Caddyfile (#2502) to have Caddy generate a quick, self-signed certificate for use in local/dev environments. But the cert was only stored in memory, so its key had to continually be re-trusted every time you restarted Caddy. It was a quick and bad implementation.

In Caddy 2, we can do better than that. These days, internal/local certificates can be very useful when done properly, and there is now tooling in Go to make this possible.

This design implements a proper, fully-managed, self-contained PKI (Public Key Infrastructure) system. Caddy generates a root and intermediate certificate, stores the root key offline, and uses the intermediate to sign leaf certificates. The intermediate is short-lived by default (7 days) and the leaf certificates are even shorter (12 hours -- maybe shorter before long).

Features:

Generated roots can be installed into the system trust store automatically (with authorization). We might even do this by default if the Caddyfile is used, since we expect most developers will want this.
Intermediates get renewed automatically (and soon roots will too, but their default lifetime is currently 10 years, so we'll get around to this later)
Command to uninstall root certificates (caddy untrust)

mkcert is a tool by @FiloSottile which generates certificates and installs them into the root stores for you. It's really popular because it's so useful in local dev environments. This PR is basically mkcert, but with short cert lifetimes, proper renewals, and totally automatic and integrated into your web server! Think of this like a "mkcert server" and powered by Smallstep.

Configuration

The minimal required configuration to add to your Caddy JSON is an automation policy that uses an "internal" issuer:

{
	"issuer": {
		"module": "internal"
	}
}

That will assume a CA named "local". You can define and customize CAs, including the default "local" CA with the pki app:

{
	"apps": {
		"pki": {
			"certificate_authorities": {
				"local": {
					"install_trust": true
				}
			}
		}
	},
	...
}

That config will install the root into your system trust store. Then:

Of course, this can be used for any hostname or IP address, not just localhost. We expect most users will want to use *.localhost as well.

WORK IN PROGRESS

This is not a finished PR, but please try it out and submit your feedback! Lots of TODOs to implement, and Caddyfile support is yet to come.

A later PR will add support for running an ACME server.

cf. #3021

/cc @mmalone @maraino -- I will have specific questions for your review sooner or later. Thanks!

mholt · 2020-03-08T00:17:49Z

modules/caddytls/internalissuer.go

+	// TODO: eliminate placeholders / needless values
+	cfg := &authority.Config{
+		Address:          "placeholder_Address:1",
+		Root:             []string{"placeholder_Root"},
+		IntermediateCert: "placeholder_IntermediateCert",
+		IntermediateKey:  "placeholder_IntermediateKey",
+		DNSNames:         []string{"placeholder_DNSNames"},
+		AuthorityConfig: &authority.AuthConfig{
+			Provisioners: provisioner.List{},
+		},
+	}


@mmalone @maraino Do you know if there's any chance we can drop these placeholders/meaningless values?

mholt · 2020-03-08T00:19:03Z

modules/caddytls/internalissuer.go

+// TODO: borrowing from https://github.com/smallstep/certificates/blob/806abb6232a5691198b891d76b9898ea7f269da0/authority/provisioner/sign_options.go#L191-L211
+// as per https://github.com/smallstep/certificates/issues/198.
+// profileDefaultDuration is a wrapper against x509util.WithOption to conform
+// the SignOption interface.
+type profileDefaultDuration time.Duration
+
+// TODO: is there a better way to set cert lifetimes than copying from the smallstep libs?
+func (d profileDefaultDuration) Option(so provisioner.Options) x509util.WithOption {
+	var backdate time.Duration
+	notBefore := so.NotBefore.Time()
+	if notBefore.IsZero() {
+		notBefore = time.Now().Truncate(time.Second)
+		backdate = -1 * so.Backdate
+	}
+	notAfter := so.NotAfter.RelativeTime(notBefore)
+	return func(p x509util.Profile) error {
+		fn := x509util.WithNotBeforeAfterDuration(notBefore, notAfter, time.Duration(d))
+		if err := fn(p); err != nil {
+			return err
+		}
+		crt := p.Subject()
+		crt.NotBefore = crt.NotBefore.Add(backdate)
+		return nil
+	}
+}


@mmalone @maraino It would be nice if there was an easier way to simply specify the certificate lifetime. I just feel kinda not-awesome copying unexported code out of your repo, especially for keeping up with changes. Open to discussing ideas here!

mholt · 2020-03-08T00:23:08Z

modules/caddypki/crypto.go

+	"strings"
+)
+
+func pemDecodeSingleCert(pemDER []byte) (*x509.Certificate, error) {


@mmalone @maraino Would you be willing to add exported versions of these functions (the ones in this file here) in your x509util package? At least pemDecodeSingleCert, pemEncodePrivateKey, and pemDecodePrivateKey? (Or maybe they do exist and I just glossed over them)

mholt · 2020-03-08T02:29:13Z

This is pretty close to being done.

Some remaining questions pertain mostly to automatic HTTPS. As the PR currently stands, localhost and other non-public-qualifying subjects won't automatically be served over HTTPS. Which is fine for now, but...

I intend for all sites -- including localhost/dev ones -- to use HTTPS by default. This is just a bit tricky in practice.

Most dev machines can't conveniently bind to low ports (80 and 443)
It's not uncommon for production configs to have both public-facing names (e.g. example.com) mixed with local-bound ones (e.g. 127.0.0.1:5000, for internal services). The public-facing name would need to use a public CA and the IP address would use Caddy's CA, but that could potentially cause trust issues if accessed from another machine...

These are the two main pain points I haven't resolved yet.

More visually, what behavior results from these Caddyfiles:

localhost

# dev site
...

example.com {
   # production site
    ...
}

10.0.1.3:5000 {
   # internal service; and what if the port was omitted?
   ...
}

We'll have the ability to serve all these over HTTPS, but in the first case, binding to port 443 might be problematic. In the second case, using a non-publicly trusted cert means that any client accessing it must trust Caddy's root -- not a big deal, but just not something that is automatic yet.

Oh, and what about this:

:5000
...

?

xdevs23 · 2020-03-09T16:57:19Z

I'd suggest use tls by default only on hostnames without a port, i. e. localhost, foo.bar.localhost, foo.bar, ...

Don't use tls by default on localhost:5050, foo.bar:8080, 127.0.0.1, 0.0.0.0:8088, :5000, ...

Only exception would be for the port 443 in which case I'd expect a HTTPs endpoint.

Of course a tls self_signed (or similar) could still enable TLS regardless of the default.

mholt · 2020-03-09T17:43:10Z

@xdevs23 Thanks for your input! I don't think that's going to fly though, because most dev environments can't (freely) bind to low ports, so defaulting to low ports for internal/localhost names will usually result in an error.

I think this is what I'll end up doing for now:

Keep public-looking hosts (e.g. example.com) the same in terms of auto HTTPS
Make HTTPS opt-in for non-public-looking hosts (e.g. localhost or 127.0.0.1); to opt-in:
- Use tls internal directive,
- Or specify https:// (with or without port -- as a special case, port 443 implies https://)
- Or specify a global option to use the internal issuer, maybe
ACME and internal issuers cannot be mixed in a site block
- If https://localhost and example.com are specified for the same site block, use local issuer, I guess...

xdevs23 · 2020-03-09T17:50:33Z

because most dev environments can't (freely) bind to low ports, so defaulting to low ports for internal/localhost names will usually result in an error

Hasn't this always been the case? Using automatic tls only changes the bound ports from just 80 to 80 and 443. Also, since there is a Docker image of Caddy v2, I think that might be the best solution overall and would also solve this issue.

mholt · 2020-03-09T17:56:33Z

Hasn't this always been the case?

Yes, and no, because automatic HTTPS never triggered in dev environments by default. If anyone wanted to use a public DNS name in a local dev environment, they'd turn off automatic HTTPS using http:// or port :80 unless they're set up to get a public cert for the name on their local environment.

Using automatic tls only changes the bound ports from just 80 to 80 and 443.

That is so untrue. Automatic HTTPS does a lot more to work properly: https://caddyserver.com/docs/automatic-https#effects - and the default port is 2015, not 80 (because of the low port permissions problem). This has always been the case, since Caddy v0.5.0 back in early 2015.

Also, since there is a Docker image of Caddy v2, I think that might be the best solution overall and would also solve this issue.

I don't think Docker has anything to do with it.

xdevs23 · 2020-03-09T18:01:30Z

That is so untrue. Automatic HTTPS does a lot more to work properly: https://caddyserver.com/docs/automatic-https#effects - and the default port is 2015, not 80 (because of the low port permissions problem). This has always been the case, since Caddy v0.5.0 back in early 2015.

I got something mixed up then, my bad.

I don't think Docker has anything to do with it.

Docker has the permission to bind low ports which means Caddy can do that, too, as long as Caddy itself inside the Docker container has that permission, too.

I think you're right — binding 80 and 443 by default for non-public domains is not a good idea. Perhaps a global directive explicitly allowing this could be the solution – or tls on.

mholt · 2020-03-09T18:09:40Z

In the case of Docker or any other environment where you are able to bind to low ports, you can certainly opt-in to the automatic HTTPS for an internal name by omitting the port and using https://:

https://localhost
localhost:443
https://localhost:5000

or

localhost:5000

tls internal

... I think, anyway. That's what I'm working on today.

xdevs23 · 2020-03-09T18:12:17Z

I think the https:// approach might be the best in this case.

mholt · 2020-03-09T18:15:27Z

Well, my point is that both methods are valid; tls internal will be needed if the site address looks like a public DNS name but isn't (or when you want to use an internal cert anyway).

xdevs23 · 2020-03-09T18:17:09Z

Yeah, I think that's a good approach.

mholt · 2020-03-12T22:38:52Z

@mmalone @maraino Ready for review! Will probably merge this soon.

mholt · 2020-03-13T04:24:27Z

Wow our CI tests disappeared

francislavoie · 2020-03-13T15:47:13Z

Woohoo! CI is back 🎉

mholt · 2020-03-13T15:47:35Z

@mohammed90 I got our CI back, but it's still broken... I had to delete the old pipelines and re-make it... any chance you could help? 😅

go.mod

mholt · 2020-03-13T15:55:26Z

Gah, I pushed a small commit and the checks disappeared again, now the only check is CLA Assistant again... 😓

mohammed90 · 2020-03-13T16:12:26Z

Let's try adding PR trigger to the azure-pipelines.yml file. At the top, add this:

pr:
- v2

mholt · 2020-03-13T16:27:09Z

@mohammed90 Hm, but we didn't have to do that before, right?

Anyway, we can try it if you want, but in a separate PR / let's chat on Slack -- I'm gonna merge this up for now. Thanks!

mholt · 2020-03-13T17:05:42Z

There is one question as to the auto-redirects on port 80. If a user can't bind to low ports, they can just specify higher ones in their config, but auto-redirects still happen from port 80 (unless the http_port is changed). I suppose that's fine for now, but I bet we're gonna get questions about this. 👌

Bawa75 · 2023-12-26T12:43:23Z

Hello, finally you have found a way to launch https://localhost on the current V2. If yes please help me

Bawa75 · 2023-12-26T12:44:57Z

Hello @mholt , finally you have found a way to launch https://localhost on the current V2. If yes please help me

mohammed90 · 2023-12-26T12:44:58Z

Hello, finally you have found a way to launch https://localhost on the current V2. If yes please help me

Thanks for your question, and we're thrilled that you're using Caddy! This looks more like a question about how to use Caddy rather than a bug report or feature request. Since this issue tracker is reserved for actionable development items, I'm going to close this, but we have a community forum where more people will be exposed to your question, including people who may be more expert or experienced with the specific question you're facing. I hope you'll ask your question there, and thanks for understanding!

pki: Initial commit of PKI app (WIP) (see #2502 and #3021)

03499f6

mholt added in progress 🏃‍♂️ Being actively worked on v2 labels Mar 7, 2020

mholt added this to the v2.0.0-beta.16 milestone Mar 7, 2020

mholt self-assigned this Mar 7, 2020

This was referenced Mar 7, 2020

Tls self_signed - Not working correctly since a few versions #2502

Closed

Caddy+Smallstep integration (trusted certs for localhost and internal sites) #3021

Closed

pki: Ability to use root/intermediates, and sign with root

a0d3781

mholt commented Mar 8, 2020

View reviewed changes

mholt added 2 commits March 7, 2020 17:24

pki: Fix benign misnamings left over from copy+paste

4e6bfdd

pki: Only install root if not already trusted

f0f5490

mholt mentioned this pull request Mar 10, 2020

Critical: Nondeterministic 100% CPU spin during init() causes programs to hang census-instrumentation/opencensus-go#1200

Closed

mholt added 2 commits March 12, 2020 16:34

Make HTTPS port the default; all names use auto-HTTPS; bug fixes

6d9b8ad

Merge branch 'v2' into pki

06d9607

mholt marked this pull request as ready for review March 12, 2020 22:35

mholt mentioned this pull request Mar 12, 2020

v2: Issues when reloading config while using a custom tls certificate #3138

Closed

Fix build - what happened to our CI tests??

f76b7f2

mholt closed this Mar 13, 2020

mholt reopened this Mar 13, 2020

mohammed90 reviewed Mar 13, 2020

View reviewed changes

go.mod Outdated Show resolved Hide resolved

Fix go.mod

b2ef3f8

mholt mentioned this pull request Mar 13, 2020

caddyhttp: Fix terms error by provisioning new ACMEIssuer #3129

Closed

mholt merged commit 5a19db5 into v2 Mar 13, 2020

mholt deleted the pki branch March 13, 2020 17:12

mholt removed the in progress 🏃‍♂️ Being actively worked on label Mar 13, 2020

caddyserver locked as resolved and limited conversation to collaborators Dec 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2: Implement 'pki' app powered by Smallstep for localhost certificates #3125

v2: Implement 'pki' app powered by Smallstep for localhost certificates #3125

mholt commented Mar 7, 2020 •

edited

Loading

mholt Mar 8, 2020

mholt Mar 8, 2020

mholt Mar 8, 2020

mholt commented Mar 8, 2020 •

edited

Loading

xdevs23 commented Mar 9, 2020 •

edited

Loading

mholt commented Mar 9, 2020 •

edited

Loading

xdevs23 commented Mar 9, 2020

mholt commented Mar 9, 2020

xdevs23 commented Mar 9, 2020

mholt commented Mar 9, 2020 •

edited

Loading

xdevs23 commented Mar 9, 2020

mholt commented Mar 9, 2020

xdevs23 commented Mar 9, 2020

mholt commented Mar 12, 2020

mholt commented Mar 13, 2020

francislavoie commented Mar 13, 2020

mholt commented Mar 13, 2020

mholt commented Mar 13, 2020

mohammed90 commented Mar 13, 2020 •

edited

Loading

mholt commented Mar 13, 2020

mholt commented Mar 13, 2020

Bawa75 commented Dec 26, 2023

Bawa75 commented Dec 26, 2023

mohammed90 commented Dec 26, 2023

v2: Implement 'pki' app powered by Smallstep for localhost certificates #3125

v2: Implement 'pki' app powered by Smallstep for localhost certificates #3125

Conversation

mholt commented Mar 7, 2020 • edited Loading

Configuration

WORK IN PROGRESS

mholt Mar 8, 2020

Choose a reason for hiding this comment

mholt Mar 8, 2020

Choose a reason for hiding this comment

mholt Mar 8, 2020

Choose a reason for hiding this comment

mholt commented Mar 8, 2020 • edited Loading

xdevs23 commented Mar 9, 2020 • edited Loading

mholt commented Mar 9, 2020 • edited Loading

xdevs23 commented Mar 9, 2020

mholt commented Mar 9, 2020

xdevs23 commented Mar 9, 2020

mholt commented Mar 9, 2020 • edited Loading

xdevs23 commented Mar 9, 2020

mholt commented Mar 9, 2020

xdevs23 commented Mar 9, 2020

mholt commented Mar 12, 2020

mholt commented Mar 13, 2020

francislavoie commented Mar 13, 2020

mholt commented Mar 13, 2020

mholt commented Mar 13, 2020

mohammed90 commented Mar 13, 2020 • edited Loading

mholt commented Mar 13, 2020

mholt commented Mar 13, 2020

Bawa75 commented Dec 26, 2023

Bawa75 commented Dec 26, 2023

mohammed90 commented Dec 26, 2023

mholt commented Mar 7, 2020 •

edited

Loading

mholt commented Mar 8, 2020 •

edited

Loading

xdevs23 commented Mar 9, 2020 •

edited

Loading

mholt commented Mar 9, 2020 •

edited

Loading

mholt commented Mar 9, 2020 •

edited

Loading

mohammed90 commented Mar 13, 2020 •

edited

Loading