net/url: add URL.Redacted to return password-free string

[nrxr]

Hiding the password from an URL is very useful for logging purposes. I have seen in the past code for doing this (or just plain passwords in logs). I built a small helper for this and I use it in my projects but I think it makes sense to have it in the standard library.

I wrote a small PR for this #34686 with the code (and test suite) for this feature.

It's a simple derivation from URL.String() that masks the password if exists from the string being passed. It makes no modification at all to the URL itself but to a copy of it. URL.Redacted() is just filtering passwords out of URL.String().

For example https://user@host.tld would be https://user@host.tld but https://user:password@host.tld would be https://user:xxxxx@host.tld. Making obvious that a password has been masked is good for visual-debugging purposes, so it's known a password is being passed in the URL.

[ianlancetaylor]

Do people still use passwords in URLs?

There are many different kinds of data in URLs. Some URL parameters may be sensitive, so why single out passwords?

The helper package is small, and seems to work well. Is this really widely used enough to add to the standard library? https://golang.org/doc/faq#x_in_std

[nrxr]

@ianlancetaylor Hi!

Yes, people still does. Specially when connecting to other servers, not over HTTP. Here's a list from the top of my head of services to which I connect my go apps and have passwords in its URLs: postgresql, mongo, rabbitmq and mysql.

I singled out passwords because the use case I can see for this is logging purposes, mainly. I've been a DevOps at a pair of fairly large companies and saw lots of passwords in our logs.

I thought we were alone in this until I listened to Go Time's podcast on security and actually one of the things mentioned were hiding passwords from logs. That's when I saw it was not just me and hence why I proposed it here.

Widely used: no. Should be widely used? Probably.

Huge point against the inclusion of this: there's no other programming language that includes such method in their standard library.

[rsc]

I don't believe Mask is the right name. I thought it was about IP masking when I first saw the issue.
cmd/go uses a function like this called Redacted.
We could expose that code under that name if needed.
One nice thing about the API we designed for cmd/go is that Redacted returns a string, not a *url.URL. That makes it clearer that the result is for printing and not for future direct use in an HTTP request.

/cc @bcmills

[nrxr]

@rsc this is the case for Mask as well, it returns a string. I like Redacted better, indeed.

[rsc]

I retitled the issue to use the Redacted name.
Does anyone object to adding this functionality?

[rsc]

Everyone seems in favor of this, so it seems like a likely accept.
Leaving open for a week for final comments.

[bvisness]

As a small detail, you might also want to add an analogous Redacted method to the UserInfo type from net/url.

I think if you are handling passwords you might as well provide a way to easily hide them, but to @ianlancetaylor's point, the user password field is not the only thing you may want to redact from a URL, so perhaps a Redact method on the base URL type should also be able to redact query parameters, or even parts of the path. But those options might just add too much complexity to be worth it.

I certainly wouldn't want anyone to think they were safe just because they called Redacted, but didn't actually use plaintext passwords in this way.

[bvisness]

Given that the func Redacted used internally by cmd/go also only redacts the password, I have put my thumbs-up on this, but I think it's worth thinking about whether a method in the standard library would need more flexibility.

[rsc]

@bvisness, if you need a super-fancy redactor that can strip things from URL parameters, probably you should reach for a third-party package. Or maybe think about not putting authentication information into URL parameters, where it leaks via Referer and other headers.

No change in consensus, so accepting.

[nrxr]

@ianlancetaylor @rsc Hi! What’s the fix needed and I’ll write it in a PR tonight.

[ianlancetaylor]

No rush, we are in the release freeze for 1.14 right now.

[nrxr]

Already understood what NeedsFix means (if somehow you got here because of a search on that sentence, here's the meaning of NeedsFix in this context: https://golang.org/doc/contribute.html#check_tracker).

Already created a PR with the implementation I think fulfills this need.

[gopherbot]

Change https://golang.org/cl/207082 mentions this issue: net/url: add URL.Redacted() to return password-free string

[odeke-em]

Awesome, thank you @nrxr and great to have you contributing to and using Go!
I’ve reviewed your CL and added some suggestions, please take a look. Thanks.

[odeke-em]

@oiooj @bcmills @jayconrod, turns out there is a parallel issue here that #37873 would benefit from when CL https://golang.org/cl/207082 is merged, and we won’t worry about percent encoring of [].

We could then perhaps make an edit to remove usages of cmd/go/internal/web.Redacted(), in favor of (*url.URL).Redacted()

[nrxr]

@odeke-em Hi! Sorry for the delay; couple of busy weeks. I got around and wrote the suggestions you made and sent them to gerrit.