proposal: net/http: StripPrefix for path patterns

[egtann]

Proposal Details

With the upcoming wildcard prefixes arriving in the stdlib Mux in Go 1.22, it makes sense to revisit some functions in the stdlib to ensure they're compatible with the change.

http.StripPrefix does not currently support the newly added wildcards. This means using any wildcard in a path will break routing when used with http.StripPrefix. As one example:

r.Handle("/{username}/*", http.StripPrefix("/{username}", userHandler)) // Does not work

There are two clear paths forward:

• Add a new function which supports wildcards, StripWildcardPrefix, or
• Modify the existing StripPrefix to support wildcards

I propose we modify http.StripPrefix, since the performance in stripping wildcards will not have any impact on existing, non-wildcard paths.

Either option is better than exists today, with every project that wants both wildcards and StripPrefix needing to roll their own implementation, despite there being a StripPrefix in the standard library.

I've created an example implementation with a few tests and benchmarks here: https://github.com/egtann/strip-wildcard-prefix

If no action is taken, the documentation for http.StripPrefix should still be updated to clearly state that it does not strip wildcards.

Updates

We can add support for this to http.StripPrefix without any impact for existing, non-wildcard paths by simply using the original http.StripPrefix implementation if the prefix has no wildcards (which is known at initialization, not when routing). I've adjusted the text above to clarify.

[seankhliao]

if you strip the wildcard, should the value still be available in pathvalue?

and if you're already using wildcards, why does your handler care about the path?

[egtann]

The existing behavior of http.StripPrefix copies the entire Request and only modifies the path. I think maintaining the existing behavior in this version results in fewer surprises, so yes, the stripped wildcard should still be available in PathValue. It's also very useful to have access to it in sub-handlers.

[adonovan]

@jba

[jba]

I'm with @seankhliao on this. I don't see the usefulness. If we did do it, we'd have to keep access to the stripped wildcard values, as @egtann points out, but that puts the http.Request into an odd state—wildcard values with no corresponding wildcard in the path. (Admittedly, you could also get to that state with SetPathValue.)

@egtann, can you motivate this with two or three examples?

[egtann]

Sure, the big use-case is prefixing a path with some customer-specific identifier. Apps which allow a user to access different company accounts with a single login may employ a similar approach, so the URL itself is tied to a particular company when shared.

Here are some large (non-Go) examples. Shopify does this:

/store/{storeSlug}/...

Each handler needs access to that slug to do its job; it can't query the database for "all products" unless it knows "for which store?"

So if I hypothetically had Shopify routing set up in Go, it might use a top-level router to apply auth centrally, then have many sub-routers attached at different paths:

r.Handle("/store/{storeSlug}", http.StripPrefix("/store/{storeSlug}", storeHandler))
r.Handle("/settings", http.StripPrefix("/settings", userSettingsHandler))

// Then inside storeHandler:
sh.Handle("/settings", http.StripPrefix("/settings", storeSettingsHandler))
...

Gmail uses the same pattern in its URLs:

/mail/u/{intToIdentifyTheUser}/...

There are some workarounds that come to mind:

• Build a middleware which tosses things into context.Values.
• Pass the entire path to every sub-handler, and don't strip it at all.
• Roll your own replacement function for StripPrefix supporting this behavior.

All of these workarounds are fine. It just seemed like a surprising limitation of http.StripPrefix to need to build around given wildcard support is being added to the stdlib in other areas.

As just my own experience (without ever having looked into the StripPrefix implementation before), I naively expected it to strip a prefix in this way, and it took a while debugging to uncover the reason why my routes were responding with 404s. I've been coding professionally in Go for many years, so while it was a face-palm moment for me, I have to imagine I won't be the only one to hit this after wildcards land in 1.22.

[jba]

Thanks for the examples, and for explaining exactly what your concern is.

I think the workarounds are fine, especially the second—don't use StripPrefix here at all. It seems to create more work than whatever benefit it provides.

But your point about expectations is well taken. We should add doc to StripPrefix that explains that it doesn't work with wildcards. It already says "the prefix must match exactly," so perhaps a bit more verbiage at that point would suffice.

[rsc]

StripPrefix is about constant string prefixes, not about paths and patterns. Leaving it alone is the right path forward.

[rsc]

This proposal has been declined as infeasible.
— rsc for the proposal review group

[covrom]

@rsc maybe you can add something like this:

func StripSegments(pat string, h http.Handler) http.Handler {
	wilds := wildcards(pat)
	if len(wilds) == 0 {
		return h
	}
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		p := stripToLastSlash(r.URL.Path, len(wilds))

		// fmt.Println("strip", r.URL.Path, "to", p)

		rp := stripToLastSlash(r.URL.RawPath, len(wilds))
		if len(p) < len(r.URL.Path) && (r.URL.RawPath == "" || len(rp) < len(r.URL.RawPath)) {
			r2 := (&http.Request{
				Method:           r.Method,
				Proto:            r.Proto,
				ProtoMajor:       r.ProtoMajor,
				ProtoMinor:       r.ProtoMinor,
				Header:           r.Header,
				Body:             r.Body,
				GetBody:          r.GetBody,
				ContentLength:    r.ContentLength,
				TransferEncoding: r.TransferEncoding,
				Close:            r.Close,
				Host:             r.Host,
				Form:             r.Form,
				PostForm:         r.PostForm,
				MultipartForm:    r.MultipartForm,
				Trailer:          r.Trailer,
				RemoteAddr:       r.RemoteAddr,
				RequestURI:       r.RequestURI,
				TLS:              r.TLS,
				Cancel:           r.Cancel,
				Response:         r.Response,
			}).WithContext(r.Context())

			r2.URL = new(url.URL)
			*r2.URL = *r.URL
			r2.URL.Path = p
			r2.URL.RawPath = rp

			for _, ws := range wilds {
				if ws == "" {
					continue
				}
				r2.SetPathValue(ws, r.PathValue(ws))
			}

			h.ServeHTTP(w, r2)
		} else {
			http.NotFound(w, r)
		}
	})
}

func wildcards(s string) []string {
	var wilds []string

	for len(s) > 0 {
		idx := strings.IndexRune(s, '/')
		if idx < 0 {
			if ws := toWildcard(s); ws != "" {
				wilds = append(wilds, ws)
			}
			break
		}
		wilds = append(wilds, toWildcard(s[:idx]))
		s = s[idx+1:]
	}

	return wilds
}

func toWildcard(s string) string {
	if !(strings.HasPrefix(s, "{") && strings.HasSuffix(s, "}")) {
		return ""
	}
	if s == "{$}" {
		return ""
	}
	return strings.TrimSuffix(s[1:len(s)-1], "...")
}

func stripToLastSlash(s string, cnt int) string {
	pos := 0
	for i, r := range s {
		if r == '/' {
			pos = i
			cnt--
			if cnt <= 0 {
				break
			}
		}
	}
	return s[pos:]
}

Usage:

mux.Handle(pattern, StripSegments(pattern, handler))