net/http: copy headers when following redirects

[rsc]

http changed recently to follow a redirect on POST. However, the followup request does
not include the headers from the original request. This caused a problem in a Selenium
setting because the Accept: app/json header was not propagated. Is there a standard for
which headers should be copied when following a redirect? Should we, uh, follow it?

[bradfitz]

Comment 1:

I see nothing either way in:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3
On one hand, it seems sane to copy all the headers when following a redirect.
On the other hand, what if one of those headers is "Authorization" and it contains
credentials you don't want to pass along to another party?
I don't know the answer.  I'm curious what other implementations do.

[bradfitz]

Comment 2:

Copy/paste of an internal G+ comment I got about this:
-----------
As per these threads, it's not totally specified by HTTP itself:
http://lists.w3.org/Archives/Public/ietf-http-wg/2011AprJun/0489.html
http://lists.w3.org/Archives/Public/ietf-http-wg/2011JulSep/0014.html
See HTML which specifies some part of it:
http://www.whatwg.org/specs/web-apps/current-work/multipage/fetching-resources.html
"""
First, apply any relevant requirements for redirects (such as showing any appropriate
prompts). Then, redo main step, but using the target of the redirect as the resource to
fetch, rather than the original resource. For HTTP requests, the new request must
include the same headers as the original request, except for headers for which other
requirements are specified (such as the Host header). [HTTP]
"""
As far as Chrome's HTTP stack, we only preserve embedder headers. Here's the algorithm:
* Embedder (Chrome/WebKit) specifies "extra" request headers in the URL request. A URL
request follows HTTP redirects, so since these are tied to the URL request, they are
sent on all HTTP requests.
* All other headers are generated internally during HTTP processing:
  - At the HTTP transaction level (actually, in URLRequestHttpJob), we add some headers, like cookies.
  - At the network transaction level, we generate the Host header and others. This is also the level at which we do authorization. We re-run the authorization algorithm there, which includes querying our auth cache and re-applying a token if applicable.

[gopherbot]

Comment 3 by stephane.travostino:

+1, in a Go crawler I've written recently, I had to create a custom http.Client function
which, upon redirect, kept the "Range" and "User-Agent" headers I set.
Looking at Python's urllib2 implementation, they have a map of "unredirected headers"
such as "Authorization" and other internally generated ones, while headers set by the
user are kept upon redirection.
What if a user sets an Authorization header manually and the URL redirects to a
different domain? 
I think the implementation should keep it, so the idea of having a "custom headers"
array and "unredirected headers" one is the way to go, IMHO.

[rsc]

Comment 4:

Labels changed: added go1.1maybe, removed go1.1.

[gopherbot]

Comment 5 by alberto.garcia.hierro:

I've checked what libcurl does (which I'd consider a pretty solid implementation) and it
forwards any headers but authentication info. From the curl manual:
"When authentication is used, curl only sends its credentials to the initial host. If a
redirect takes curl to a different host, it won't be able to intercept the
user+password. See also --location-trusted on how to change this."
 --location-trusted
              (HTTP/HTTPS) Like -L, --location, but will allow sending the name + password to all hosts that the site may redirect to. This may or may not introduce a security breach if the site redirects you to a site to which you'll send your authentication info (which  is  plain-
              text in the case of HTTP Basic authentication).
I think this is a pretty sensible behavior and I already have a patch which implements
this is the net/http package. By default, all headers but WWW-Authenticate are copied
and the User property in the initial URL is not copied. If the property TrustRedirects
in the request it set, it will copy everything, including the User property of the
initial URL. So, should I just submit this patch as a CL or do you have any objections?

[robpike]

Comment 6:

Labels changed: added go1.2maybe, removed go1.1maybe.

[rsc]

Comment 7:

Issue #5782 has been merged into this issue.

[rsc]

Comment 8:

Labels changed: added feature.

[robpike]

Comment 9:

Not for 1.2.

Labels changed: removed go1.2maybe.

[rsc]

Comment 10:

Labels changed: added go1.3maybe.

[rsc]

Comment 11:

Labels changed: removed feature.

[rsc]

Comment 12:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 13:

Labels changed: added repo-main.

[bradfitz]

Comment 14:

Issue #7801 has been merged into this issue.

[bradfitz]

Comment 15:

Issue #8027 has been merged into this issue.

[gopherbot]

Comment 16 by steven.hartland@multiplay.co.uk:

Just hit this with our code, specifically not preserving the range request headers which
means it prevents any sort of attempt to process ranged downloads unless a custom 
transport / client is written :(

[gopherbot]

Comment 17 by steven.hartland@multiplay.co.uk:

@ alberto from #5 do you have your proposed patch hosted somewhere we can look at?

[rogpeppe]

Comment 18:

I've just hit this issue when using cookies against http.ServeMux.
The cookies are lost when the ServeMux redirects.
Here's an example of it happening:
http://play.golang.org/p/sMzbiLDm2i
and here's a simple workaround:
http://play.golang.org/p/JiGekzeAAY

[gopherbot]

Comment 19 by gyuho.cs:

I just hit this issue when using proxy to redirects. Any updates?

[gopherbot]

Comment 20 by sulochan:

I am using go version go1.3.3, and i am still seeing this issue. When
doFollowingRedirects is followed the new request is not copying the headers. Any updates
?

[gopherbot]

Comment 21 by gyuho.cs:

I don't think we need to enforce the headers to redirects. It's totally doable with
CheckRedirect... I use this for my personal API calls and http request. Works well.
https://github.com/gyuho/gon/blob/master/httpclient/httpclient.go#L72

[karalabe]

Just got bitten by this one. I was implementing an REST client to eventbrite and if I don't specify a trailing slash after a resource, eventbrite will redirect to the slashed version. However, this means that all the auth headers get dropped, and the redirected GET fails.

Wouldn't it be possible/worthwhile to check if the redirect refers to the same host, and copy the headers in such cases?

[grafana-dee]

Also just been bitten by this, calling an API with an Authorization header, and the API does a redirect. I can't avoid the redirect, the API implements a /whoami endpoint that looks at the identity of the owner of the Authorization token and then redirects to the given /profile/%d that is the person. The issue is that without the Authorization header being present on that subsequent call I get back a guest view of the profile in question, rather than the profile owners view of that profile.

The suggestion of implementing a solution similar to cURL wherein headers are retained for redirects on the same host (and scheme would be wise too) is a good one.

This is go1.7 btw.

[ghost]

Any updates? This bug can confuse Go newbies. For example, if one uses Reddit API and, having set a unique User-Agent, issues GET requests to, say, https://reddit.com/r/golang/top.json, one will get 429 Too Many Requests most of the time, because reddit.com redirects to www.reddit.com and User-Agent is quietly replaced upon redirect by Go-http-client/1.1, which is, being too common, is heavily rate limited. Meanwhile, in Python the same code just works. This isn't exactly what would attract beginners.

[campoy]

I ran on an issue related to this with talks.godoc.org

Details: golang/gddo#440

[gopherbot]

CL https://golang.org/cl/28930 mentions this issue.

[gopherbot]

CL https://golang.org/cl/31435 mentions this issue.