-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/net/http2: connecting to non-compliant HTTP2 server returns Client.Timeout exceeded; fallback to HTTP/1? #20979
Comments
@tkng could you please try with the latest Go and see what the result is? Perhaps the bug was fixed in Go1.9. |
I've made a bug repro at https://github.com/odeke-em/bugs/blob/master/golang/20979/main.go or package main
import (
"io"
"log"
"net/http"
"os"
)
func main() {
res, err := http.Get("https://precious.jp")
if err != nil {
log.Fatal(err)
}
if res.Body != nil {
defer res.Body.Close()
}
if !statusOK(res.StatusCode) {
log.Fatalf("res: %s", res.Status)
}
io.Copy(os.Stdout, res.Body)
}
func statusOK(code int) bool { return code >= 200 && code <= 299 } which when run seems like an infinite loop of renegotiating headers, until I hit Ctrl+C $ GODEBUG=http2debug=2 go run main.go
2017/07/11 01:25:03 http2: Transport failed to get client conn for precious.jp:443: http2: no cached connection was available
2017/07/11 01:25:04 http2: Transport creating client conn 0xc42014c1a0 to 202.238.151.220:443
2017/07/11 01:25:04 http2: Framer 0xc42044d0a0: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2017/07/11 01:25:04 http2: Framer 0xc42044d0a0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2017/07/11 01:25:04 http2: Transport encoding header ":authority" = "precious.jp"
2017/07/11 01:25:04 http2: Transport encoding header ":method" = "GET"
2017/07/11 01:25:04 http2: Transport encoding header ":path" = "/"
2017/07/11 01:25:04 http2: Transport encoding header ":scheme" = "https"
2017/07/11 01:25:04 http2: Transport encoding header "accept-encoding" = "gzip"
2017/07/11 01:25:04 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2017/07/11 01:25:04 http2: Framer 0xc42044d0a0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=33
2017/07/11 01:25:04 http2: Framer 0xc42044d0a0: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:04 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:04 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 32768]
2017/07/11 01:25:04 http2: Framer 0xc42044d0a0: wrote SETTINGS flags=ACK len=0
2017/07/11 01:25:04 http2: Framer 0xc42044d0a0: read GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:04 http2: Transport received GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:04 http2: Transport failed to get client conn for precious.jp:443: http2: no cached connection was available
2017/07/11 01:25:05 http2: Transport creating client conn 0xc42014c4e0 to 202.238.151.220:443
2017/07/11 01:25:05 http2: Framer 0xc42044d500: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2017/07/11 01:25:05 http2: Framer 0xc42044d500: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2017/07/11 01:25:05 http2: Transport encoding header ":authority" = "precious.jp"
2017/07/11 01:25:05 http2: Transport encoding header ":method" = "GET"
2017/07/11 01:25:05 http2: Transport encoding header ":path" = "/"
2017/07/11 01:25:05 http2: Transport encoding header ":scheme" = "https"
2017/07/11 01:25:05 http2: Transport encoding header "accept-encoding" = "gzip"
2017/07/11 01:25:05 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2017/07/11 01:25:05 http2: Framer 0xc42044d500: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=33
2017/07/11 01:25:05 http2: Framer 0xc42044d500: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:05 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:05 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 32768]
2017/07/11 01:25:05 http2: Framer 0xc42044d500: wrote SETTINGS flags=ACK len=0
2017/07/11 01:25:05 http2: Framer 0xc42044d500: read GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:05 http2: Transport received GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:05 http2: Transport failed to get client conn for precious.jp:443: http2: no cached connection was available
2017/07/11 01:25:06 http2: Transport creating client conn 0xc42014c820 to 202.238.151.220:443
2017/07/11 01:25:06 http2: Framer 0xc42044d960: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2017/07/11 01:25:06 http2: Framer 0xc42044d960: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2017/07/11 01:25:06 http2: Transport encoding header ":authority" = "precious.jp"
2017/07/11 01:25:06 http2: Transport encoding header ":method" = "GET"
2017/07/11 01:25:06 http2: Transport encoding header ":path" = "/"
2017/07/11 01:25:06 http2: Transport encoding header ":scheme" = "https"
2017/07/11 01:25:06 http2: Transport encoding header "accept-encoding" = "gzip"
2017/07/11 01:25:06 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2017/07/11 01:25:06 http2: Framer 0xc42044d960: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:06 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:06 http2: Framer 0xc42044d960: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=33
2017/07/11 01:25:06 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 32768]
2017/07/11 01:25:06 http2: Framer 0xc42044d960: wrote SETTINGS flags=ACK len=0
2017/07/11 01:25:06 http2: Framer 0xc42044d960: read GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:06 http2: Transport received GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:06 http2: Transport failed to get client conn for precious.jp:443: http2: no cached connection was available
2017/07/11 01:25:06 http2: Transport creating client conn 0xc420488340 to 202.238.151.220:443
2017/07/11 01:25:06 http2: Framer 0xc4201481c0: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2017/07/11 01:25:06 http2: Framer 0xc4201481c0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2017/07/11 01:25:06 http2: Transport encoding header ":authority" = "precious.jp"
2017/07/11 01:25:06 http2: Transport encoding header ":method" = "GET"
2017/07/11 01:25:06 http2: Transport encoding header ":path" = "/"
2017/07/11 01:25:06 http2: Transport encoding header ":scheme" = "https"
2017/07/11 01:25:06 http2: Transport encoding header "accept-encoding" = "gzip"
2017/07/11 01:25:06 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2017/07/11 01:25:06 http2: Framer 0xc4201481c0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=33
2017/07/11 01:25:06 http2: Framer 0xc4201481c0: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:06 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:06 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 32768]
2017/07/11 01:25:06 http2: Framer 0xc4201481c0: wrote SETTINGS flags=ACK len=0
2017/07/11 01:25:06 http2: Framer 0xc4201481c0: read GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:06 http2: Transport received GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:06 http2: Transport failed to get client conn for precious.jp:443: http2: no cached connection was available
2017/07/11 01:25:07 http2: Transport creating client conn 0xc42014cb60 to 202.238.151.220:443
2017/07/11 01:25:07 http2: Framer 0xc42044ddc0: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2017/07/11 01:25:07 http2: Framer 0xc42044ddc0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2017/07/11 01:25:07 http2: Transport encoding header ":authority" = "precious.jp"
2017/07/11 01:25:07 http2: Framer 0xc42044ddc0: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:07 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:07 http2: Transport encoding header ":method" = "GET"
2017/07/11 01:25:07 http2: Transport encoding header ":path" = "/"
2017/07/11 01:25:07 http2: Transport encoding header ":scheme" = "https"
2017/07/11 01:25:07 http2: Transport encoding header "accept-encoding" = "gzip"
2017/07/11 01:25:07 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2017/07/11 01:25:07 http2: Framer 0xc42044ddc0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=33
2017/07/11 01:25:07 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 32768]
2017/07/11 01:25:07 http2: Framer 0xc42044ddc0: wrote SETTINGS flags=ACK len=0
2017/07/11 01:25:07 http2: Framer 0xc42044ddc0: read GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:07 http2: Transport received GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:07 http2: Transport failed to get client conn for precious.jp:443: http2: no cached connection was available
2017/07/11 01:25:08 http2: Transport creating client conn 0xc42014cea0 to 202.238.151.220:443
2017/07/11 01:25:08 http2: Framer 0xc4202288c0: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2017/07/11 01:25:08 http2: Framer 0xc4202288c0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2017/07/11 01:25:08 http2: Transport encoding header ":authority" = "precious.jp"
2017/07/11 01:25:08 http2: Transport encoding header ":method" = "GET"
2017/07/11 01:25:08 http2: Transport encoding header ":path" = "/"
2017/07/11 01:25:08 http2: Transport encoding header ":scheme" = "https"
2017/07/11 01:25:08 http2: Transport encoding header "accept-encoding" = "gzip"
2017/07/11 01:25:08 http2: Framer 0xc4202288c0: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:08 http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=32768, MAX_HEADER_LIST_SIZE=32768
2017/07/11 01:25:08 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2017/07/11 01:25:08 http2: Framer 0xc4202288c0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=33
2017/07/11 01:25:08 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 32768]
2017/07/11 01:25:08 http2: Framer 0xc4202288c0: wrote SETTINGS flags=ACK len=0
2017/07/11 01:25:08 http2: Framer 0xc4202288c0: read GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:08 http2: Transport received GOAWAY len=8 LastStreamID=0 ErrCode=NO_ERROR Debug=""
2017/07/11 01:25:08 http2: Transport failed to get client conn for precious.jp:443: http2: no cached connection was available
signal: interrupt /cc @tombergan @bradfitz |
@odeke-em thanks for your advice, I tried the latest golang master branch.
but, unfortunately, I still see the same error message as above.
|
What software is
That looks busted on their side. If they didn't like something about our request, they sure didn't say why. We write:
And then we read:
Maybe their implementation expects the SETTINGS ACK immediately after the initial SETTINGS frame (requiring a full RTT), instead of the the WINDOW_UPDATE + HEADERS in the middle? But the spec says in http://httpwg.org/specs/rfc7540.html#rfc.section.3.5:
So it does look like their server is at fault. /cc @tombergan |
It works in Chrome, though. What's the difference? |
@bradfitz thanks, your comment is helpful. Unfortunately, I don't know what kind of software is used. I'm just a developer of an image resizing proxy with Golang, and found that I can not access to the site through HTTP2 via Golang. I'll try to contact them, ask what software is used there. But there's little hope to obtain detailed information. (I don't have a direct connection to an engineer.) |
@tatsuhiro-t, do you happen to have any idea here? Either of what software that site runs, or what their server might not like about Go's HTTP/2 client? |
It looks like the server does not like SETTINGS_MAX_HEADER_LIST_SIZE. It smells like a bug of the server side. |
@tatsuhiro-t, oh, indeed! Thanks. It accepts a value up to 32KB, but one byte over that and it's rejected:
|
The precious.jp server is probably misinterpreting the meaning of We need to figure out which server this is and fix the interop problem. |
@tatsuhiro-t @bradfitz thanks for great progress. Though I haven't receive any replying from |
The reply came from I asked them to check their setting which related to SETTINGS_MAX_HEADER_LIST_SIZE if possible. |
There's no reply from By the way, I checked whether other http2 clients can access to
Since the most of major browsers can access to Though it's just an idea, is it possible to fallback to HTTP/1.1 when a server throws a GO_AWAY frame? |
Sending a GOAWAY frame is a totally reasonable thing to do. If we fell back to HTTP/1.1, we'd need to do it only if the GOAWAY's LastStreamID=0. But I'd also like to fix this in Netty. Hey netty-http2 authors, @yschimke, @atollena, @caniszczyk, @nhnFreespirit, it seems that Netty-http2 might be misinterpreting the meaning of SETTINGS_MAX_HEADER_LIST_SIZE and improperly rejecting clients. Go, for instance, says we're cool with up to 1MB of headers in the response, but Netty says "1MB? Whoa whoa, that's more than the 32KB I want to allocate for you, so GOAWAY." Whereas it should think, "1MB? Yeah whatever, my internal hpack buffer for you is way smaller than 1MB. You'll be fine with my little responses." |
Is there another Netty server directly facing the Internet? netty.io is behind Cloudflare's "cloudflare-nginx" http2 implementation. |
Is it? That sounds like a painful workaround for a broken server that will helpfully soon be fixed.
Are we sure the server is using netty? This comment says they're using BIG-IP, which is a load balancer from F5. Also, I'm not sure this is a bug in netty, at least not in the current version of netty. Here's the code that processes a SETTINGS frame: MAX_HEADER_LIST_SIZE = 2^32-1, which looks fine.
In my reading of the netty source, that code triggers only if an actual HEADERS frame is received where the size of the HEADERS exceeds goAwayMax. But Brad's example from earlier shows that we receive GOAWAY immediately after sending SETTINGS. |
@bradfitz, do you have any contacts at F5? |
I have an ex-F5 contact I can ask for referrals. Unfortunately https://f5.com/products/big-ip is not served with http2 so I can't verify. Apologies if I was assuming Netty incorrectly. I might've misinterpreted the comment above. |
in response to #20979 (comment) ... Netty does not do any immediate allocation as a result of receiving |
Sorry for my wrong comment about Netty caused confusions. If my understanding is correct, Unfortunately, I don't have a reply from site owner for several days, there's little hope that reply will come.
Sadly, I don't think we can hope the server will soon be fixed. Even contacting them is hard. Put it aside, IMO it's basically a good idea for smooth transition to downgrade a protocol to an old one if a new protocol doesn't work. (If there's no security issue.) Whether the standard library should automatically downgrade the protocol that is using is controversial, however I hope that the programmer is possible to downgrade if it's needed. Current implementation doesn't allow that behavior to developers, since it doesn't return any error message (it seems not returns infinitely, or returns with timeout error if timeout is set). I think there is room for improvement for this aspect. I hope golang's HTTP library to return an error message when the HTTP2 protocol was used and it received GO_AWAY frame before received anything. |
We just got another case of this in #21301 (closed as dup). But in #21301 you can see the same behavior but can also see @mnot, do you have HTTP/2 contacts at F5? We have an interop problem here with the BIG-IP server implementation. Thanks. |
Hello everyone, I'm in charge of F5's HTTP/2 code. We are aware of the problem and patched it in our upcoming release months ago. In a matter of days, this will also be available as a patch to deployed versions. If this problem is affecting you, please contact F5 support and ask for the patch for bug number 677119. |
@tombergan, what do you think of changing Go's http Transport to retry Requests with HTTP/1.1 if it sees a GOAWAY with LastStreamID=0? |
We could do that but it would be somewhat complex: The H2 transport needs to return errGoAwayBeforeAnyRequests if it sees GOAWAY with LastStreamID=0. Then pconn.shouldRetryRequest needs to recognize this error. So far not too bad. Line 418 in 75ab613
When that error happens (and only that error), we need to disable TLSNextProto on the next call to Transport.getConn and Transport.dialConn. Currently there's no easy way to do this, and adding such plumbing could get ugly. IMO, I'd rather wait until we had a second case where we could use this feature before adding it. The current issue can be avoided by having the client set http2.Transport.MaxHeaderListSize to 32768 or lower. That seems like a better short-term fix while clients wait for the next F5 release to roll out. |
I'm not sure this ticket is the correct place to post this. I found this ticket from #21301 which seemed like a better description of my situation, but it was closed as a duplicate so now I'm posting here. I was interacting with a TLS host where I tried changing the HTTP version with no result. The solution that ended up working was to use ...
tls.Config{
Certificates: []tls.Certificate{cert},
Renegotiation: tls.RenegotiateOnceAsClient,
DynamicRecordSizingDisabled: true,
} Again, Curl handles this seamlessly. So assuming there is a way to detect hosts that require |
@HaraldNordgren, please file a separate crypto/tls bug for your case. It's unrelated to this. It sounds like a TLS issue instead of an HTTP issue. |
hello, our client is as follows: `
however, when we visit google with host maps.googleapis.com, timeout sometimes does not work。 My question is that:
|
`tls.Config {
}` it works. however, i don't know why it works. when DynamicRecordSizingDisabled is true, we disables adaptive sizing of TLS records. Does it depend on the implementation of tls in server side? |
Any news on this issue? Because I encountered the same problem. I try to make a simple GET request. In the browser (Chrome), command line (cURL) and ozher tools like Postman the request is working fine. But if I try to do the request with the http.Client, nothing happens and the request will timeout. GO env
Sample Programm code
Output Unfortunately, I can't provide the real URL (and in addition, a whitelisting is needed to call the url) Here the curl output (ignore the 404, this is intended)
|
Seems OKTA has a broken implementation of HTTP2? There seem to be some quirks with they way golang handles HTTP2 requests (specifically when receiving GOAWAY frames) golang/go#20979, which result in the HTTP2 connection not terminating.
What version of Go are you using (
go version
)?What operating system and processor architecture are you using (
go env
)?What did you do?
Following snippet code should be enough to reproduce the problem.
https://play.golang.org/p/PmZWp6NJlq
If I disabled HTTP2 by the environment variable
GODEBUG=http2client=0
, then I see no error. Hence I guess this issue is related to HTTP2. This is might be the same as #13959, but I'm not sure.Also, I'm not sure this is the problem of golang's HTTP2 client library, or the problem of server side. (In most case, HTTP2 client works nicely.)
What did you expect to see?
I can connect to the server (by http2, or falling back to http1.1). At least, I want to see a more suggestive error message.
What did you see instead?
I saw an error message like following:
The text was updated successfully, but these errors were encountered: