-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/net/http: PROTOCOL_ERROR with HTTP2 #29125
Comments
Let me add that the problem arises seldom, so some downloads go just fine, while other present this problem. The overall workflow is to get the tar.gz from the docker hub, pass the byte stream to "compress/gzip" (so something like gzip.NewReader(resp.Body) ) and finally pass the reader created by gzip to the STDIN of another software (using "os.exec" package). Moreover, if instead of passing the result of the uncompression to the STDIN of another software I just save it on file I cannot reproduce the problem. |
Hi @siscia - can you post an independent code sample that we can use to reproduce this issue ? |
Hi, I was not able to re-produce without using cvmfs_server, the custom software that read from STDIN. Maybe is a problem from our side but I don't know what it could be... Should I make a small code sample that use cvmfs_server and reproduce the problem or I should just close the issue? |
/cc @bradfitz |
So I am trying to find all the places we just report a stream error without any cause. There aren't many. Adding a cause to all places that return an error would at least identify where its coming from. |
@fraenkel I thought this was some mystery server sending us a PROTOCOL_ERROR because it didn't like our Transport's HTTP/2 behavior. |
@bradfitz I see them against the docker registries, in particular I can see them against the standard registries hosted by docker hub itself. I believe they are hosted on AWS. Just to clarify, the Docker Registries servers are sending data, a go program is piping those data into the STDIN of another program, the |
@bradfitz Sorry I missed the RST_STREAM at the top. I find it odd that all we see are just WINDOW_UPDATE. The fact that "all" or many streams hit this at the same time could mean that we exceeded the connection window? I am guessing only because we are missing information. |
Do you guys need any more info from my side? |
It may be same as #31534 ? |
It may it may not, I worked around the issue and I can't recal the problem! Feel free to close though! |
How did your workaround, if I may ask @siscia ? |
I don't remember honestly, two options:
I don't remember which option, but nothing technically interesting I am afraid :( |
We see some test failures (and probably real-world failures as well) in the downloads with something like: ``` + snap install test-snapd-content-circular1 error: cannot perform the following tasks: - Download snap "test-snapd-content-circular2" (2) from channel "stable" (stream error: stream ID 1; PROTOCOL_ERROR) ``` Looking at the logs it seems that we get the protocol error from fastly and that it is sent to us from the remote side: ``` Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Transport failed to get client conn for fastly.cdn.snapcraft.io:443: http2: no cached connection was available Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Transport creating client conn 0xc4200baa80 to 151.101.250.217:443 ... c4204fbce0: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=869 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: read SETTINGS len=6, settings: MAX_CONCURRENT_STREAMS=100 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: wrote SETTINGS flags=ACK len=0 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: read WINDOW_UPDATE len=4 (conn) incr=16711681 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: read SETTINGS flags=ACK len=0 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: read HEADERS flags=END_HEADERS stream=1 len=378 Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: read RST_STREAM stream=1 len=4 ErrCode=PROTOCOL_ERROR Aug 09 20:03:16 aug091949-909109 snapd[7255]: 2019/08/09 20:03:16 http2: Framer 0xc4204fbce0: wrote RST_STREAM stream=1 len=4 ErrCode=CANCEL ... ``` There is a bugreport upstream that looks a bit like our issue: golang/go#29125 and there retrying seemed to have solved the issue. We should dig a bit deeper and see if we can build a small reproducer using fastly so that we can figure out if go2 is doing something that fastly is not expecting or if fastly is the issue. But in the meantime this hopefully unblocks us.
I am seeing something similar to this. On the client side I see I've been able to get around this by using FYI I'm using a non tls http2 client:
I'm unable to provide http2 logs because I only see the issue at scale. |
Just in case anyone else finds this GitHub issue after making the same mistake as me: Using
I would love to have amore descriptive error message. Shouldn't it be possible to check against a list of valid (according to specification) HTTP verbs and then say the passed one is not one of them? Anyway, I hope this helps someone. |
There is http.MethodGet so you don't have to think about it. |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
Currently just a "me too", but I'm running into a problem very similar to @stan-sack where sometimes file uploads are terminated when trying to read request body with .NET Framework HttpClient (HTTP/1.1) performing a multipart/form-data upload to a GCP load balancer over HTTPS with a H2C Cloud Run backend. This is with go 1.20. I captured some additional debug details with
This is the error I see on the client side when it happens:
|
@stan-sack, was your issue happening in GCP and/or Envoy proxy? |
I was calling non-TLS HTTP/2 (h2c) between two internal services via envoy client load balancing. It was in AWS and there was no actual load balancer. |
Seems suspicious though. In my case, envoy (inside GCP load balancer) is also calling non-TLS HTTP/2 (h2c) |
This was a long time ago. I dont work at the company where I had this problem anymore. If I was still trying to solve I'd start playing with TCP keepalive and connection pooling in both go and envoy. I didnt have very much experience with envoy at that time but in hindsight it seems like it could be the source of the problem. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes, which I believe is the one I am running, hence 1.11.2
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
I am downloading tar files from the docker hub, uncompress them using the standard gzip, and passing the result to another software using STDIN.
If is necessary I will try to reproduce this using something simpler like sha256 sum in linux.
What did you expect to see?
I would not expect a PROTOCOL_ERROR
What did you see instead?
A PROTOCOL_ERROR
Below the relevant log using
GODEBUG=http2debug=2
Maybe is related to #26321 but I don't know how to tell.
The text was updated successfully, but these errors were encountered: