Replay process can not tee some request with error "Can not parse request"

[TigerMee]

In my case, I use gor(0.7.0) like:
sudo gor --input-raw :80 --output-http xxxx:80

However, about 20% request can not be transferred. The error is "Can not parse request". I want to figure out why this happens. One of the error log is:

2014/01/16 10:15:29 Can not parse request jUyMEhlYXRlcnMlMjUyMEdlbmVyYWwtXy1ob3Qrd2F0ZXIrdGFuayUyNnVybCU1YiU1ZCUzZGh0dHAlMjUzQSUyNTJGJTI1MkZ3d3cubG93ZXMuY29tJTI1MkZIZWF0aW5nLUNvb2xpbmclMjUyRldhdGVyLUhlYXRlcnMlMjUyRl8lMjUyRk4tMXoxMW9uZyUyNTJGcGwlMjUzRnJwcCUyNTNEMjQqCTE5MjQwODE3NTILMTEwNzUyNTEyNjA4ggJAAkhSUIYXWANhDonW5e+50D9oAHBSeFKBAYXrUbgehfc/iQE9CtejcD3qP5ABAJgBBAqQBQoZQS5PLiBTbWl0aCBIeWJyaWQgSGVhdGVycxJGSGlnaCBFZmZpZWNpZW5jeSBBLk8uIFNtaXRoIEh5YnJpZCBXYXRlciBIZWF0ZXJzLiBGaW5kIGEgRGVhbGVyIE9ubGluZRoTaHR0cDovL2hvdHdhdGVyLmNvbSLHA2h0dHA6Ly8yMDgxOTY4LnIubXNuLmNvbS8/bGQ9OHZkeFFBRW43WTZXSUhuY2t4UUgtWWd6VlVDVXdjUG12dnpSQVJaaXV5Z0xDa0lFQ1pJNExNaEVDbU1Yalh5UHBKckxtenN3RmFOeG9 malformed HTTP request "jUyMEhlYXRlcnMlMjUyMEdlbmVyYWwtXy1ob3Qrd2F0ZXIrdGFuayUyNnVybCU1YiU1ZCUzZGh0dHAlMjUzQSUyNTJGJTI1MkZ3d3cubG93ZXMuY29tJTI1MkZIZWF0aW5nLUNvb2xpbmclMjUyRldhdGVyLUhlYXRlcnMlMjUyRl8lMjUyRk4tMXoxMW9uZyUyNTJGcGwlMjUzRnJwcCUyNTNEMjQqCTE5MjQwODE3NTILMTEwNzUyNTEyNjA4ggJAAkhSUIYXWANhDonW5e+50D9oAHBSeFKBAYXrUbgehfc/iQE9CtejcD3qP5ABAJgBBAqQBQoZQS5PLiBTbWl0aCBIeWJyaWQgSGVhdGVycxJGSGlnaCBFZmZpZWNpZW5jeSBBLk8uIFNtaXRoIEh5YnJpZCBXYXRlciBIZWF0ZXJzLiBGaW5kIGEgRGVhbGVyIE9ubGluZRoTaHR0cDovL2hvdHdhdGVyLmNvbSLHA2h0dHA6Ly8yMDgxOTY4LnIubXNuLmNvbS8/bGQ9OHZkeFFBRW43WTZXSUhuY2t4UUgtWWd6VlVDVXdjUG12dnpSQVJaaXV5Z0xDa0lFQ1pJNExNaEVDbU1Yalh5UHBKckxtenN3RmFOeG9"

[buger]

I released new version, pls check it https://github.com/buger/gor/releases/tag/0.7.5

[TigerMee]

The problem is still there, but the error is different.

[TigerMee]

For some request, I remove "Expect: 100-continue" from http header, and it works. So I guess perhaps Go's http library does not support "Expect: 100-continue". Is that a bug?

[joekiller]

I believe what you are seeing here is the tail end of a post that has already had most of the request sent. Basically the listener thinks the message is complete and sends it to the replay, then the listener gets more packets from the message which has already been shipped and then forwards that message to the replay.

[buger]

There were some changes in recent v0.8.4 which may help

[ghost]

This is my first time posting here and I just want to congratulate you on this excellent piece of software. Its been working great for us. This is the first issue we've experienced with it so far.

I've still observed this issue in the 0.8.4 release. I run gor as follows :

sudo gor --input-raw :8080 --output-http xxx:8080 --output-http-method POST

The replay breaks on the following conversation.

POST /some/path HTTP/1.1
Content-Type: application/xml
Content-Length: xxx
Host: xxx:8080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.0.3 (java 1.5)
Expect: 100-Continue

HTTP/1.1 100 Continue

<?xml version="1.0" encoding="UTF-8"?>
<X></X>

HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 10 Sep 2014 09:29:44 GMT
Server: Apache

The following errors are outputted by gor.

2014/09/10 09:00:46 Request error: Post http://xxx:8080/some/path: unexpected EOF
2014/09/10 09:00:46 Cannot parse request <?xml version="1.0" encoding="UTF-8"?>
<X></X> malformed HTTP version "encoding=\"UTF-8\"?>"

The 'Expect: 100-Continue' header is used by clients to give the server a chance to reject the request before the client goes to the trouble of sending the request body. It's normally used when the request body is large. The client sends the request headers including 'Expect: 100-Continue'. If it gets a 'HTTP/1.1 100 Continue' back it knows it's then good to send on the request body. http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3

It looks as though gor's request parsing does not understand this header. For the first part of the conversation it expects to find a request body. It doesn't find anything so outputs the error 'unexpected EOF'. For the second part of the conversation it expects a HTTP version header, but instead finds a request body.

[ghost]

It looks like output_http just delegates the request parsing to http://golang.org/pkg/net/http/#ReadRequest which does not have any special handling of this header.

[buger]

Yeah, its pretty simple parser right now...

[ghost]

OK. Thanks. I see there is another issue for this #77 where you mention that handling this isn't so straightforward. I didn't see this earlier.

[buger]

@stephencurran @TigerMee well, it took a while, but now it should fixed by #158