Some http responses come back from http_client mangled

[dt]

I think response bodies are only looked at by the elastic plugin so this probably isn't a big deal for most users, but it looks like http_client sometimes corrupts responses.

I modified http_output.go to record and send proto.Status(resp)[0] to graphite, and then observed some really strange values. After a little digging, I added a panic if the first 5 bytes of resp weren't HTTP/1.1 and discovered that sometimes they indeed were not (they looked like the middle of a response).

Our responses are on the large side, so I tried increasing the size of respBuff in http_client.go, but to no effect. I then tried swapping the c.conn.Read(c.respBuf) for ioutil.ReadAll(c.conn) which fixed it.

[buger]

I see what you mean, due how traffic interception works, sometimes you can get partial messages. In previous version validation was done by golang http library, now we need something similar. Can you approximately say what percent of bad requests it shows on your data?

[buger]

Ah I see that I misunderstood a bit. Now I get what you mean. I will try to create test which replicate this behavior.

On Friday, July 31, 2015, David Taylor notifications@github.com wrote:

I think response bodies are only looked at by the elastic plugin so this
probably isn't a big deal for most users, but it looks like http_client
sometimes corrupts responses.

I modified http_output.go to record and send proto.Status(resp)[0] to
graphite, and then observed some really strange values. After a little
digging, I added a panic if the first 5 bytes of resp weren't HTTP/1.1
and discovered that sometimes they indeed were not (they looked like the
middle of a response).

Our responses are on the large side, so I tried increasing the size of
respBuff in http_client.go, but to no effect. I then tried swapping the
c.conn.Read(c.respBuf) for ioutil.ReadAll(c.conn) which fixed it.

—
Reply to this email directly or view it on GitHub
#184.

Sincerely yours Leonid Bugaev
leonsbox@gmail.com
http:// http://buger.github.comleonsbox.com

[buger]

I think that problem happens because response buffer too small, and if body larger then buffer, parts of unread responses gets read during next call, corrupting data. Frankly using ioutil.ReadAll just hangs tests fro me...

[buger]

Replicated with test, and fixed #185