Additional data/context causes the target java service to fail with buffer overflow error

[sshklianko]

Hi
I tried to use the thrift-client-ttwitter-filter and thrift-client-context-filter to send extra headers/context data to my java services that use finagle/thrift stack, and I used TCP connection for that.

Problem is, that, for example, if I use the thrift-client-ttwitter-filter and its underlying upgradeRequest (upgrade protocol) feature, then, after a few successful requests all the following requests start to get an error:

Client error: [thrift-client,error,TcpConnection] Error sending Thrift request: Thrift connection ended

Server error:
09:54:27.527 [finagle/netty4-2-6] DEBUG com.fitbit.core.finagle.prometheus.PrometheusReceiver - [IGNORED METRIC][counter]: Vector(thrift, thriftmux, downgraded_connects) 'thrift_thriftmux_downgraded_connects'
09:54:28.175 [ZkAnnouncer Mutator-SendThread(localhost:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x1000392b8ea0b2e after 0ms
09:54:28.187 [finagle/netty4-2-6] WARN com.twitter.finagle.util.DefaultMonitor - Exception propagated to the default monitor (upstream address: /127.0.0.1:58354, downstream address: n/a, label: thrift).
org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed. Tried to read 251660300 bytes, but only got 1002 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:88)
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:236)

... same errors occur if I simply use the above js filters without the upgradeRequest feature

Im thinking, that an attempt to upgrade the protocol, as well as a simpler attempt to attach an additional buffer with headers/context data to my request's payload (in the context of the multiple tcp connections pool) messes up with finagle and causes the server to fail, so maybe you have ideas how to address that?