HTTP/1.1, chunked encoding and trailers

[htuch]

We've had a number of internal questions about how Envoy handles trailers with chunked encoded HTTP/1.1 when transcoding back and forth between HTTP/2. I.e. how do HTTP/2 trailers get represented. Looking at the HTTP/1.1 codec implementation, it doesn't seem like it does anything useful with the trailers in StreamEncoderImpl::encodeTrailers.

I'm thinking I'm missing something here, since I thought we had reasonably complete support for HTTP/1.1 chunked encoding, but this isn't my area really. @mattklein123 I assume you know the full story here :)

CC @wlhee

[wlhee]

Thanks @htuch
Another corner case is HTTP/2 can send "trailers-only" (used by gRPC error message), how does envoy convert the trailers to HTTP/1.1?

[lizan]

@wlhee "trailers-only" is actually "headers-only" in term of HTTP/2, so that one can be transcoded to a normal HTTP/1.1 response without body.

[lizan]

@htuch AFAIK, encodeTrailers for HTTP/1.1 codec is not implemented just because no real world client handles that correctly, and emitting trailers with HTTP/1.1 chunked encoding might cause unexpected behavior. Also the gRPC-JSON transcoder expect that behavior of HTTP/1.1 codec.

[wlhee]

@wlhee "trailers-only" is actually "headers-only" in term of HTTP/2, so that one can be transcoded to a normal HTTP/1.1 response without body.

I see, this matches my observation. We want to transfer gRPC error message, which is a Trailers-only h2 response, over an intermediate HTTP/1.1 response. We can work around it for gRPC error message only.

[wlhee]

@htuch AFAIK, encodeTrailers for HTTP/1.1 codec is not implemented just because no real world client handles that correctly, and emitting trailers with HTTP/1.1 chunked encoding might cause unexpected behavior. Also the gRPC-JSON transcoder expect that behavior of HTTP/1.1 codec.

Is HTTP/1.1 codec used for h2->h1 transferring or h1->h1 transferring?
My observation shows h2->h1, trailers are actually encoded.
However, for h1->h1 via chunked encoding, trailers are dropped.

[mattklein123]

@htuch AFAIK, encodeTrailers for HTTP/1.1 codec is not implemented just because no real world client handles that correctly, and emitting trailers with HTTP/1.1 chunked encoding might cause unexpected behavior. Also the gRPC-JSON transcoder expect that behavior of HTTP/1.1 codec.

This is correct. It would not be hard to implement HTTP/1.1 trailers, but I've never seen a real-world case in which this would be useful.

re: H2 <-> H1 bridging, check out the filters defined here https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/grpc#grpc-bridging. They are implemented to help do bridging for gRPC to H1.

[wlhee]

@htuch AFAIK, encodeTrailers for HTTP/1.1 codec is not implemented just because no real world client handles that correctly, and emitting trailers with HTTP/1.1 chunked encoding might cause unexpected behavior. Also the gRPC-JSON transcoder expect that behavior of HTTP/1.1 codec.

This is correct. It would not be hard to implement HTTP/1.1 trailers, but I've never seen a real-world case in which this would be useful.

Do you mean "it would not be hard to implement converting HTTP/2 trailers to HTTP/1.1 trailers"?

This would be useful in our use case as we have another proxy in front of envoy:
client -> HTTP/2 -> proxy -> HTTP/1.1 -> envoy -> HTTP/2 -> server

We need tell the proxy response trailers to help it reconstruct the HTTP/2 response to the client.

[htuch]

@mattklein123 @lizan the specific use case that is interesting here to us is a multi-hop transcode:

HTTP/2 <-> Proxy A <-> HTTP/1.1 <-> Proxy B <-> HTTP/2

Assume that proxy B is Envoy but proxy A is just a normal standards compliant H2 <-> H1 transcoding proxy with support for trailers, and you want to transcode H2 to H1 for the purpose of tunneling (those details are not material I think). I think this is where we're hitting the gap in existing support. It's not possible to use the reverse gRPC bridge at A.

[mattklein123]

Do you mean "it would not be hard to implement converting HTTP/2 trailers to HTTP/1.1 trailers"?

Correct, it would not be difficult. It just needs to be implemented in the codec to correctly deal with the chunked trailer.

[wlhee]

Something I don't quite understand:
when I enabled the debug log in envoy and sent a gRPC request via the data path mentioned above, I saw the following logs:
2019-05-29 22:25:15.819][60332][debug][http] [third_party/envoy/src/source/common/http/conn_manager_impl.cc:1329] [C0][S3357797201862188036] encoding headers
via codec (end_stream=false):
':status', '200'
'content-type', 'application/grpc'
'x-envoy-upstream-service-time', '1'
'date', 'Thu, 30 May 2019 05:25:15 GMT'
'server', 'envoy'

[2019-05-29 22:25:15.820][60332][debug][client] [third_party/envoy/src/source/common/http/codec_client.cc:95] [C1] response complete
[2019-05-29 22:25:15.820][60332][debug][pool] [third_party/envoy/src/source/common/http/http2/conn_pool.cc:237] [C1] destroying stream: 0 remaining
[2019-05-29 22:25:15.824][60332][debug][http] [third_party/envoy/src/source/common/http/conn_manager_impl.cc:1512] [C0][S3357797201862188036] encoding trailer
s via codec:
'grpc-status', '0'
'grpc-message', ''

If the trailers conversion is missing, why would I see "encoding trailer" part?

[lizan]

@wlhee it is the downstream codec (after HCM log) eating the trailers, so you still see them in HCM log.

[wlhee]

I see. But the weird thing is, when my upstream HTTP/1.1 server sends trailers, I did NOT see them in the HCM log:
[2019-06-04 09:26:46.282][257872][debug][router] [third_party/envoy/src/source/common/router/router.cc:893] [C6][S12922332826246243808] upstream headers complete: end_stream=false
[2019-06-04 09:26:46.282][257872][debug][http] [third_party/envoy/src/source/common/http/conn_manager_impl.cc:1336] [C6][S12922332826246243808] encoding headers via codec (end_stream=false):
':status', '200'
'content-type', 'text/plain; charset=utf-8'
'trailer', 'AtEnd1'
'trailer', 'Atend2'
'trailer', 'AtEnd3'
'date', 'Tue, 04 Jun 2019 16:26:45 GMT'
'x-envoy-upstream-service-time', '1'
'server', 'envoy'

However, my server is actually sending trailers, the followings are response directly from curling the http/1.1 server:
curl -v --raw localhost:8081/trailers

• Trying ::1...
• TCP_NODELAY set
• Connected to localhost (::1) port 8081 (#0)

GET /trailers HTTP/1.1
Host: localhost:8081
User-Agent: curl/7.50.2
Accept: /

< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Trailer: AtEnd1
< Trailer: Atend2
< Trailer: AtEnd3
< Date: Tue, 04 Jun 2019 16:27:39 GMT
< Transfer-Encoding: chunked
<
9
Chunk #1

9
Chunk #2

0
Atend1: VaLue 1
Atend2: VaLue 2
Atend3: VaLue 3

• Connection #0 to host localhost left intact

[lizan]

@wlhee that because the upstream HTTP/1.1 codec ate them, so they are not even hitting HCM.

[wlhee]

I see, to confirm my understanding:

1. upstream HTTP/2 trailers -> HCM log -> HTTP/1.1 codec eats the trailers -> downstream HTTP/1.1 client doesn't see trailers
2. upstream HTTP/1.1 trailers -> HTTP/1.1 codec eats the trailers -> no HTCM log -> downstream HTTP/1.1 client doesn't see trailers

[lizan]

@wlhee That's right.

[mattklein123]

Closing as answered.