[8.0] HTTP/2 request to cloud hosted services is aborted by server with ENHANCE_YOUR_CALM

[JamesNK]

Description

Two gRPC users have reported problems with streaming calls make to services hosted by GCP/Google:

• gRPC stream "Response ended prematurely (ResponseEnded)" grpc/grpc-dotnet#2361
• gRPC Streaming From C# Resulting in GoAway error from server grpc/grpc-dotnet#2358

The streaming calls end after a small amount of activity with this error:

System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)

They're both verified that only the .NET client has this problem. Other gRPC implementations (Rust, go, Python) are successful.

Debugging logs shows that the problem is caused by the server aborting the connection because the client is sending too many pings. Logs:

poolId=44799463 workerId=57352375 requestId=0 memberName=ProcessOutgoingFramesAsync message=writeBytes=17
poolId=44799463 workerId=57352375 requestId=0 memberName=SendPingAsync message=Started writing. pingContent=-5
poolId=44799463 workerId=57352375 requestId=0 memberName=FlushOutgoingBytesAsync message=ActiveLength=17
poolId=44799463 workerId=57352375 requestId=0 memberName=ReadFrameAsync message=initialFrame=False
poolId=44799463 workerId=57352375 requestId=0 memberName=ProcessIncomingFramesAsync message=Frame 13: StreamId=0; Type=GoAway; Flags=None; PayloadLength=22.
poolId=44799463 workerId=57352375 requestId=0 memberName=ReadGoAwayFrame message=lastStreamId=1, errorCode=EnhanceYourCalm

The client sends a ping to the server, and the server responds with a GOAWAY with an error code of ENHANCE_YOUR_CALM.

The problem can be worked around by disabling RTT pings:

AppContext.SetSwitch("System.Net.SocketsHttpHandler.Http2FlowControl.DisableDynamicWindowSizing", true);

Reproduction Steps

A customer has created a reproduction - grpc/grpc-dotnet#2361

Expected behavior

HttpClient should be able to successfully call GCP/Google hosted endpoints without the server aborting the connection.

Resolving this problem requires low level knowledge of HTTP/2 and client/server interactions. There is no way developers will understand how to solve this problem.

Actual behavior

GOAWAY + ENHANCE_YOUR_CALM

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Two gRPC users have reported problems with streaming calls make to services hosted by GCP/Google:

• gRPC stream "Response ended prematurely (ResponseEnded)" grpc/grpc-dotnet#2361
• gRPC Streaming From C# Resulting in GoAway error from server grpc/grpc-dotnet#2358

The streaming calls end after a small amount of activity with this error:

System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)
   at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
   at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
   at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)

They're both verified that only the .NET client has this problem. Other gRPC implementations (Rust, go, Python) are successful.

Debugging logs shows that the problem is caused by the server aborting the connection because the client is sending too many pings. Logs:

poolId=44799463 workerId=57352375 requestId=0 memberName=ProcessOutgoingFramesAsync message=writeBytes=17
poolId=44799463 workerId=57352375 requestId=0 memberName=SendPingAsync message=Started writing. pingContent=-5
poolId=44799463 workerId=57352375 requestId=0 memberName=FlushOutgoingBytesAsync message=ActiveLength=17
poolId=44799463 workerId=57352375 requestId=0 memberName=ReadFrameAsync message=initialFrame=False
poolId=44799463 workerId=57352375 requestId=0 memberName=ProcessIncomingFramesAsync message=Frame 13: StreamId=0; Type=GoAway; Flags=None; PayloadLength=22.
poolId=44799463 workerId=57352375 requestId=0 memberName=ReadGoAwayFrame message=lastStreamId=1, errorCode=EnhanceYourCalm

The client sends a ping to the server, and the server responds with a GOAWAY with an error code of ENHANCE_YOUR_CALM.

The problem can be worked around by disabling RTT pings:

AppContext.SetSwitch("System.Net.SocketsHttpHandler.Http2FlowControl.DisableDynamicWindowSizing", true);

Reproduction Steps

A customer has created a reproduction - grpc/grpc-dotnet#2361

Expected behavior

HttpClient should be able to successfully call GCP/Google hosted endpoints without the server aborting the connection.

Resolving this problem requires low level knowledge of HTTP/2 and client/server interactions. There is no way developers will understand how to solve this problem.

Actual behavior

GOAWAY + ENHANCE_YOUR_CALM

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author:	JamesNK
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[antonfirsov]

Backport will happen after 9.0 GA, moving to Future.