Quic connection idle timeout kicks in while HTTP/3 request is in progress

[bentoi]

Description

This is related to the missing Quic KeepAlive option issue I reported a while ago: #70090.

Because Quic doesn't provide an option to enable keep alive, an HTTP/3 request can fail while being processed by the server if the Quic connection idle timeout kicks in.

I'm attaching a test case that demonstrates this. This test case shows a number of issues:

• the PooledConnectionIdleTimeout property is used to configure the QuicConnection idle timeout. To me these properties are unrelated.
• Kestrel shows a critical error when the Quic stream is shutdown
• the HTTP/3 request doesn't complete because the connection idle timeout kicks in

keepalive.zip

Reproduction Steps

dotnet run --project kestrel-server/kestrel-server.csproj
dotnet run --project http-client/http-client.csproj

Expected behavior

No exceptions, the PUT request should succeed.

Actual behavior

The PUT request fails with the following exception:

Unhandled exception. System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: The connection timed out from inactivity.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.Main(String[] args) in /home/vagrant/workspace/testcases/keepalive/http-client/Program.cs:line 26
   at Program.<Main>(String[] args)

Kestrel shows a critical error:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowIfMsQuicError(Int32 status, String message)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Regression?

No response

Known Workarounds

No response

Configuration

.NET SDK:
Version: 7.0.302
Commit: 990cf98a27

Runtime Environment:
OS Name: ubuntu
OS Version: 22.04
OS Platform: Linux
RID: ubuntu.22.04-x64
Base Path: /home/vagrant/dotnet/sdk/7.0.302/

Host:
Version: 7.0.5
Architecture: x64
Commit: 8042d61

.NET SDKs installed:
7.0.302 [/home/vagrant/dotnet/sdk]

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

This is related to the missing Quic KeepAlive option issue I reported a while ago: #70090.

Because Quic doesn't provide an option to enable keep alive, an HTTP/3 request can fail while being processed by the server if the Quic connection idle timeout kicks in.

I'm attaching a test case that demonstrates this. This test case shows a number of issues:

• the PooledConnectionIdleTimeout property is used to configure the QuicConnection idle timeout. To me these properties are unrelated.
• Kestrel shows a critical error when the Quic stream is shutdown
• the HTTP/3 request doesn't complete because the connection idle timeout kicks in

keepalive.zip

Reproduction Steps

dotnet run --project kestrel-server/kestrel-server.csproj
dotnet run --project http-client/http-client.csproj

Expected behavior

No exceptions, the PUT request should succeed.

Actual behavior

The PUT request fails with the following exception:

Unhandled exception. System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: The connection timed out from inactivity.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final)
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken)
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.Main(String[] args) in /home/vagrant/workspace/testcases/keepalive/http-client/Program.cs:line 26
   at Program.<Main>(String[] args)

Kestrel shows a critical error:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in HttpConnection.ProcessRequestsAsync.
      System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
         at System.Net.Quic.ThrowHelper.ThrowIfMsQuicError(Int32 status, String message)
         at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Internal.QuicStreamContext.Abort(ConnectionAbortedException abortReason)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3FrameWriter.Abort(ConnectionAbortedException error)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Stream.AbortCore(Exception exception, Http3ErrorCode errorCode)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http3.Http3Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync[TContext](IHttpApplication`1 httpApplication)

Regression?

No response

Known Workarounds

No response

Configuration

.NET SDK:
Version: 7.0.302
Commit: 990cf98a27

Runtime Environment:
OS Name: ubuntu
OS Version: 22.04
OS Platform: Linux
RID: ubuntu.22.04-x64
Base Path: /home/vagrant/dotnet/sdk/7.0.302/

Host:
Version: 7.0.5
Architecture: x64
Commit: 8042d61

.NET SDKs installed:
7.0.302 [/home/vagrant/dotnet/sdk]

Other information

No response

Author:	bentoi
Assignees:	-
Labels:	untriaged, area-System.Net.Quic
Milestone:	-

[ManickaP]

Triage: I just retested this with H/2 and HTTP 1.1 and it doesn't fail there. We should fix this for H/3. However, the test is very artificial, setting up idle connection lifetime as low as 1s and having server pause for 5 seconds with the reply doesn't seem like realistic scenario. Workaround for this is not to set idle connection lifetime so low. Therefore Future for now.

For the QUIC pings, we could reuse existing H/2 settings https://learn.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.keepalivepingtimeout?view=net-7.0 and pass it to QuicConnection, obviously that would need above mentioned #70090. However, this should work without using pings, i.e. we should strive for as similar behavior as possible to H/2 and HTTP 1.1. One option would be to not to set QUIC idle timeout and implement #54968.

[bentoi]

Allowing to configure keep alive through the QuicConnectionOptions as suggested by #70090 could still be useful for application that want to use Quic.

Also, if the idle timeout is not set and keep alive are disabled, how will an HTTP/3 client figure out that the Quic connection is dead? For example the HTTP/3 client sends a GET request and waits for the response. If the connection link dies before the response is sent, won't the client hang indefinitely? Or is there a specific request timeout implemented at the HTTP level that handles this scenario?

The HTTP/3 spec also specifies that clients are expected to keep the Quic connection open while there are request/push in progress: https://datatracker.ietf.org/doc/html/rfc9114#name-idle-connections

[ManickaP]

Allowing to configure keep alive through the QuicConnectionOptions as suggested by #70090 could still be useful for application that want to use Quic.

I didn't mean that we shouldn't do #70090, just that we should fix this regardless of it. We might need to implement it to fix this though.

if the idle timeout is not set and keep alive are disabled, how will an HTTP/3 client figure out that the Quic connection is dead?

Anything called on that connection will fail, i.e. the next request will fail and the connection gets scrapped.

If the connection link dies before the response is sent, won't the client hang indefinitely? Or is there a specific request timeout implemented at the HTTP level that handles this scenario?

There is request timeout by default.

---

What I think should be the goal of this issues is that if there's a request on a connection, we never kill the connection until the request is done (which might happen to be request timeout). How is that achieved should be implementation detail hidden from the user (from HttpClient perspective) as this should behave the same for all HTTP versions.