[release/6.0] Add additional trace to HttpConnectionPool #66958
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of #66605 to release/6.0
Customer Impact
.NET 6 introduced changes to
HttpConnectionPool
that decouples HTTP connection creation from a request that initiated it. This made issues related to connection creation harder to troubleshoot.We had recently 2 investigations which took unnecessarily long to root cause due to suboptimal logging. As a result, we want to add additional logging to make such investigations in future easier for us (Networking team) and for the customers (self-diagnostics).
For example on of the investigations was about hanged "pending" HTTP/1.1 connection issues (i.e. the reasons why new connections were not created) in
HttpConnectionPool
. The issue manifests as growing number of timeouted requests. It is not easy to link a timeouted request (esp. with a small timeout) to a connection that was created 3 minutes ago but still is not connected (Linux default connection timeout is more than 3 minutes). The trace of connection being initiated may even not be captured, if tracing was turned on when the visible trouble occur. There were no traces of the state ofHttpConnectionPool
, e.g. number of pending connections, so it is unclear from existing traces why the pool decided not to create a connection.The additional trace should help troubleshoot why new connection was not created and why the pending connection failed.
Testing
Presence of new traces verified by customer deploying private bits and by a test app.
Risk
Low. This is private-telemetry-only (
Private.InternalDiagnostics.System.Net.Http
) change.