[BUG] Connection reset by peer is not handled well by retry policy

[kasobol-msft]

We observe Connection reset by peer being propagated as an error in tests and reported by customers.
Looking for holistic solution for this problem in the core.

References:

Customer reported: #21066

[somanshreddy]

@kasobol-msft A similar issue seen here #15215

CC: @rickle-msft @alzimmermsft

[alzimmermsft]

Thanks for submitting this issue @kasobol-msft.

Just finished a rough investigation on this. It appears that when the expected response body of a request is void/Void the networking layer won't eagerly read the response body (as there isn't expected to be one). But, later in the call stack when converting the response headers, and body but there is none here, into the returned response there is an attempt to consume/fast-forward the response body. This is where the issue lies as doing this can result in a state when the remote closes/closed the connection triggering an exception to be thrown. Unfortunately, at this point the response has egressed past the retry policy, resulting in the exception being thrown to the caller instead of being captured and retried.

[kasobol-msft]

There seems to be few missing cases that we should cover:

• buffered upload
• download
• pageable operations

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=900469&view=ms.vss-test-web.build-test-results-tab&runId=19372912&resultId=103744&paneView=debug

[kasobol-msft]

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=938985&view=ms.vss-test-web.build-test-results-tab&runId=20123708&resultId=101687&paneView=debug

[kasobol-msft]

Looks like there are still some gaps https://dev.azure.com/azure-sdk/internal/_build/results?buildId=954096&view=ms.vss-test-web.build-test-results-tab&runId=20421522&resultId=100738&paneView=debug

[alzimmermsft]

@kasobol-msft, has this happened recently or has this issue been fixed by #22647

[kasobol-msft]

This has happened recently, most likely because we exhausted retry count (though we're not 100% sure). We should follow up with aggregated exception and logs from retry policy.
The frequency of this dropped a lot though.

[alzimmermsft]

Closing as this hasn't been a reported issue for a while and if this is seen again a new issue should be opened.