S3 ListObjectsV2Async hangs despite cancellation token in case of network connectivity issues

[david-helf]

Describe the bug

In certain cases of network connectivity issues, calling AmazonS3Client.ListObjectsV2Async to get a list of files in the bucket causes the method to hang indefinitely. This occurs despite passing a CancellationToken with a timeout, which should cancel the request after the specified period. While we have not been able to reproduce the issue manually, it has intermittently occurred in production, where the call does not respect the cancellation token, leading to prolonged hangs and impacting service stability.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

ListObjectsV2Async is terminated after the token is cancelled

Current Behavior

ListObjectsV2Async is not terminated after the token is cancelled and hangs indefinitely

Reproduction Steps

We cannot manually reproduce the issue. Therefore i provide an example code snippet to show how we use the client.

using Amazon;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;

namespace iXOdes.Infrastructure.AWS.S3;

public class AwsConnector
{
    private readonly IAmazonS3 mClient;
    private readonly string mBucketName;
    private readonly string mPrefix;

    public AwsConnector(string accessKey, string secretKey, string bucketName, string prefix, string endpoint)
    {
        mBucketName = bucketName;
        mPrefix = prefix;
        
        var credentials = new BasicAWSCredentials(accessKey, secretKey);
        mClient = new AmazonS3Client(credentials, RegionEndpoint.GetBySystemName(endpoint));
    }

    public async Task<List<S3Object>> GetFilteredBucketObjectsAsync()
    {
        var objects = new List<S3Object>();

        await foreach (var obj in ListBucketObjectsAsync())
        {
            // some filtering is done here in the real code before adding the s3 objects to the list
            objects.Add(obj);
        }

        return objects;
    }

    private async IAsyncEnumerable<S3Object> ListBucketObjectsAsync()
    {
        var request = new ListObjectsV2Request
        {
            BucketName = mBucketName,
            Prefix = mPrefix
        };

        ListObjectsV2Response response;

        do
        {
            using (var tokenSource = new CancellationTokenSource())
            {
                tokenSource.CancelAfter(TimeSpan.FromSeconds(300));
                
                // this call sometimes hangs forever even it has a cancellation token
                response = await mClient.ListObjectsV2Async(request, tokenSource.Token);
            }

            foreach (var obj in response.S3Objects)
            {
                yield return obj;
            }

            request.ContinuationToken = response.NextContinuationToken;

        } while (response.IsTruncated && response.NextContinuationToken != null);
    }
}

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.305.7

Targeted .NET Platform

.NET 8

Operating System and version

Debian 12

[bhoradc]

Hello @david-helf,

Thank you for reporting the issue. Since you mentioned that this issue is non-reproducible at will, are you able to enable verbose logging for the SDK? It may provide valuable insights into the state of the operation.

You could enable verbose logging by adding the following lines of code to your application before calling ListObjectsV2Async.

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

Additionally, I can attempt to reproduce the issue at my end by simulating network disruptions while running your provided code snippet.

Having said that, the .NET SDK is likely passing the CancellationToken down to the underlying .NET Framework or .NET Core/runtime. Therefore, to better understand the root cause and determine if it's an issue with the AWS SDK or the .NET Framework/runtime, it would be helpful to have the verbose logs requested above.

Regards,
Chaitanya

[david-helf]

hello @bhoradc, thank you for the quick response and suggestion. We will look into how we can integrate the verbose logging within our application.

[github-actions]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[david-helf]

@bhoradc It may take a while to retrieve the logs on our end. Could you archive any key findings from your testing in the meantime?

[ashishdhingra]

@bhoradc It may take a while to retrieve the logs on our end. Could you archive any key findings from your testing in the meantime?

@david-helf Please update once you have verbose logs or are able to come up with code which reproduces this scenario consistently. I guess @bhoradc used your code to reproduce the issue, but wasn't able to get the same error as you were getting in production environment. In your use case, SDK retry mechanism might be kicking in although this could be confirmed from verbose logs.

Thanks,
Ashish