"http_status=503" When downloading with high concurrency

[likewtf1]

I want to use mountpoint-s3 for 100 ec2 machines, which each machine runs as many programs as the number of cpus , to access some files in S3 at the same time. Sometimes it works fine, but sometimes i meet the warning message below:

Oct 24 10:58:39 ip-172-31-135-62 mount-s3[453]: [WARN] lookup{req=20094 ino=109 name="***"}:list_objects{id=16589 bucket=*** continued=false delimiter=/ max_keys=1 prefix=***}: mountpoint_s3_client::s3_crt_client: request failed request_type=Default http_status=503 range=None duration=426.995805ms ttfb=Some(7.681499ms) request_id=***
Oct 24 10:58:40 ip-172-31-135-62 mount-s3[453]: [WARN] open{req=20158 ino=1706 pid=1759}:list_objects{id=16643 bucket=*** continued=false delimiter=/ max_keys=1 prefix=***}: mountpoint_s3_client::s3_crt_client: request failed request_type=Default http_status=503 range=None duration=314.021865ms ttfb=Some(8.180981ms) request_id=***

I have three questions regarding this:

1. Is this warning report fine? Or how to avoid this? My current structure is like "//files.dat" where there are almost 5000 files in one date. And my program will read 10 days of data simultaneously.
2. Each program will be run 5 hours. During program running ,i find that the memory usage of mount-s3 is steady increasing to almost the whole memory, is it right?
3. In the Cost Explore, i find 'ListBucket' API cost five times than 'GetObject' API. Is that correctly? Or how to reduce the cost?

[sauraank]

1. 503 response typically indicates that the number of requests to your S3 bucket is high. For example, you can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket. However, Amazon S3 can return a 503 Slow Down response if your requests exceed the amount of bandwidth that's available. You can try to mitigate it by distributing objects across multiple prefixes if your use case allows it. Amazon S3 gradually scales up to handle requests for each of the prefixes separately.
2. We may expect it to grow if there's a lot of concurrent file operations, but it should also be deallocating the memory too. You may share the logs so that we can take a look at the metrics to know if there is any memory leak. Please use --log-metrics option with mountpoint for the logs.

[sauraank]

3. While using Mountpoint, there are higher number of List bucket requests because it is made everytime we have to Lookup for a file/directory. We are working on reduce the cost on repeated access. You can track it here: Reduce cost for repeated accesses #255.

[likewtf1]

Thanks! I am going to tar my files and change the structure in s3 to avoid 503 warning. I will update this issue when finished test.

[dannycjones]

I hope the issue was resolved! Please do re-open the issue or open a new issue if you are encountering problems.