S3 mountpoint is being unmounted right after mounting attempt

[sedrichaus]

Mountpoint for Amazon S3 version

1.3.1

AWS Region

us-east-1

Describe the running environment

• EC2
• AMI - Amazon Linux 2
• permissions granted via an instance profile

Mountpoint options

mount-s3 bucket --prefix prefix /data/s3/prefix --max-threads=1 --debug --log-directory /var/log/mount-s3
Same behavior observed if executed with the required options only
mount-s3 bucket /data/s3/prefix

What happened?

• Separate mount of BTRFS on /data - EBS/instance store, separate for each instance
• S3 is mounted on /data/s3/prefix, using a specific prefix on a bucket
• Initial S3 mount from user data on instances is successful
• For faster restart of our services, we need to format mount on /data device, meaning that we need to unmount /data/s3/prefix, format /data, and then remount /data/s3/prefix back
• After remounting the S3 mountpoint, we get a message that it was successful:

2024-01-09T12:09:52.754068Z  INFO mount_s3: successfully mounted prefix 'prefix' of bucket 'bucket' at /data/s3/prefix

And then we get back to list the content of the S3 mountpoint to find it empty.
From the "mount-s3 --debug --debug-crt" we saw that the mountpoint is automatically unmounted for some reason:

2024-01-09T12:09:52.753567Z  INFO fuser::session: Mounting- /data/s3/prefix   
2024-01-09T12:09:52.754068Z  INFO mount_s3: successfully mounted prefix 'prefix' of bucket 'bucket' at /data/s3/prefix
2024-01-09T12:09:52.754152Z DEBUG mount_s3: success status flag received from child process
2024-01-09T12:09:52.755034Z DEBUG mountpoint_s3::fuse::session: starting fuse worker 0 (thread id 9586)
2024-01-09T12:09:52.755055Z DEBUG fuser::request: FUSE(  1) ino 0x0000000000000000 INIT kernel ABI 7.26, capabilities 0x1ffffb, max readahead 131072    
2024-01-09T12:09:52.755099Z DEBUG fuser::request: INIT response: ABI 7.28, flags 0x2021, max readahead 131072, max write 16777216    
2024-01-09T12:09:52.784233Z  INFO fuser::session: Unmounted /data/s3/prefix
2024-01-09T12:09:52.784265Z DEBUG awscrt::S3Client: id=0x560d6b3316f0 Client starting destruction.    
2024-01-09T12:09:52.784318Z DEBUG awscrt::channel-bootstrap: id=0x560d6b31cc40: releasing bootstrap reference

Direct listing via "aws s3 ls" works fine.
Complete log is attached to "relevant log output" section.

Relevant log output

Will try to attach via a dedicated file

[sedrichaus]

Attaching the complete log file with
--log-metrics
--debug
--debug-crt
flags enabled
mountpoint_s3.log

[dannycjones]

Hey @sedrichaus, sorry you're seeing this issue. Those logs do not give us any information on what happened, do they? 😅

Let's try this instead - I trust this issue is frequently reproducible.

Do not use --debug and --debug-crt, and instead set the environment variable MOUNTPOINT_LOG=trace,awscrt=warn. This will configure Mountpoint's logging to put everything out at trace level, excluding the AWS CRT. I suspect that the CRT is not the issue here. There's more information on this particular logging configuration in Mountpoint's logging documentation.

As an example shell execution:

MOUNTPOINT_LOG=trace,awscrt=warn mount-s3 bucket /data/s3/prefix

Can you share the new logs from that?

[monthonk]

I'm closing this issue for now. If you are still experiencing the issue and have more information you can share, please re-open it. Thank you.