Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add docs clarifications about which FS operations can be served from cache and when #627

Merged
merged 3 commits into from
Dec 8, 2023
Merged

Add docs clarifications about which FS operations can be served from cache and when #627

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ea2f1d4
Add clarification on which file system operations can be served from …
dannycjones Nov 23, 2023
ba1fd9e
Add clarification on caching behavior with writing
dannycjones Nov 23, 2023
d6d10b8
Update based on feedback
dannycjones Dec 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/CONFIGURATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ WantedBy=remote-fs.target
## Caching configuration

Mountpoint can optionally cache object metadata and content to reduce cost and improve performance for repeated reads to the same file.
Mountpoint can serve all file system requests from the cache, excluding listing of directory contents.

To enable caching, use the `--cache <CACHE_DIR>` command-line flag, specifying the directory in which to store cached object content.
This flag will also enable caching of metadata using a default time-to-live (TTL) of 1 second,
Expand Down
5 changes: 5 additions & 0 deletions doc/SEMANTICS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,11 @@ To force an up-to-date view of a file, use the `O_DIRECT` flag when opening the
When this option is provided, Mountpoint will check S3 to ensure the object exists and return the latest object content.
Unlike other file systems, Mountpoint does not support setting the `O_DIRECT` flag via `fcntl` after the file has been opened.

Caching does not affect the behavior of writing new files.
New files that are being written to remain unavailable for reading until the file is closed, consistent with behavior without caching.
After the new file is closed, it is possible to open it for reading.
Parts of the file that are read from S3 will then be cached and available for subsequent repeated reads.

## Durability

Mountpoint translates file operations like `read` and `write` into API calls to Amazon S3, which uses a combination of Content-MD5 checksums, secure hash algorithms (SHAs), and cyclic redundancy checks (CRCs) to verify data integrity. S3 performs these checksums on data at rest and repairs any disparity using redundant data. In addition, S3 calculates checksums on all internal network traffic to detect alterations of data packets when storing or retrieving data. However, POSIX file operations like `read` and `write` do not offer a built-in integrity mechanism. Like any file system operation, it is possible for data integrity to be lost in transit between your application and Mountpoint. If your application needs to verify data integrity, we recommend you use an AWS SDK instead of Mountpoint, and use [end-to-end checksums](https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/) for all object read and write operations.
Expand Down
Loading