Allow --metadata-ttl without --cache and set default with --cache to 60s

[passaro]

Description of change

Allow users to explicitly enable metadata caching, independently of object content caching, by setting the --metadata-ttl option. Before this change, users were required to also set --cache <DIR> in order to configure the metadata TTL. To enable just the metadata cache they had to resort to the workaround of restricting the cache size to zero (--max-cache-size 0).
We are also introducing the following special values for --metadata-ttl:

• minimal: describe the default setting (when --cache is not used) where Mountpoint caches metadata for up to 1s, refreshes it on open, and does not keep a negative cache.
• indefinite: new value recommended for users with static buckets, disables refreshing of metadata. (Implemented by setting the TTL to the maximum allowed value, ~100 years).

When --cache is set, we are now also setting the default TTL for metadata to 60s (1min) vs the previous 1s.

Finally, setting --metadata-ttl 0 now resolves to minimal and logs a warning to notify users.

The caching options section looks like this:

Caching options:
      --cache <DIRECTORY>
          Enable caching of object content to the given directory and set metadata TTL to 60s
      --metadata-ttl <SECONDS|indefinite|minimal>
          Time-to-live (TTL) for cached metadata in seconds [default: minimal, or 60s if --cache is set]
      --max-cache-size <MiB>
          Maximum size of the cache directory in MiB [default: preserve 5% of available space]

Relevant issues: #752, #759, #768

Does this change impact existing behavior?

Two changes to existing behavior:

1. When setting --cache but not --metadata-ttl, the default TTL is 60s vs the previous 1s.
2. Setting --metadata-ttl 0 resolves to minimal and logs a warning.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

[dannycjones]

Looks good!

Before merging:

• Add changelog entry
• Update CONFIGURATION.md https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#caching-configuration

[jamesbornholt]

Just some minor stuff

[jamesbornholt]

One minor suggestion, and then needs a rebase.

[jamesbornholt]

LGTM, just one changelog thing.

[jamesbornholt]

Maybe split this up because only part of it is breaking

New features

• Metadata caching can now be configured independently of data caching. When passing the --metadata-ttl <seconds> argument without also specifying --cache <directory>, Mountpoint will cache file metadata in memory for up to the given TTL, but will not cache object data. The --metadata-ttl argument also accepts two special values: minimal to enable only the minimal necessary caching, and indefinite to cache indefinitely. These modes can help accelerate workloads that touch many files but do not need to cache object data for re-use (for example, listing a directory and then reading each file within it once).

Breaking changes

• The --metadata-ttl 0 setting is no longer supported and will be removed in a future release. The new --metadata-ttl minimal has a similar effect, but behaves better when latency for S3 requests is high.

[passaro]

I've also added a breaking change entry for the new default TTL with cache.