Add support for caching, exporting, importing bucket list

[tchaton]

Tell us more about this new feature.

Hey there,

Assuming a dataset isn't going to change, I would like to configure mountpoint-s3 to keep the bucket listing in RAM or stored within a file (preferred as it can be collected and recycled for later re-mount).

Right now, I need to store the listing to a file to avoid occurring the cost of listing again (up to 20min on reasonable dataset).

    def _cached_list_filepaths(self, root: str) -> List[str]:
        algo = hashlib.new("sha256")
        algo.update(root.encode("utf-8"))
        root_hash = algo.hexdigest()

        filepath = f"/cache/{root_hash}/filepaths.txt"

        if os.path.exists(filepath):
            lines = []
            with open(filepath) as f:
                for line in f.readlines():
                    lines.append(line.replace("\n", ""))
            return lines

        filepaths = []
        for dirpath, _, filenames in os.walk(root):
            for filename in filenames:
                filepaths.append(os.path.join(dirpath, filename))

        os.makedirs(os.path.dirname(filepath), exist_ok=True)

        with open(filepath, "w") as f:
            for filepath in filepaths:
                f.write(f"{filepath}\n")

        return filepaths

Additionally, I would love to pay the cost of listing only one ever. Ideally, I would like the ability to export and import filesystem index.

[dannycjones]

Interesting feature request! Serving directory listing from cache is something we've been considering as part of #255, but ultimately won't be part of the initial solution. However, the entries we discover as part of a directory listing will be included in subsequent lookup, open, etc. requests.

Leaving this open to track as a further enhancement in the future.

[tchaton]

Keep me updated on this @dannycjones

This is one of the major practical usage of mountpoint-s3 for any serious deep learning training. Here is another related issue: #554

[tchaton]

@dannycjones Any updates ?

[dannycjones]

@dannycjones Any updates ?

Not on this issue in particular. Work is on-going on #255, with metadata caching (for open, etc; but not list) available under a build-time feature flag. It isn't ready for production use yet but can be built from source for non-production testing.