ENH: Directly open compressed files

[cphyc]

PR Summary

This PR allows the user to directly load a dataset stored in a .tar.gz file (or any foramt supported by ratarmount).
It works by first mounting the .tar file as a read-only filesystem, which can then be used directly in yt.

PR Checklist

• New features are documented, with docstrings and narrative docs
• Adds a test for any bugs fixed. Adds tests for new features.

Example

# Create .tar file
tar czf output_00080.tar.gz output_00080

Now, one can load the dataset in the archive directly using

import yt
ds = yt.load_archive("output_00080.tar.gz", "output_00080")

[neutrinoceros]

here is a first batch of comments

[cphyc]

For reference, here is a simplistic benchmark:

Access	Time (s)
Direct	1.75
Through .tar.gz	5.57
Through .tar	2.03

This uses the following code:

>>>  %%timeit
... ds = yt.load("output_00080")
... ad = ds.all_data()
... ad["density"]
>>> %%timeit
... ds = yt.load_archive("output_00080.tar.gz", "output_00080")
... ad = ds.all_data()
... ad["density"]
>>> %%timeit
... ds = yt.load_archive("output_00080.tar", "output_00080")
... ad = ds.all_data()
... ad["density"]

Note that the first access is much slower as the entire tar file has to be read from disk.

[matthewturk]

@yt-fido test this please

[cphyc]

@yt-fido test this please

[neutrinoceros]

@yt-fido test this please

[cphyc]

pre-commit.ci autofix

[cphyc]

I don't really understand why it fails on Jenkins but it's really hard to find out without having direct access to it (it just doesn't seem to mount properly). I'd be happy to somehow mark to feature as experimental for the time being and see what happens with users on real cases. Wdyt @munkm @matthewturk @neutrinoceros?

[neutrinoceros]

I'd be happy to somehow mark to feature as experimental for the time being and see what happens with users on real cases. Wdyt @munkm @matthewturk @neutrinoceros?

I'd get behind such a plan, as long as it is impossible to use the feature without being warned that it's still experimental. It would also need to be clearly stated in docs and release notes, but this can be done later AFAIC.

[matthewturk]

That plan sounds good to me.

[cphyc]

@matthewturk and @neutrinoceros, I have added a warning and I am skipping the tests on Jenkins (at least, that's what I am hoping) as they are failing with a timeout and I cannot understand why (there must be some permission error, ...)

[neutrinoceros]

I see a minor flake8 issue so disabling auto-merge while I fix that.

[neutrinoceros]

now mypy is unhappy (I think this branch is older than type-checking so it's not too surprising). @cphyc, do you want to tackle these nits yourself ? Otherwise I'm happy to step in.