Use fsspec for fetching external testing files #178
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This enables external files stored on all of the systems supported by fsspec. Also includes: - Two `extra_requires` options. `stactools[s3]` installs `s3fs`, which is required to use fsspec on s3 hrefs. `stactools[all]` includes `s3` and any future extra_requires. - CI tests for `stactools[all]`, to make sure we are testing both with and without `extra_requires`. - Version bump for `fssepc`. - Unit tests for `stactools.testing`, which leads to saying "test" a lot :-). - README instructions for install w/ optional dependencies
|
Converting to draft because I'm going to add some kwargs to the external data structure, as I need to use an endpoint url as documented here: https://s3fs.readthedocs.io/en/latest/#self-hosted-s3. |
gadomski
force-pushed
the
feature/external-data-with-fsspec
branch
from
f49d32e to
da35a91
Compare
There's a package named `s3` so I couldn't use a dependency chain :-(.
The GOES data needed anon=True, and there's no way to default to that w/o an s3 config object, which we're already testing, so get it out of here.
|
Nice! This will be really useful. |
|
@gadomski Sorry for messing with your branch, but I updated the Dockerfile so that I could test this. I also got the |
cuttlefish
reviewed
Jul 23, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issue(s): n/a
Description: This enables external files stored on all of the systems supported by fsspec.
Also includes:
extra_requiresoptions.stactools[s3]installss3fs, which is required to use fsspec on s3 hrefs.stactools[all]includess3and any future extra_requires.stactools[all], to make sure we are testing both with and withoutextra_requires.fssepc.stactools.testing, which leads to saying "test" a lot :-).PR checklist:
scripts/format).scripts/lint).scripts/test).