Adds auto object store creation to get_file
#1750
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
I have had to write code that autoparses the URI to create an
S3ObjectStore
and then callsget_file
with that newly created object store a couple of times, and the util to pull huggingface tokenizer/model out of a composer checkpoint is going to need to do the same, so I added the same auto parsing support as in trainer to theget_file
function. It shouldn't change any behavior of any existing code interacting withget_file
.Manual tested that I can download a file from S3 using this, and also that checkpointing/autoresume from S3 still works.
What issue(s) does this change relate to?
Part of CO-1110
Before submitting
pre-commit
on your change? (see thepre-commit
section of prerequisites)