Implement fs copy from HTTP

[maddenp-noaa]

Synopsis

Fixes #667.

See updated docs for an example but, in short, for file copies (CLI uw fs copy, API uwtools.api.fs.copy()), interpret paths starting with http:// or https:// as URLs of remote files and fetch them to the local filesystem with requests. For example:

$ echo "gpl: https://www.gnu.org/licenses/gpl.txt" | uw fs copy --target-dir licenses
[2024-12-07T01:12:25]     INFO Validating config against internal schema: files-to-stage
[2024-12-07T01:12:25]     INFO 0 UW schema-validation errors found in fs config
[2024-12-07T01:12:25]     INFO File copies: Initial state: Not Ready
[2024-12-07T01:12:25]     INFO File copies: Checking requirements
[2024-12-07T01:12:25]     INFO Copy https://www.gnu.org/licenses/gpl.txt -> licenses/gpl: Initial state: Not Ready
[2024-12-07T01:12:25]     INFO Copy https://www.gnu.org/licenses/gpl.txt -> licenses/gpl: Checking requirements
[2024-12-07T01:12:26]     INFO Copy https://www.gnu.org/licenses/gpl.txt -> licenses/gpl: Requirement(s) ready
[2024-12-07T01:12:26]     INFO Copy https://www.gnu.org/licenses/gpl.txt -> licenses/gpl: Executing
[2024-12-07T01:12:26]     INFO Copy https://www.gnu.org/licenses/gpl.txt -> licenses/gpl: Final state: Ready
[2024-12-07T01:12:26]     INFO File copies: Final state: Ready

$ tree licenses
licenses
└── gpl

1 directory, 1 file

Unavailable upstream files are detected, leading to a no-op:

$ echo "gpl: https://www.gnu.org/licenses/nope.txt" | uw fs copy --target-dir licenses
[2024-12-07T01:16:41]     INFO Validating config against internal schema: files-to-stage
[2024-12-07T01:16:41]     INFO 0 UW schema-validation errors found in fs config
[2024-12-07T01:16:41]     INFO File copies: Initial state: Not Ready
[2024-12-07T01:16:41]     INFO File copies: Checking requirements
[2024-12-07T01:16:41]     INFO Copy https://www.gnu.org/licenses/nope.txt -> licenses/gpl: Initial state: Not Ready
[2024-12-07T01:16:41]     INFO Copy https://www.gnu.org/licenses/nope.txt -> licenses/gpl: Checking requirements
[2024-12-07T01:16:41]  WARNING Remote object https://www.gnu.org/licenses/nope.txt: State: Not Ready (external asset)
[2024-12-07T01:16:42]     INFO Copy https://www.gnu.org/licenses/nope.txt -> licenses/gpl: Requirement(s) not ready
[2024-12-07T01:16:42]  WARNING Copy https://www.gnu.org/licenses/nope.txt -> licenses/gpl: Final state: Not Ready
[2024-12-07T01:16:42]  WARNING File copies: Final state: Not Ready

$ tree licenses
licenses  [error opening dir]

0 directories, 0 files

Type

• Documentation
• Enhancement (adds new functionality)

Impact

• This is a non-breaking change (existing functionality continues to work as expected)

Checklist

• I have added myself and any co-authors to the PR's Assignees list.
• I have reviewed the documentation and have made any updates necessitated by this change.

[christinaholtNOAA]

LGTM.

Probably a dumb question, but this code is the back end for the fs tool and the files_to_copy/files_to_link sections for drivers, right?

[maddenp-noaa]

@christinaholtNOAA It's actually a great question: The core support is indeed implemented in code deeper in the library than the drivers' files_to_copy / files_to_link support. For example the common MPAS driver code calls filecopy() from uwtools.utils.tasks, which now supports both local-filesystem and HTTP source paths. But the correctness checks on the source and destination paths are assumed to have already been done, and I only implemented those for the uw fs mode code -- not in drivers. So I think I need to generalize that. Good catch!