Add test and option to disable online retrieve rules

[euronion]

Changes proposed in this Pull Request

Minor improvement.

snakemake tries to access the internet for remote providers like HTTP while constructing the DAG. This prevents the execution of a workflow even when copies of all remote files are already locally available, as the DAG construction will simply fail.

This PR add's an option (config[enable][retrieve] = {auto,True,False}) and an automatic check (auto) on whether an internet connection can be made. If enabled/an internet connection is available, the retrieve.smk rules are included in the workflow. If disabled / no internet connection is available, the rules are not included, thus not part of the potential DAG and thus snakemake can run.

Encountered this problem locally when I had no internet connection and on a HPC cluster recently.

Interesting addition for PyPSA-EUR?

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• [n/a] Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in all of config.default.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv.
• A release note doc/release_notes.rst is added.

[FabianHofmann]

Very nice. I would propose to move the if else statements to retrieve.smk file in order to have the main snakefile clean. There is already a lot of if statements in the .smk, which can be expanded by retrieve_enabled, defined through

retrieve_enabled = config["enable"]["retrieve"]
if retrieve_enabled ==  "auto":
    if has_internet_access():
        retrieve_enabled = True
    else:
        print("No internet access provided. Datafile downloads automatically disabled.")
        retrieve_enabled = False

E.g. the retrieve cutout function would be adjusted to

if config["enable"].get("retrieve_cutout", True) and retrieve_enabled:
...

[euronion]

Thanks for the suggestion, RTR @FabianHofmann

[FabianHofmann]

great!