Set up config for CI runs

[lindnemi]

Just 1 scenario, no solar thermal.

Still this workflow takes ~30 minutes locally due to all the pre-processing. I guess some further simplifications are necessary for the CI runs, e.g. disabling "estimate_renewables" and using a default cutout. Here i would need some input from @fneum

[lkstrp]

@lindnemi config/scenarios.automated.yaml is set in the config bus but is missing in the repo

[lindnemi]

we usually generate it via

snakemake -j1 build_scenarios

[lindnemi]

I talked to Fabian and there are basically two ways to reduce the runtime of the workflow:

1. Use Github CI with Caching: The most compute intensive rule at the moment is build_renewable_profiles. We could cache the output similiar to how it is done here in the pypsa-eur repo:

https://github.com/PyPSA/pypsa-eur/blob/450f75a989dd9b84e27a7690a96107bdca4e8035/.github/workflows/ci.yaml#L75C1-L81C71

2. Forward the CI job to the z1 cluster and execute it there. @lkstrp would you know how to do that? The upside would be that we can use more resources and compute multiple scenarios.

[lkstrp]

Option 1 does not work because of disk space issues in a GitHub runner. There are some hacks around this, and using a default cutout would also help, but I think it's better to set up the workflow straight with the cluster. Also to be more future proof. I will dig into it.

[github-actions]

Validator Report

I am the Validator. Download all artifacts here.
I'll be back and edit this comment for each new commit.

Config
prefix: 20240801offwindconnectioncosts

Result plots comparison

	Base branch	Feature branch
primary_energy.png
secondary_energy.png
final_energy.png

Result files comparison

	Numeric	Status	NMAE	MAPE
20240731newcutout/KN2045_Bal_v4/csvs/curtailment.csv	72.2%	⚠️Changed	0.09	31.5%
20240731newcutout/KN2045_Bal_v4/csvs/nodal_supply_energy.csv	59.2%	✅ Almost equal	0.00	1.51e+07%
20240731newcutout/KN2045_Bal_v4/csvs/costs.csv	65.1%	✅ Almost equal	0.02	37.9%
20240731newcutout/KN2045_Bal_v4/csvs/nodal_cfs.csv	64.1%	✅ Almost equal	0.02	5.59e+02%
20240731newcutout/KN2045_Bal_v4/csvs/market_values.csv	77.9%	✅ Almost equal	0.03	1.6%
20240731newcutout/KN2045_Bal_v4/csvs/supply_energy.csv	64.7%	✅ Almost equal	0.03	46.7%
20240731newcutout/KN2045_Bal_v4/csvs/cfs.csv	72.3%	✅ Almost equal	0.04	3.3%
20240731newcutout/KN2045_Bal_v4/csvs/cumulative_cost.csv	84.7%	✅ Almost equal	0.09	0.0%
20240731newcutout/KN2045_Bal_v4/csvs/energy.csv	72.9%	✅ Almost equal	0.02	36.9%
20240731newcutout/KN2045_Bal_v4/csvs/prices.csv	81.1%	✅ Almost equal	0.00	1.4%
20240731newcutout/KN2045_Bal_v4/csvs/nodal_costs.csv	58.2%	✅ Almost equal	0.00	3.14e+07%
20240731newcutout/KN2045_Bal_v4/csvs/price_statistics.csv	57.1%	✅ Almost equal	0.08	0.9%
20240731newcutout/KN2045_Bal_v4/csvs/supply.csv	64.7%	✅ Almost equal	0.01	52.2%
20240731newcutout/KN2045_Bal_v4/csvs/nodal_capacities.csv	64.6%	✅ Almost equal	0.00	6.54e+06%
20240731newcutout/KN2045_Bal_v4/csvs/capacities.csv	72.6%	✅ Almost equal	0.01	45.1%
20240731newcutout/KN2045_Bal_v4/csvs/metrics.csv	54.5%	✅ Almost equal	0.05	16.5%
20240731newcutout/KN2045_Bal_v4/csvs/weighted_prices.csv		✅ Equal

MAPE: Mean Absolute Percentage Error
NMAE: Mean Absolute Error on Min-Max Normalized Data
Status Thresholds: NMAE > 0.05 and MAPE > 5%

Comparing ci-config (15efdb1) with main (e2e4149).
Branch is 2 commits ahead and 0 commits behind main.
Last updated on 2024-08-05 20:36:53 CEST.

[lkstrp]

Here we go. See the Validator Report above.

• It is triggered by any new commit in pr.
• Runs on a self-hosted github runner and solves on z1, so basically any configuration can be set. No restrictions. It just takes a while to update the comment if you choose a computationally intensive config.
• Compares the base branch (main) with the latest commit in the pr branch. So runs both, but also caches (in addition to snakemake caching) based on commit hashes. Which basically means that the main branch is not run redundantly.
• Edits one main comment when new commits are pushed to the repo, to avoid spamming.
• Set all settings in workflow file (mainly which plots to show, pre-commands, environment selection, etc.)
• I can adapt the template to whatever you want to see there. This is just what I came up with. @lindnemi
• Two main things to improve:
  • Right now environment changes are not handled automatically. I will add this at some point.
  • This is triggered on every commit. So if you just do a docs update, it will happen. This may not be a problem because of the snakemake caching. But I can also add some conditions to the trigger.

I expect bugs. There is a lot of persistent data. So I guess those will have to be figured out over time. Just be cautious with the results for now.

For pypsa-eur this would basically look the same @fneum @FabianHofmann.

[lindnemi]

Great work @lkstrp. I had a quick look yesterday and the images looked very helpful. However, now they seem to be gone. Does GitHub automatically delete them after X hours?

Regarding the results_files and plots: It would be nice if only the current prefix from the config.yaml is shown. At the moment it seems that files from other branches are shown as well.

Similarly, the validate.yml has the prefix hard coded in the key plots. Could that be filled automatically from the config?

I like the idea to show MAPE in the Status column. As long as it's small enough it can be ignored. However, for most files there seems to be a NAN error instead of the MAPE.

[lindnemi]

Regarding this:

Right now environment changes are not handled automatically. I will add this at some point.

I was thinking if we should have a snakemake rule in pypsa-eur which verifies that the current conda environment is up to date? I remember several puzzled bug hunts that could have been avoided by simplying upgrading the env file