config["electricity"]["estimate_renewable_capacities"]["enable"] == True incompatible with myopic foresight

[koen-vg]

Checklist

• I am using the current master branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-eur environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

I'm sure this is known amongst serious users of myopic foresight optimisation, but as it stands, the option to estimate existing renewable capacities in add_electricity.py is incompatible with myopic foresight. In particular, existing capacities are added twice; once in add_electricity.py and a second time in add_existing_baseyear.py.

The fix of course is just to turn off estimating existing renewable capacities in the electricity config section. But that option is turned on by default, presenting a certain pitfall for new users.

There are a couple of potential ways of dealing with this:

• Document the problem; expect people to read the documentation.
• Issue a warning for this combination of configuration.
• Do the above, but also automatically turn off estimating existing renewables in add_electricity.py if foresight is myopic.
• Remove any existing renewable capacities in add_existing_baseyear.py (or even in prepare_sector_network.py) before adding them back in (with correct build year etc.) in add_existing_baseyear.py.

Let me know which option seems the most appealing, and I'd be happy to submit a PR.

[fneum]

1. Document the problem; expect people to read the documentation.
2. Issue a warning for this combination of configuration.
3. Do the above, but also automatically turn off estimating existing renewables in add_electricity.py if foresight is myopic.
4. Remove any existing renewable capacities in add_existing_baseyear.py (or even in prepare_sector_network.py) before adding them back in (with correct build year etc.) in add_existing_baseyear.py.

In #1080, I went for option 3. I view this as a temporary fix, as we should uniformize the way existing capacities are added. It's messy because the merging of PyPSA-Eur and PyPSA-Eur-Sec is not 100% complete on that matter.

One caveat remains if resources (including outputs of add_electricity) are shared between scenarios with different foresight modes. I don't think this is a common modus operandi.

[koen-vg]

Great, thanks for the fix.

I'm sure you're aware of this now, but it seems to me that it's not 100% trivial to "merge" how renewables are added in electricity-only and sector-coupled models because of myopic foresight! Adding different generators for renewables with different build-years leads to a large increase in solving time and memory, so you wouldn't want to do that for single horizon optimisations.

This really implies things should be handled differently between overnight and myopic foresight but not necessarily electricity-only and sector-coupled networks; those things should be orthogonal. It's just that myopic foresight is only implemented for sector-coupled networks, and in fact the only way to run an overnight brownfield optimisation is to use myopic foresight anyway but with a single planning horizon (as far as I understand). I guess ideally one would want to untangle foresight from sector-coupling.

[lindnemi]

I am glad this issue got cleared up and fixed! It was throwing off the reporting of installed renewable capacities for pypsa-ariadne

[fneum]

Yes, I fully agree. I think the first problem is that the powerplants are added at drastically different stages in the workflow which makes it hard to keep an overview.

Maybe one could make the grouping of the year contingent on whether the foresight is myopic or not. Well, we'll figure it out at some point.