Update calculations of retrofit costs

[ekatef]

The PR is aimed to update retrofit calculations to make the model work with retro_endogen: false. The main reasons why updates are needed is that data structure may have been altered and pandas version has changed.

Changes proposed in this Pull Request

• replace pandas.append() was deprecated for a while and has been removed since pandas 2.0;
• fix structure of area_tot (= total area) grouped dataframe to avoid messed concatenated columns;
• fix a column name which contains subsector names for a building dataset: building_data.detail currently contains only [nan 'insulation' 'low-e'], while the script looks for "Totals" value which is currently contained in building_data.subsector;
• deal with AssertionError: Length of new_levels (2) must be same as self.nlevels (3) which is currently arisen by reindex() function for prepare_building_topology( ) and map_tabula_to_hotmaps( ). That seems to be caused BUG: Regression in 2.1.0 when inserting usign MultiIndex frames pandas-dev/pandas#54909 for operation on insertion of smaller MultiIndex into larger MultiIndex and has been fixed recently with REGR: setitem with part of a MultiIndex raises pandas-dev/pandas#54885, which is one of milestones of pandas 2.1.1;
• fix country codes in building_data to avoid messing-up "GB" and "UK";
• remove duplicated country column in area_reordered;
• fix call of difference( ) function;
• fix computation of grouped means and total averages;
• replace a level name with a number of the level.

Apart of the code fixes, the datakit structure should be ajusted a bit: tabula-calculator-calcsetbuilding.csv should be moved to /pypsa-eur/data/retro, where the workflow expects to find it.

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to envs/environment.yaml.
• Changes in configuration options are added in all of config.default.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv.
• A release note doc/release_notes.rst is added.

[FabianHofmann]

Thanks, @ekatef for the PR. Looks good! CI is failing due to some internet connection problems, but let's wait till it's through. @lisazeyen, you approve?

[martacki]

Can confirm that the workflow runs with suggested data structure changes. But instead of moving the tabula data around, we could also change the respective path. Then the sector-bundle doesn't need updating (might be confusing if the files appears twice), i.e.
in rules/build_sector.smk, change

data_tabula="data/retro/tabula-calculator-calcsetbuilding,csv",

to

data_tabula="data/bundle-sector/retro/tabula-calculator-calcsetbuilding.csv",

Thoughts ?

[ekatef]

Can confirm that the workflow runs with suggested data structure changes. But instead of moving the tabula data around, we could also change the respective path. Then the sector-bundle doesn't need updating (might be confusing if the files appears twice), i.e. in rules/build_sector.smk, change

data_tabula="data/retro/tabula-calculator-calcsetbuilding,csv",

to

data_tabula="data/bundle-sector/retro/tabula-calculator-calcsetbuilding.csv",

Thoughts ?

Agree, fixed.

[lisazeyen]

Thanks, @ekatef for the PR. Looks good! CI is failing due to some internet connection problems, but let's wait till it's through. @lisazeyen, you approve?

Looks good to me, only the CI needs to run through

[fneum]

None of the changes affect the CI, since retrofitting path is not covered by tests. Therefore, I am merging without passing tests (URL connection error).

[martacki]

.