Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Industry workflow #238

Merged
merged 38 commits into from
Nov 27, 2023
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
b8beda6
adapting the industry scripts 1
hazemakhalek Aug 31, 2023
b49209a
distribute the industries with no geospatial locations to the GDP dis…
hazemakhalek Sep 12, 2023
85ff984
rewrite build_industrial_production_tomorrow.py
hazemakhalek Sep 15, 2023
d3142f3
adding industry growth cagr csv
hazemakhalek Sep 15, 2023
f149a17
adapt snakefile
hazemakhalek Sep 15, 2023
78c7f07
rename build_industry_production_tomorrow to build_base_industry_totals
hazemakhalek Sep 15, 2023
9508ff7
add documenetion to the script
hazemakhalek Sep 16, 2023
48761cd
remove hardcoded line
hazemakhalek Sep 16, 2023
47c6cdc
first complete version of build_base_industry_totals
hazemakhalek Sep 20, 2023
2948f80
fill csv with values for EG and fix small bug in base industry totals
hazemakhalek Sep 28, 2023
e392470
new industry workflow
hazemakhalek Oct 12, 2023
9ee15d3
Merge branch 'main' into industry_workflow
hazemakhalek Oct 12, 2023
8f39d06
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 12, 2023
91a6fff
remove unnecessary inputs to industry rule
hazemakhalek Oct 12, 2023
461758c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 12, 2023
cd1ac16
Update build_base_industry_totals.py
hazemakhalek Nov 8, 2023
a4153f4
add AL data for NG and BJ
hazemakhalek Nov 14, 2023
fcf3d05
enhance build_base_energy_totals.py
hazemakhalek Nov 14, 2023
6d665f7
fix bug in build_industry_demand to work for seveeral countries
hazemakhalek Nov 14, 2023
ec9d42c
fix minor bug in prepare sector network
hazemakhalek Nov 14, 2023
9babb9e
fix bug in build_base_industry_totals
hazemakhalek Nov 15, 2023
abe16b4
adapt snakefile
hazemakhalek Nov 15, 2023
10f15c7
adapt stest config
hazemakhalek Nov 15, 2023
dd998fe
Merge branch 'main' into industry_workflow
hazemakhalek Nov 15, 2023
49abb1f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 15, 2023
1e34539
test CI
hazemakhalek Nov 15, 2023
935cdd1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 15, 2023
818e479
fix CI 2
hazemakhalek Nov 15, 2023
dec1fe7
fix CI 3
hazemakhalek Nov 15, 2023
6f61d9f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 15, 2023
1b39b6a
Update prepare_sector_network.py
hazemakhalek Nov 21, 2023
8f358d2
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 21, 2023
7228de0
Update build_industry_demand.py
hazemakhalek Nov 21, 2023
8f3609c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 21, 2023
c118559
complete the Aluminium production data
hazemakhalek Nov 23, 2023
4da3fee
adding a warning message about the other industries
hazemakhalek Nov 23, 2023
0668ad1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2023
1dffd7a
remove wrong line in AL data
hazemakhalek Nov 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions Snakefile
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ rule build_population_layouts:
pop_layout_total="resources/population_shares/pop_layout_total.nc",
pop_layout_urban="resources/population_shares/pop_layout_urban.nc",
pop_layout_rural="resources/population_shares/pop_layout_rural.nc",
gdp_layout="resources/gdp_shares/gdp_layout.nc",
resources:
mem_mb=20000,
benchmark:
Expand All @@ -340,12 +341,14 @@ rule build_clustered_population_layouts:
pop_layout_total="resources/population_shares/pop_layout_total.nc",
pop_layout_urban="resources/population_shares/pop_layout_urban.nc",
pop_layout_rural="resources/population_shares/pop_layout_rural.nc",
gdp_layout="resources/gdp_shares/gdp_layout.nc",
regions_onshore=pypsaearth(
"resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson"
),
cutout=pypsaearth(CUTOUTS_PATH),
output:
clustered_pop_layout="resources/population_shares/pop_layout_elec_s{simpl}_{clusters}.csv",
clustered_gdp_layout="resources/gdp_shares/gdp_layout_elec_s{simpl}_{clusters}.csv",
resources:
mem_mb=10000,
benchmark:
Expand Down Expand Up @@ -586,9 +589,12 @@ if config["custom_data"].get("industry_demand", False) == True:
"resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson"
),
clustered_pop_layout="resources/population_shares/pop_layout_elec_s{simpl}_{clusters}.csv",
clustered_gdp_layout="resources/gdp_shares/gdp_layout_elec_s{simpl}_{clusters}.csv",
industrial_database="resources/custom_data/industrial_database.csv",
#shapes_path=pypsaearth("resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson")
shapes_path=PYPSAEARTH_FOLDER + "/resources/shapes/MAR2.geojson",
shapes_path=pypsaearth(
"resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson"
),
#shapes_path=PYPSAEARTH_FOLDER + "/resources/shapes/MAR2.geojson",
output:
industrial_distribution_key="resources/demand/industrial_distribution_key_elec_s{simpl}_{clusters}.csv",
threads: 1
Expand Down Expand Up @@ -625,6 +631,7 @@ if config["custom_data"].get("industry_demand", False) == False:
"resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson"
),
clustered_pop_layout="resources/population_shares/pop_layout_elec_s{simpl}_{clusters}.csv",
clustered_gdp_layout="resources/gdp_shares/gdp_layout_elec_s{simpl}_{clusters}.csv",
industrial_database="data/industrial_database.csv",
shapes_path=pypsaearth(
"resources/bus_regions/regions_onshore_elec_s{simpl}_{clusters}.geojson"
Expand All @@ -639,25 +646,29 @@ if config["custom_data"].get("industry_demand", False) == False:
script:
"scripts/build_industrial_distribution_key.py"

rule build_industrial_production_per_country_tomorrow: #default data
rule build_base_industry_totals: #default data
input:
industrial_production_per_country="data/industrial_production_per_country.csv",
#unsd_path="data/demand/unsd/data/",
energy_totals_base="data/energy_totals_base.csv",
output:
industrial_production_per_country_tomorrow="resources/demand/industrial_production_per_country_tomorrow_{planning_horizons}_{demand}.csv",
base_industry_totals="resources/demand/base_industry_totals_{planning_horizons}_{demand}.csv",
threads: 1
resources:
mem_mb=1000,
benchmark:
"benchmarks/build_industrial_production_per_country_tomorrow_{planning_horizons}_{demand}"
"benchmarks/build_base_industry_totals_{planning_horizons}_{demand}"
script:
"scripts/build_industrial_production_tomorrow.py"
"scripts/build_base_industry_totals.py"

rule build_industry_demand: #default data
input:
industry_sector_ratios="data/industry_sector_ratios.csv",
industrial_distribution_key="resources/demand/industrial_distribution_key_elec_s{simpl}_{clusters}.csv",
industrial_production_per_country_tomorrow="resources/demand/industrial_production_per_country_tomorrow_{planning_horizons}_{demand}.csv",
industrial_production_per_country="data/industrial_production_per_country.csv",
#industrial_production_per_country_tomorrow="resources/demand/industrial_production_per_country_tomorrow_{planning_horizons}_{demand}.csv",
#industrial_production_per_country="data/industrial_production_per_country.csv",
base_industry_totals="resources/demand/base_industry_totals_{planning_horizons}_{demand}.csv",
industrial_database="data/industrial_database.csv",
costs=CDIR
+ "costs_{}.csv".format(config["scenario"]["planning_horizons"][0]),
output:
Expand Down
2 changes: 2 additions & 0 deletions config.default.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ demand_data:
update_data: true # if true, the workflow downloads the energy balances data saved in data/demand/unsd/data again. Turn on for the first run.
base_year: 2019

other_industries: false # Whether or not to include industries that are not specified. some countries have has exageratted numbers, check carefully.
hazemakhalek marked this conversation as resolved.
Show resolved Hide resolved

H2_network: false
H2_network_limit: 2000 #GWkm
H2_repurposed_network: false
Expand Down
34 changes: 34 additions & 0 deletions data/AL_production.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
country,production[ktons/a],Year,source
hazemakhalek marked this conversation as resolved.
Show resolved Hide resolved
CN,36000,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
IN,3700,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
RU,3600,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
CA,2900,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
AE,2700,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
AU,1580,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
BH,1400,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
NO,1300,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
US,1100,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
SA,932,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
IS,850,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
MY,760,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
ZA,714,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
BR,659,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
QA,616,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
MZ,571,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
DE,529,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
AR,438,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
FR,380,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
OM,380,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
IR,370,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
ES,350,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
NZ,340,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
EG,317,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
RO,283,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
KZ,258,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
ID,242,2018,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
SK,212,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
GR,184,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
SE,126,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
VE,108,2019,https://en.wikipedia.org/wiki/List_of_countries_by_aluminium_production
NG,0,2019
BJ,0,2019
10 changes: 10 additions & 0 deletions data/demand/industry_growth_cagr.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
country,chemical and petrochemical,construction,food and tobacco,iron and steel,machinery,mining and quarrying,non-ferrous metals,non-metallic minerals,paper pulp and print,textile and leather,transport equipment,wood and wood products,other
MA,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
NG,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
BJ,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
NA,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
BR,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
AE,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
EG,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
CL,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
BH,0.03,0.015,0.02,0.017,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03,0.03
3 changes: 3 additions & 0 deletions data/export_ports.csv
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,6 @@ dummy port 1,BJ,1,6.47,2.63
dummy port 1,BR,1,-22.9068,-43.1729
dummy port 1,NA,1,-21.05431,13.50664
dummy port 1,AE,1,25.2048,55.2708
AS Suways,EG, 1, 29.966667,32.55
Damietta,EG,1,31.483333,31.75
Port Said,EG,3,31.266667,32.3
Loading
Loading