Add a reproducible workflow strategy compatible with soft-fork (#19)

* Add OET flavor to repository to enable reproducible projects

* Remove cookiecutter references to clean the code

* Remove plots directory as graphics should be in results

* Remove raw environment

* Add the report in the all rule

* Improve licensing scheme

* Add AGPL-3.0-or-latex license

* Remove reference to submodule

* Add reference to micromamba and conda

* Update README.md

Co-authored-by: Siddharth Krishna <siddharth-krishna@users.noreply.github.com>

* Update README.md

Co-authored-by: Siddharth Krishna <siddharth-krishna@users.noreply.github.com>

* Update README.md

* Add reference to soft-fork strategy

* Switch default license from AGPL to MIT

---------

Co-authored-by: Siddharth Krishna <siddharth-krishna@users.noreply.github.com>

.gitignore

@@ -10,6 +10,7 @@ __pycache__
 *dconf
 gurobi.log
 .vscode
+.idea
 *.orig
 
 /bak

.reuse/dep5

@@ -30,3 +30,7 @@ License: CC0-1.0
 Files: borg-it
 Copyright: 2017-2024 The PyPSA-Eur Authors
 License: CC0-1.0
+
+Files: report/*
+Copyright: Open Energy Transition gGmbH
+License: MIT

README.md

@@ -13,6 +13,62 @@ SPDX-License-Identifier: CC-BY-4.0
 [![REUSE status](https://api.reuse.software/badge/github.com/pypsa/pypsa-eur)](https://api.reuse.software/info/github.com/pypsa/pypsa-eur)
 [![Stack Exchange questions](https://img.shields.io/stackexchange/stackoverflow/t/pypsa)](https://stackoverflow.com/questions/tagged/pypsa)
 
+# {{project_name}}
+<img src="https://raw.githubusercontent.com/open-energy-transition/oet-website/main/assets/img/oet-logo-red-n-subtitle.png" alt="Open Energy Transition Logo" width="260" height="100" align="right">
+
+This repository is a soft-fork of [PyPSA-Eur](https://github.com/pypsa/pypsa-eur) and contains the entire project `{{project_name}}` supported by [Open Energy Transition (OET)](https://openenergytransition.org/)<sup>*</sup>, including code and report. The philosophy behind this repository is that no intermediary results are included, but all results are computed from raw data and code. The structure is also inspired by [cookiecutter-project](https://github.com/PyPSA/cookiecutter-project).
+
+This repository is maintained using [OET's soft-fork strategy](https://open-energy-transition.github.io/handbook/docs/Engineering/SoftForkStrategy). OET's primary aim is to contribute as much as possible to the open source (OS) upstream repositories. For long-term changes that cannot be directly merged upstream, the strategy organizes and maintains OET forks, ensuring they remain up-to-date and compatible with upstream, while also supporting future contributions back to the OS repositories.
+
+
+# Repository structure
+
+* `benchmarks`: will store `snakemake` benchmarks (does not exist initially)
+* `config`: configurations used in the study
+* `cutouts`: will store raw weather data cutouts from `atlite` (does not exist initially)
+* `data`: includes input data that is not produced by any `snakemake` rule
+* `doc`: includes all files necessary to build the `readthedocs` documentation of PyPSA-Eur
+* `envs`: includes all the `mamba` environment specifications to run the workflow
+* `logs`: will store log files (does not exist initially)
+* `notebooks`: includes all the `notebooks` used for ad-hoc analysis
+* `report`: contains all files necessary to build the report; plots and result files are generated automatically
+* `rules`: includes all the `snakemake`rules loaded in the `Snakefile`
+* `resources`: will store intermediate results of the workflow which can be picked up again by subsequent rules (does not exist initially)
+* `results`: will store the solved PyPSA network data, summary files and plots (does not exist initially)
+* `scripts`: includes all the Python scripts executed by the `snakemake` rules to build the model
+
+# Installation and Usage
+
+## 1. Installation
+
+Clone the repository:
+
+    git clone https://github.com/open-energy-transition/{{repository}}
+
+You need [mamba](https://mamba.readthedocs.io/en/latest/) to run the analysis. Users may also prefer to use [micromamba](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html) or [conda](https://docs.conda.io/projects/conda/en/latest/index.html). Using `mamba`, you can create an environment from within you can run it:
+
+    mamba env create -f environment.yaml
+
+Activate the newly created `{{project_short_name}}` environment:
+
+    mamba activate {{project_short_name}}
+
+## 2. Run the analysis
+
+    snakemake -call
+
+This will run all analysis steps to reproduce results and build the report.
+
+To generate a PDF of the dependency graph of all steps `resources/dag.pdf` run:
+
+    snakemake -c1 dag
+
+<sup>*</sup> Open Energy Transition (g)GmbH, Königsallee 52, 95448 Bayreuth, Germany
+
+----
+
+----
+
 # PyPSA-Eur: A Sector-Coupled Open Optimisation Model of the European Energy System
 
 PyPSA-Eur is an open model dataset of the European energy system at the
@@ -105,4 +161,4 @@ We strongly welcome anyone interested in contributing to this project. If you ha
 The code in PyPSA-Eur is released as free software under the
 [MIT License](https://opensource.org/licenses/MIT), see [`doc/licenses.rst`](doc/licenses.rst).
 However, different licenses and terms of use may apply to the various
-input data, see [`doc/data_sources.rst`](doc/data_sources.rst).
+input data, see [`doc/data_sources.rst`](doc/data_sources.rst).

Snakefile

@@ -54,6 +54,7 @@ include: "rules/solve_electricity.smk"
 include: "rules/postprocess.smk"
 include: "rules/validate.smk"
 include: "rules/development.smk"
+include: "rules/report.smk"
 
 
 if config["foresight"] == "overnight":
@@ -74,6 +75,7 @@ if config["foresight"] == "perfect":
 rule all:
     input:
         expand(RESULTS + "graphs/costs.svg", run=config["run"]["name"]),
+        "report/report.pdf"
     default_target: True
 
 

report/references.bib

@@ -0,0 +1,8 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+@misc{ Nobody06,
+       author = "Nobody Jr",
+       title = "My Article",
+       year = "2006" }

report/report.tex

@@ -0,0 +1,205 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\documentclass[5p]{elsarticle}
+
+\journal{TBA}
+
+\bibliographystyle{elsarticle-num}
+\biboptions{numbers,sort&compress}
+
+% format hacks
+\usepackage{libertine}
+\usepackage{libertinust1math}
+% \usepackage{geometry}
+% \geometry{
+%     top=30mm,
+%     bottom=35mm,
+% }
+
+\usepackage{amsmath}
+\usepackage{bbold}
+\usepackage{graphicx}
+\usepackage{eurosym}
+\usepackage{mathtools}
+\usepackage{url}
+\usepackage{booktabs}
+\usepackage{epstopdf}
+\usepackage{xfrac}
+\usepackage{tabularx}
+\usepackage{bm}
+\usepackage{subcaption}
+\usepackage{blindtext}
+\usepackage{longtable}
+\usepackage{multirow}
+\usepackage{threeparttable}
+\usepackage{pdflscape}
+\usepackage[export]{adjustbox}
+\usepackage[version=4]{mhchem}
+\usepackage[colorlinks]{hyperref}
+\usepackage[parfill]{parskip}
+\usepackage[nameinlink,sort&compress,capitalise,noabbrev]{cleveref}
+\usepackage[leftcaption,raggedright]{sidecap}
+\usepackage[prependcaption,textsize=footnotesize]{todonotes}
+
+\usepackage{siunitx}
+\sisetup{
+	range-units = single,
+	per-mode = symbol
+}
+\DeclareSIUnit\year{a}
+\DeclareSIUnit{\tco}{t_{\ce{CO2}}}
+\DeclareSIUnit{\sieuro}{\mbox{\euro}}
+
+\usepackage{lipsum}
+
+\usepackage[resetlabels,labeled]{multibib}
+\newcites{S}{Supplementary References}
+\bibliographystyleS{elsarticle-num}
+
+\graphicspath{
+    {../results/graphics/},
+}
+
+% \usepackage[
+% 	type={CC},
+% 	modifier={by},
+% 	version={4.0},
+% ]{doclicense}
+
+\newcommand{\abs}[1]{\left|#1\right|}
+\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
+\newcommand{\set}[1]{\left\{#1\right\} }
+\DeclareMathOperator*{\argmin}{\arg\!\min}
+\newcommand{\R}{\mathbb{R} }
+\newcommand{\B}{\mathbb{B} }
+\newcommand{\N}{\mathbb{N} }
+\newcommand{\co}{\ce{CO2}~}
+\def\el{${}_{\textrm{el}}$}
+\def\th{${}_{\textrm{th}}$}
+\def\deg{${}^\circ$}
+
+\begin{document}
+
+	\begin{frontmatter}
+
+	\title{ project name }
+
+	\author[affil]{ name surname } % \corref{correspondingauthor}
+	\ead{ name.surname@openenergytransition.org }
+
+	\address[affil]{ affiliation }
+
+	\begin{abstract}
+		\input{sections/abstract.tex}
+	\end{abstract}
+
+	\begin{keyword}
+		TODO
+	\end{keyword}
+
+	% \begin{graphicalabstract}
+	% \end{graphicalabstract}
+
+	\begin{highlights}
+		\item A
+		\item B
+		\item C
+	\end{highlights}
+
+\end{frontmatter}
+
+% \listoftodos[TODOs]
+
+% \tableofcontents
+
+\section{Introduction}
+\label{sec:intro}
+
+\input{sections/introduction.tex}
+
+\section{Methods}
+\label{sec:methods}
+
+\input{sections/methods.tex}
+
+\section{Results}
+\label{sec:results}
+
+\input{sections/results.tex}
+
+\section{Discussion}
+\label{sec:discussion}
+
+\input{sections/discussion.tex}
+
+\section{Conclusion}
+\label{sec:conclusion}
+
+\input{sections/conclusion.tex}
+
+\section*{Acknowledgements}
+
+% XY gratefully acknowledge funding from XY under grant no. 00000.
+
+\section*{License}
+% \doclicenseLongText
+% \doclicenseIcon
+
+\section*{Author Contributions}
+
+% following https://casrai.org/credit/
+
+\textbf{Author XY}:
+Conceptualization --
+Data curation --
+Formal Analysis --
+Funding acquisition --
+Investigation --
+Methodology --
+Project administration --
+Resources --
+Software --
+Supervision --
+Validation --
+Visualization --
+Writing - original draft --
+Writing - review \& editing
+
+\section*{Data and Code Availability}
+
+A dataset of the model results is available at \href{zenodoTBA}{doi:zenodoTBA}.
+The code to reproduce the experiments is available at \href{https://github.com/repository}{github.com/repository}.
+
+% tidy with https://flamingtempura.github.io/bibtex-tidy/
+\addcontentsline{toc}{section}{References}
+\renewcommand{\ttdefault}{\sfdefault}
+\bibliography{references}
+
+% supplementary information
+
+\newpage
+
+\makeatletter
+\renewcommand \thesection{S\@arabic\c@section}
+\renewcommand\thetable{S\@arabic\c@table}
+\renewcommand \thefigure{S\@arabic\c@figure}
+\makeatother
+\renewcommand{\citenumfont}[1]{S#1}
+\setcounter{equation}{0}
+\setcounter{figure}{0}
+\setcounter{table}{0}
+\setcounter{section}{0}
+
+
+\section{Supplementary Information}
+\label{sec:si}
+
+\input{sections/supplementary.tex}
+
+\addcontentsline{toc}{section}{Supplementary References}
+\renewcommand{\ttdefault}{\sfdefault}
+\bibliographyS{references}
+
+\end{document}

report/sections/.gitkeep

@@ -0,0 +1,3 @@
+# SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+#
+# SPDX-License-Identifier: CC0-1.0

report/sections/abstract.tex

@@ -0,0 +1,5 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\lipsum[1]

report/sections/conclusion.tex

@@ -0,0 +1,5 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\lipsum[1]

report/sections/discussion.tex

@@ -0,0 +1,5 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\lipsum[1]

report/sections/introduction.tex

@@ -0,0 +1,5 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\lipsum[1]

report/sections/methods.tex

@@ -0,0 +1,6 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\lipsum[1]
+Blablabla said Nobody ~\cite{Nobody06}.

report/sections/results.tex

@@ -0,0 +1,5 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\lipsum[1]

report/sections/supplementary.tex

@@ -0,0 +1,5 @@
+% SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+%
+% SPDX-License-Identifier: MIT
+
+\lipsum[1]

report/static/.gitkeep

@@ -0,0 +1,3 @@
+# SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+#
+# SPDX-License-Identifier: CC0-1.0

rules/report.smk

@@ -0,0 +1,19 @@
+# SPDX-FileCopyrightText:  Open Energy Transition gGmbH
+#
+# SPDX-License-Identifier: MIT
+
+rule report:
+    message: "Compile report."
+    params:
+        fn="report"
+    input:
+        tex="report/report.tex",
+        bib="report/references.bib"
+    output: "report/report.pdf"
+    shell:
+        """
+        pdflatex -output-directory report {input.tex}
+        cd report; bibtex {params.fn}; cd ..
+        pdflatex -output-directory report {input.tex}
+        pdflatex -output-directory report {input.tex}
+        """