Optimized dispatches from HERON to HYBRID

doc/user_manual/src/ExternalCodeIntegration.tex

@@ -1,12 +1,13 @@
 \section{External Code Integration}
-HERON can exchange data with other Integrated Energy Systems (IES) codes to conduct technical or economic analyses for various electricity market structures. More information about integrating HERON with other IES software is here: \url{https://ies.inl.gov/SitePages/FORCE.aspx}
+HERON can exchange data with other Integrated Energy Systems (IES) tools to conduct technical or economic analyses for various electricity market structures. More information about integrating HERON with other IES software is here: \url{https://ies.inl.gov/SitePages/FORCE.aspx}
 
-The integration between HERON and other IES tools is still a work in progress. Currently, HERON can communicate with the following codes
+Integrating HERON and other IES tools is still a work in progress. Currently, HERON can communicate with HYBRID only. HERON provides algorithms for analyzing the long-term viability of potential IES technologies, while HYBRID provides models to achieve high-resolution analysis over a short time. More information about HYBRID is here: \url{https://github.com/idaholab/HYBRID}
 
-\subsection{HYBRID}
-The HYBRID repository contains a collection of models representing the physical dynamics of various integrated energy systems and processes. HERON has the capability to load the economic information about the components of the grid system automatically from HYBRID. More information about HYBRID is here: \url{https://github.com/idaholab/HYBRID}
 
-An example that demonstrates this capability can be found at: 
+HYBRID models determine the fundamental properties of the IES components. These fundamental properties include transfer functions, operational ramping limits, and economics. The fundamental properties of generating units are used in HERON as constraints and economic drivers for long-term portfolio optimization. On the other side, the feasibility of the optimized energy portfolio, calculated by HERON, is analyzed using HYBRID to ensure that no physical or operational constraints are violated. HYBRID and HERON integration through automating the data exchange between HYBRID and HERON is demonstrated in the following subsections.
+
+\subsection{Auto-loading the components' economic information from HYBRID to HERON}
+HERON can load the economic information about the components of the grid. An example that demonstrates this capability can be found at: 
 \begin{lstlisting}
 /HERON/tests/integration_tests/mechanics/hybrid_load/
 \end{lstlisting}
@@ -121,4 +122,76 @@ \subsubsection{HYBRID and HERON keywords}
  For example, the HYBRID variable, \verb|"Activity"|, will be included if the \verb|"VOM"| cash flow is present since the node \xmlNode{CashFlow name="VOM"} will be incomplete if the \xmlNode{activity} sub-node is missing. Therefore, for the HYBRID variable, \verb|"Activity"|, the corresponding value under the \verb|"Required if HYBRID keyword?"| column is \verb|"VOM"|
 
  \item \textbf{Default value for the required variable}: This column assigns a default value for any HYBRID variable whose value is not provided by the HYBRID text files if this HYBRID variable is required (see the column \verb|"Required if HYBRID keyword?"|). For example, the default value of the HYBRID variable, \verb|"Activity"|, if required, is \xmlString{electricity} 
-\end{itemize}
+\end{itemize}
+
+\subsection{Auto-loading the optimized dispatches from HERON to HYBRID}
+The optimized dispatches, that HERON calculates, can be converted to a text file compatible with HYBRID. An example that demonstrates this capability can be found at: 
+\begin{lstlisting}
+/HERON/tests/integration_tests/mechanics/optimizedDispatch2Hybrid/
+\end{lstlisting}
+
+
+
+This test demonstrates converting the HERON optimized components' variables (optimized dispatch) to a text file that is compatible with HYBRID via the following three steps:
+\begin{enumerate}
+
+\item \textbf{Producing the optimized dispatches}: Initially, the optimized dispatches are calculated via HERON through two steps (two optimization loops). An outer loop for optimizing the sizes of the grid components/units and an inner loop that calculates the optimal energy dispatch. These two steps (loops) create the optimized dispatches CSV file by running the usual HERON commands such as:
+\begin{lstlisting}
+~/projects/HERON/heron heron_input.xml
+~/projects/raven/raven_framework outer.xml
+\end{lstlisting}
+Note that this HERON input XML file, \path{heron_input.xml}, is borrowed from \path{/HERON/tests/integration_tests/mechanics/debug_mode/} and is run in debug mode. The debug mode enables a reduced-size run (a single outer loop and a single inner loop). 
+
+The output of the previous two commands is the optimized dispatch printed in
+\begin{lstlisting}
+optimizedDispatch2Hybrid/Debug_Run_o/dispatch_print.csv
+\end{lstlisting}
+
+
+
+\item \textbf{Creating the user-input file for the HYBRID user}: This step enables the HYBRID user to change the names, if necessary, of the optimized components' variables and the capacities of the components. The HYBRID user input file is created by running the code \path{create_user_input.py} which extracts the optimized dispatch outputs from the dispatches CSV file and the list of capacities of the components from the HERON input XML file.
+
+
+The \path{create_user_input.py} takes two arguments: The HERON input XML file and the optimized dispatch outputs CSV file. For example, use the following command:
+ \begin{lstlisting}
+python create_user_input.py heron_input.xml Debug_Run_o/
+dispatch_print.csv
+  \end{lstlisting}
+
+
+
+This step creates \path{user_input.txt} that the HYBRID user should modify or review before moving to the next step. The \path{user_input.txt} includes a list of the HERON variables and their corresponding HYBRID variables (that the HYBRID user may change) plus the location of \path{all_variables_file}. The \path{all_variables_file} is a file that includes all the HYBRID variables. All the capacities and dispatches should be a subset of this file's variables. This file helps to ensure that the HYBRID capacities and dispatches are identified by HYBRID. Currently the \path{all_variables_file} is {"dsfinal.txt"} which is located in 
+\begin{lstlisting}
+/HERON/tests/integration_tests/mechanics/
+optimizedDispatch2Hybrid/dsfinal.txt
+\end{lstlisting}
+The user can change the file location in the \path{user_input.txt}.    
+
+
+
+\item \textbf{Creating an optimized dispatches file for HYBRID}: A text file containing the optimized dispatches and capacities of the components is created to be compatible with HYBRID using the code \path{export2Hybrid.py}. The variables' names in the generated (auto-loaded) HYBRID text file are borrowed from the \path{user_input.txt} and the \path{user_input.txt} must be in the same folder as the \path{export2Hybrid.py}. 
+ The \path{export2Hybrid.py} extracts the values of the capacities of the components from the HERON input file, \path{heron_input.xml}, and extracts the most challenging scenario from the optimized dispatch CSV file, \path{Debug_Run_o/dispatch_print.csv}. 
+
+ We assume that the most challenging scenario is the scenario (of a specific year and sample) with the highest rate of change of any components' resource (variable). This most challenging scenario is selected as follows:
+
+
+\begin{itemize}
+ \item For each scenario, calculate the rate of change at each time step for each component variable
+  \item For each scenario, calculate the maximum rate of change over all the time steps over all the variables. 
+   \item Select the scenario with the highest maximum rate of change to be the maximum scenario.
+ \end{itemize}
+
+
+
+Note that we assume that all the components' variables in the \path{dispatch_print.csv} have the same units. 
+
+ The \path{export2Hybrid.py} is run with two arguments: The HERON input XML file and the optimized dispatch outputs CSV file. For example, use the following command:
+
+ \begin{lstlisting}
+python export2Hybrid.py heron_input.xml Debug_Run_o/
+dispatch_print.csv
+ \end{lstlisting}
+
+The output will be a text file compatible with HYBRID:    \path{hybrid_compatible_dispatch.txt}. The terminal will display warning messages if the HYBRID dispatches/capacities are not a subset of the variables list in the \path{user_input.txt}.    
+
+\end{enumerate}

tests/integration_tests/mechanics/optimizedDispatch2Hybrid/ReadMe.txt

@@ -0,0 +1,43 @@
+This test demonstrates converting the HERON optimized components' variables (optimized dispatch) to a text file that is compatible with HYBRID via two steps:
+
+
+
+Step #0: Producing the HERON optimized dispatches
+In this step, the HERON optimized dispatches are created like creating the dispatches CSV file in the folder: /HERON/tests/integration_tests/mechanics/debug_mode. You can create the dispatches CSV file running the commands such as:
+~/projects/HERON/heron heron_input.xml
+~/projects/raven/raven_framework outer.xml
+
+
+
+Step #1: "create_user_input.py"
+This script is run to create a user-input file for the HYBRID user.
+It is an initial step that the HYBRID user needs to rename, if necessary, the optimized dispatch outputs and the components' capacities.
+This script extracts the optimized dispatch outputs from the dispatch prints CSV file and the list of components' capacities from the HERON input XML file.
+
+This script takes the following arguments:
+1- The HERON input XML file
+2- The optimized dispatch outputs CSV file.
+For example, to run the script, use the following command:
+    python create_user_input.py heron_input.xml Debug_Run_o/dispatch_print.csv
+The output will be a user input file: "user_input.txt"
+The user needs to review the user input file and change/review the HYBRID variables and capacities there.
+
+Next, the user can run the "export2Hybrid.py" which loads the dispatch outputs from HERON to a new file that HYBRID can use. The variables' names in the autoloaded HYBRID file are borrowed from "user_input.txt"
+
+
+
+Step #2: "export2Hybrid.py"
+A script to convert the HERON optimized components' variables (optimized dispatch) to a text file compatible with HYBRID.
+
+The user-input file "user_input.txt" must be in the same folder.
+The user must review/modify the "user_input.txt" before running this script.
+
+This script is run with the following arguments:
+1- The HERON input XML file.
+2- The optimized dispatch outputs CSV file.
+For example, to run this script, use the following command:
+    python export2Hybrid.py heron_input.xml Debug_Run_o/dispatch_print.csv
+The output will be a text file compatible with HYBRID: "hybrid_compatible_dispatch.txt
+
+
+For more details, the HERON User Manual, Sec. 7.2: Auto-loading the optimized dispatches from HERON to HYBRID

tests/integration_tests/mechanics/optimizedDispatch2Hybrid/create_user_input.py

@@ -0,0 +1,161 @@
+# Copyright 2022, Battelle Energy Alliance, LLC
+# ALL RIGHTS RESERVED
+"""
+A script to create a user-input file for the HYBRID user.
+
+This script is an initial step that the HYBRID user needs to create new names, if necessary, to: 1- The optimized dispatch outputs, 2- The components' capacities.
+This script extracts the optimized dispatch outputs from the dispatch prints csv file and the list of components (and their capacities) from the HERON input XML file.
+
+It takes the following arguments:
+1- The HERON input XML file
+2- The optimized dispatch outputs csv file.
+For example, To run the script, use the following command:
+  python create_user_input.py heron_input.xml Debug_Run_o/dispatch_print.csv
+The output will be a user input file: "user_input.txt".
+The user needs to review the user input file and change/review the HYRBID variables and capacities there.
+
+Next, the user can run the "export2Hybrid.py" which loads the dispatch outpus from HERON to a new file that HYBRID can use. The variables names in the autoloaded HYBRID file are borrowd from "user_input.txt"
+"""
+
+#####
+# Section 0
+# Importing libraries and modules
+
+import os.path
+import argparse
+from xml.etree import ElementTree as ET
+import pandas as pd
+
+
+
+#####
+# Section 1: Extracting components' capacities Function
+def extract_capacities_heron_hybrid(heron_input, output_file):
+  """
+    Creates a list of components' capacities from the HERON inpue file plus creating placeholders for the components' capacities in HYRBID
+    @ In, heron input, str, the HERON input XML file
+    @ In, output_file, str, the text file where the list of components capacities should be printed.
+    @ Out, None
+  """
+
+  # Creating the HERON XML file tree
+  HERON_inp_tree = ET.parse(heron_input)
+  # Searching for the "components" node
+  components_list = HERON_inp_tree.findall("Components")
+  if not components_list:
+    print("The 'Components' node is not found in the HERON input xml file")
+  # Searching for the node "Component"
+  for components in components_list:
+    component = components.findall("Component")
+    comp_list = []  # The list of compoenents
+
+    # Printing the components' capacities in the output file
+    with open(output_file, "a+") as u:
+      u.write(
+        "\n\n\n" + "# Below is a list of the components' capacities."
+        + "\n" + "# Components' list is extracted from HERON input file: " + '"' + args.HERON_input_file + '"'
+        + "\n" + "# The capacities are listed in the form:"
+        + "\n" + "#" + "\t\t\t\t " + "HERON_component_capacity_1 = HYBRID_component_capacity_1" + "\n"
+        + "# The user is expected to change/review the HYRBID components' capacities." + "\n"
+      )
+      # Components' capacities for HYBRID are also printed as placeholders until the user change them
+      for comp in component:
+        comp_list.append(comp.attrib["name"])
+        u.write(
+          comp.attrib["name"]
+          + "_capacity = HYBRID_"
+          + comp.attrib["name"]
+          + "_capacity"
+          + "\n"
+        )
+      print("\n")
+      print(len(comp_list), "components are found in:", heron_input)
+      print(
+        "The capacities of",
+        len(comp_list),
+        "components are printed at the",
+        output_file
+      )
+
+    if not comp_list:
+      print("No components found in the HERON input xml file")
+
+
+#####
+# Section 2: Extracting dispatch outputs Function
+def extract_dispatches_heron_hybrid(dispatch_print, output_file):
+  """
+    Creates a list of dispatches (components' optimized variables) from the HERON inpue file plus creating placeholders for the dispatches in HYRBID
+    @ In, dispatch_print, str, the distpach csv file with a list of optimized variables
+    @ In, output_file, str, the text file where the list of optimized dispatches should be printed.
+    @ Out, None
+  """
+  input_dataset = pd.read_csv(dispatch_print)
+  # Extracting the HERON csv dispatches
+  colNames = input_dataset.columns[
+    input_dataset.columns.str.contains(pat="Dispatch", case=False)
+  ]
+  # Printing the dispatches in the output file
+  with open(user_input_file, "a+") as u:
+    u.write(
+      "\n\n"
+      + "# Below are the HERON variables and their corresponding HYBRID variables" + "\n"
+      + "# HERON variables are extracted from the HERON dispatch outputs file"
+      + "\n" + "# HERON dispatch outputs file is: "
+      + '"' + args.HERON_dipatch_csv_file
+      + '"' + "\n" + "# The variables are listed in the form:"
+      + "\n" + "#" + "\t\t\t\t" + "HERON_variable_1 = HYBRID_variable_1"
+      + "\n" + "# The user is expected to change/review the HYRBID variables"
+      + "\n"
+    )
+
+    for col in colNames:
+      u.write(col + " = HYBRID_" + col)
+      u.write("\n")
+    print(
+      len(list(colNames)),
+      " optimized dispatches are found in:",
+      dispatch_print,
+      "and printed at the",
+      output_file
+    )
+
+
+#####
+# Section 3: Specifying terminal command arguments & HYBRID user-input filename
+if __name__ == "__main__":
+  parser = argparse.ArgumentParser(
+    description="Create a user-input file for the HYBRID user"
+  )
+  # Inputs from the user
+  parser.add_argument("HERON_input_file", help="HERON XML input file path")
+  parser.add_argument(
+    "HERON_dipatch_csv_file", help="HERON optimized dispatch output csv file"
+  )
+  args = parser.parse_args()
+
+# Output file: the user input
+user_input_file = "user_input.txt"
+# remove old user input if exists
+file_exists = os.path.exists(user_input_file)
+if file_exists:
+  os.remove(user_input_file)
+
+#####
+# Section 4: Creating the HYBRID user-input file
+with open(user_input_file, "a+") as u:
+  u.write(
+    "# The file that includes all the HYBRID variables." + "\n"
+    "# All the capacities and dispatches should be a subset of this file's variables." "\n" + "# The user is expected to change/review the filename"
+    + "\n" + "all_variables_file = "
+    + '"' + "dsfinal.txt"+ '"'
+  )
+
+extract_capacities_heron_hybrid(args.HERON_input_file, user_input_file)
+extract_dispatches_heron_hybrid(args.HERON_dipatch_csv_file, user_input_file)
+
+print('\033[94m',
+      f"The {user_input_file} file is created for the HYBRID user to change/review it",
+      "\n",
+      '\033[0m'
+)