Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure Interface PostProcessors #1533

Merged
merged 58 commits into from
May 12, 2021
Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
62f663d
restructure PostProcessor to use BaseEntities and BaseInterface Struc…
wangcj05 Apr 6, 2021
66ffd6f
updates
wangcj05 Apr 6, 2021
8e5c79a
update
wangcj05 Apr 7, 2021
0bd8aa2
update super() calls
wangcj05 Apr 7, 2021
7b763d8
Merge branch 'devel' into wangc/convert_pp_to_interface
wangcj05 Apr 8, 2021
108918e
update
wangcj05 Apr 8, 2021
ad53cc8
enable Assembler object for post processor
wangcj05 Apr 9, 2021
1339b1d
update collectoutput
wangcj05 Apr 9, 2021
c2186c4
update
wangcj05 Apr 9, 2021
d6a830f
use PP factory instead of Model factory
wangcj05 Apr 9, 2021
cae97e4
clean up
wangcj05 Apr 9, 2021
cf594ba
fix the way to retrieve the interface properties
wangcj05 Apr 9, 2021
a91e7a1
add docstr for classes
wangcj05 Apr 12, 2021
7458d53
resolve comments
wangcj05 Apr 13, 2021
dd437f2
postprocessor plugin API
wangcj05 Apr 13, 2021
94eeb38
convert FT Importer PP
wangcj05 Apr 15, 2021
8419f8f
convert MCS Importer
wangcj05 Apr 15, 2021
a2166f7
enable both dict and xarray.Dataset for pp plugin
wangcj05 Apr 15, 2021
63cdfad
convert DataClassifier to use dict as input
wangcj05 Apr 16, 2021
4fe0f84
convert RiskMeasures PP
wangcj05 Apr 16, 2021
3310ab8
fix precheck
wangcj05 Apr 19, 2021
59c3bb8
Merge branch 'devel' into wangc/pp_plugin_api_fy21_v3
wangcj05 Apr 22, 2021
8d435d2
update validateDict
wangcj05 Apr 22, 2021
84bb6b6
update validateDict
wangcj05 Apr 22, 2021
6f5e2f7
resolve comments
wangcj05 Apr 23, 2021
58245a4
fix MCImporter
wangcj05 Apr 23, 2021
42a6653
split PostProcessor document into separate files
wangcj05 Apr 23, 2021
25f3431
update RiskMeasureDiscrete PP document
wangcj05 Apr 23, 2021
3cadea0
Merge branch 'devel' into wangc/pp_plugin_api_fy21_v3
wangcj05 Apr 27, 2021
25f188f
convert HS2PS to use DataSet
wangcj05 Apr 28, 2021
9dfccd2
update
wangcj05 Apr 28, 2021
b7266a0
Merge branch 'devel' into wangc/pp_plugin_api_fy21_v3
wangcj05 Apr 29, 2021
5226af8
Merge branch 'wangc/pp_plugin_api_fy21_v3' into wangc/pp_dataset
wangcj05 Apr 29, 2021
db587a1
add metakeys from input dataobjects to output dataobjects
wangcj05 Apr 29, 2021
31ce4f6
update
wangcj05 Apr 29, 2021
2c9522a
move HS2PS to PostProcessors
wangcj05 Apr 29, 2021
b6c10b3
Merge branch 'devel' into wangc/pp_dataset
wangcj05 May 5, 2021
d2a19ca
convert HStoPSOperator
wangcj05 May 6, 2021
abc7e49
convert histset sampling and historyset sync pp
wangcj05 May 6, 2021
9747899
convert historySetSnapShot
wangcj05 May 6, 2021
95024cb
update
wangcj05 May 6, 2021
6915686
convert TypicalHistoryFromHS
wangcj05 May 6, 2021
4b09ac2
convert the rest of interface pp
wangcj05 May 7, 2021
1893901
remove interface pp related classes
wangcj05 May 7, 2021
b69ba93
update data mining
wangcj05 May 7, 2021
8e80690
revert changes in HS2PS
wangcj05 May 7, 2021
965cb64
update HistorySetDelay to use PostProcessorPluginBase
wangcj05 May 7, 2021
949117b
convert failed tests
wangcj05 May 7, 2021
2db39f6
update test
wangcj05 May 7, 2021
43dd641
add conversion script for PR 1533
wangcj05 May 7, 2021
b3232fe
remove RavenOut
wangcj05 May 7, 2021
89c638a
update manual for interface postprocessors
wangcj05 May 7, 2021
bf6a4a0
update plugin
wangcj05 May 7, 2021
874944a
revert HEROM commit
wangcj05 May 7, 2021
00db572
update doc
wangcj05 May 11, 2021
f58071b
resolve plugin configure file issue within plugin_handler.py
wangcj05 May 11, 2021
ff8bd23
Merge branch 'devel' into wangc/pp_dataset
wangcj05 May 11, 2021
5f32bb6
update plugin base class location
wangcj05 May 11, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions doc/user_manual/PostProcessors/HS2PS.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
\subsubsection{HS2PS}
\label{HS2PS}

This Post-Processor performs a conversion from HistorySet to PointSet.
The conversion is made so that each history $H$ is converted to a single point $P$.
Assume that each history $H$ is a dictionary (mapping) of $n$ output variables $x_1=[...],x_n=[...]$, then the resulting point $P$ is $P=concat(x_1,...,x_n)$.
Note: it is here assumed that all histories have been synced so that they have the same length, start point and end point.
If you are not sure, do a pre-processing the the original history set.

\ppType{HS2PS}{HS2PS}

In the \xmlNode{PostProcessor} input block, the following XML sub-nodes are required,
independent of the \xmlAttr{subType} specified (min, max, avg and value case):

\begin{itemize}
\item \xmlNode{pivotParameter}, \xmlDesc{string, optional field}, ID of the temporal variable (only for avg)
\end{itemize}
103 changes: 103 additions & 0 deletions doc/user_manual/PostProcessors/HStoPSOperator.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
\subsubsection{HStoPSOperator}
\label{HStoPSOperator}

This Post-Processor performs the conversion from HistorySet to PointSet performing a projection of the output space.
\ppType{HStoPSOperator}{HStoPSOperator}
In the \xmlNode{PostProcessor} input block, the following XML sub-nodes are available:

\begin{itemize}
\item \xmlNode{pivotParameter}, \xmlDesc{string, optional field}, ID of the temporal variable. Default is ``time''.
\nb Used just in case the \xmlNode{pivotValue}-based operation is requested
\item \xmlNode{operator}, \xmlDesc{string, optional field}, the operation to perform on the output space:
\begin{itemize}
\item \textbf{min}, compute the minimum of each variable along each single history
\item \textbf{max}, compute the maximum of each variable along each single history
\item \textbf{average}, compute the average of each variable along each single history
\item \textbf{all}, join together all of the each variable in
the history, and make the pivotParameter a regular
parameter. Unlike the min and max operators, this keeps
all the data, just organized differently. This operator
does this by propagating the other input parameters for
each item of the pivotParameter.
Table~\ref{operator_all_switch_before} shows an example
HistorySet with input parameter x, pivot parameter t, and
output parameter b and then
Table~\ref{operator_all_switch_after} shows the resulting
PointSet with input parameters x and t, and output
parameter b. Note that which parameters are input and which
are output in the resulting PointSet depends on the
DataObject specification.
\end{itemize}
\nb This node can be inputted only if \xmlNode{pivotValue} and \xmlNode{row} are not present
\item \xmlNode{pivotValue}, \xmlDesc{float, optional field}, the value of the pivotParameter with respect to the other outputs need to be extracted.
\nb This node can be inputted only if \xmlNode{operator} and \xmlNode{row} are not present
\item \xmlNode{pivotStrategy}, \xmlDesc{string, optional field}, The strategy to use for the pivotValue:
\begin{itemize}
\item \textbf{nearest}, find the value that is the nearest with respect the \xmlNode{pivotValue}
\item \textbf{floor}, find the value that is the nearest with respect to the \xmlNode{pivotValue} but less then the \xmlNode{pivotValue}
\item \textbf{celing}, find the value that is the nearest with respect to the \xmlNode{pivotValue} but greater then the \xmlNode{pivotValue}
\item \textbf{interpolate}, if the exact \xmlNode{pivotValue} can not be found, interpolate using a linear approach
\end{itemize}

\nb Valid just in case \xmlNode{pivotValue} is present
\item \xmlNode{row}, \xmlDesc{int, optional field}, the row index at which the outputs need to be extracted.
\nb This node can be inputted only if \xmlNode{operator} and \xmlNode{pivotValue} are not present
\end{itemize}

This example will show how the XML input block would look like:

\begin{lstlisting}[style=XML,morekeywords={subType,debug,name,class,type}]
<Simulation>
...
<Models>
...
<PostProcessor name="HStoPSperatorRows" subType="HStoPSOperator">
<row>-1</row>
</PostProcessor>
<PostProcessor name="HStoPSoperatorPivotValues" subType="HStoPSOperator">
<pivotParameter>time</pivotParameter>
<pivotValue>0.3</pivotValue>
</PostProcessor>
<PostProcessor name="HStoPSoperatorOperatorMax" subType="HStoPSOperator">
<pivotParameter>time</pivotParameter>
<operator>max</operator>
</PostProcessor>
<PostProcessor name="HStoPSoperatorOperatorMin" subType="HStoPSOperator">
<pivotParameter>time</pivotParameter>
<operator>min</operator>
</PostProcessor>
<PostProcessor name="HStoPSoperatorOperatorAverage" subType="HStoPSOperator">
<pivotParameter>time</pivotParameter>
<operator>average</operator>
</PostProcessor>
...
</Models>
...
</Simulation>
\end{lstlisting}

\begin{table}[!hbtp]
\caption{Starting HistorySet for operator all}
\label{operator_all_switch_before}
\begin{tabular}{l|l|l}
x & t & b \\
\hline
5.0 & & \\
\hline
& 1.0 & 6.0 \\
\hline
& 2.0 & 7.0 \\
\end{tabular}
\end{table}

\begin{table}[!hbtp]
\caption{Resulting PointSet after operator all}
\label{operator_all_switch_after}
\begin{tabular}{l|l|l}
x & t & b \\
\hline
5.0 & 1.0 & 6.0 \\
\hline
5.0 & 2.0 & 7.0 \\
\end{tabular}
\end{table}
35 changes: 35 additions & 0 deletions doc/user_manual/PostProcessors/HistorySetSampling.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
\subsubsection{HistorySetSampling}
\label{HistorySetSampling}

This Post-Processor performs the conversion from HistorySet to HistorySet
The conversion is made so that each history H is re-sampled accordingly to a
specific sampling strategy.
It can be used to reduce the amount of space required by the HistorySet.

\ppType{HistorySetSampling}{HistorySetSampling}

In the \xmlNode{PostProcessor} input block, the following XML sub-nodes are required,
independent of the \xmlAttr{subType} specified:

\begin{itemize}
\item \xmlNode{samplingType}, \xmlDesc{string, required field}, specifies the type of sampling method to be used:
\begin{itemize}
\item uniform: the set of \xmlNode{numberOfSamples} samples are uniformly distributed along the time axis
\item firstDerivative: the set of \xmlNode{numberOfSamples} samples are distributed along the time axis in regions with
higher first order derivative
\item secondDerivative: the set of \xmlNode{numberOfSamples} samples are distributed along the time axis in regions with
higher second order derivative
\item filteredFirstDerivative: samples are located where the first derivative is greater than the specified \xmlNode{tolerance} value
(hence, the number of samples can vary from history to history)
\item filteredSecondDerivative: samples are located where the second derivative is greater than the specified \xmlNode{tolerance} value
(hence, the number of samples can vary from history to history)
\end{itemize}
\item \xmlNode{numberOfSamples}, \xmlDesc{integer, optional field}, number of samples (required only for the following sampling
types: uniform, firstDerivative secondDerivative)
\item \xmlNode{pivotParameter}, \xmlDesc{string, required field}, ID of the temporal variable
\item \xmlNode{interpolation}, \xmlDesc{string, optional field}, type of interpolation to be employed for the history reconstruction
(required only for the following sampling types: uniform, firstDerivative secondDerivative).
Valid types of interpolation to specified: linear, nearest, zero, slinear, quadratic, cubic, intervalAverage
\item \xmlNode{tolerance}, \xmlDesc{string, optional field}, tolerance level (required only for the following sampling types:
filteredFirstDerivative or filteredSecondDerivative)
\end{itemize}
94 changes: 94 additions & 0 deletions doc/user_manual/PostProcessors/HistorySetSnapShot.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
\subsubsection{HistorySetSnapShot}
\label{HistorySetSnapShot}

This Post-Processor performs a conversion from HistorySet to PointSet.
The conversion is made so that each history $H$ is converted to a single point $P$.
There are several methods that can be employed to choose the single point from the history:
\begin{itemize}
\item min: Take a time slice when the \xmlNode{pivotVar} is at its smallest value,
\item max: Take a time slice when the \xmlNode{pivotVar} is at its largest value,
\item average: Take a time slice when the \xmlNode{pivotVar} is at its time-weighted average value,
\item value: Take a time slice when the \xmlNode{pivotVar} \emph{first passes} its specified value,
\item timeSlice: Take a time slice index from the sampled time instance space.
\end{itemize}
To demonstrate the timeSlice, assume that each history H is a dict of n output variables $x_1=[...],
x_n=[...]$, then the resulting point P is at time instant index t: $P=[x_1[t],...,x_n[t]]$.

Choosing one the these methods for the \xmlNode{type} node will take a time slice for all the variables in the
output space based on the provided parameters. Alternatively, a \xmlString{mixed} type can be used, in which
each output variable can use a different time slice parameter. In other words, you can take the max of one
variable while taking the minimum of another, etc.

\ppType{HistorySetSnapShot}{HistorySetSnapShot}

In the \xmlNode{PostProcessor} input block, the following XML sub-nodes are required,
independent of the \xmlAttr{subType} specified:

\begin{itemize}
\item \xmlNode{type}, \xmlDesc{string, required field}, type of operation: \xmlString{min}, \xmlString{max},
\xmlString{average}, \xmlString{value}, \xmlString{timeSlice}, or \xmlString{mixed}
\item \xmlNode{extension}, \xmlDesc{string, required field}, type of extension when the sync process goes outside the boundaries of the history (zeroed or extended)
\item \xmlNode{pivotParameter}, \xmlDesc{string, optional field}, name of the temporal variable. Required for the
\xmlString{average} and \xmlString{timeSlice} methods.
\end{itemize}

If a \xmlString{timeSlice} type is in use, the following nodes also are required:
\begin{itemize}
\item \xmlNode{timeInstant}, \xmlDesc{integer, required field}, required and only used in the
\xmlString{timeSlice} type. Location of the time slice (integer index)
\item \xmlNode{numberOfSamples}, \xmlDesc{integer, required field}, number of samples
\end{itemize}

If instead a \xmlString{min}, \xmlString{max}, \xmlString{average}, or \xmlString{value} is used, the following nodes
are also required:
\begin{itemize}
\item \xmlNode{pivotVar}, \xmlDesc{string, required field}, Name of the chosen indexing variable (the
variable whose min, max, average, or value is used to determine the time slice)
\item \xmlNode{pivotVal}, \xmlDesc{float, optional field}, required for \xmlString{value} type, the value for the chosen variable
\end{itemize}

Lastly, if a \xmlString{mixed} approach is used, the following nodes apply:
\begin{itemize}
\item \xmlNode{max}, \xmlDesc{string, optional field}, the names of variables whose output should be their
own maximum value within the history.
\item \xmlNode{min}, \xmlDesc{string, optional field}, the names of variables whose output should be their
own minimum value within the history.
\item \xmlNode{average}, \xmlDesc{string, optional field}, the names of variables whose output should be their
own average value within the history. Note that a \xmlNode{pivotParameter} node is required to perform averages.
\item \xmlNode{value}, \xmlDesc{string, optional field}, the names of variables whose output should be taken
at a time slice determined by another variable. As with the non-mixed \xmlString{value} type, the first
time the \xmlAttr{pivotVar} crosses the specified \xmlAttr{pivotVal} will be the time slice taken.
This node requires two attributes, if used:
\begin{itemize}
\item \xmlAttr{pivotVar}, \xmlDesc{string, required field}, the name of the variable on which the time
slice will be performed. That is, if we want the value of $y$ when $t=0.245$,
this attribute would be \xmlString{t}.
\item \xmlAttr{pivotVal}, \xmlDesc{float, required field}, the value of the \xmlAttr{pivotVar} on which the time
slice will be performed. That is, if we want the value of $y$ when $t=0.245$,
this attribute would be \xmlString{0.245}.
\end{itemize}
Note that all the outputs of the \xmlNode{DataObject} output of this postprocessor must be listed under one
of the \xmlString{mixed} node types in order for values to be returned.
\end{itemize}

\textbf{Example (mixed):}
This example will output the average value of $x$ for $x$, the value of $y$ at
time$=0.245$ for $y$, and the value of $z$ at $x=4.0$ for $z$.
\begin{lstlisting}[style=XML,morekeywords={subType,debug,name,class,type}]
<Simulation>
...
<Models>
...
<PostProcessor name="mampp2" subType="HistorySetSnapShot">
<type>mixed</type>
<average>x</average>
<value pivotVar="time" pivotVal="0.245">y</value>
<value pivotVar="x" pivotVal="4.0">z</value>
<pivotParameter>time</pivotParameter>
<extension>zeroed</extension>
</PostProcessor>
...
</Models>
...
</Simulation>
\end{lstlisting}
32 changes: 32 additions & 0 deletions doc/user_manual/PostProcessors/HistorySetSync.tex
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
\subsubsection{HistorySetSync}
\label{HistorySetSync}

This Post-Processor performs the conversion from HistorySet to HistorySet
The conversion is made so that all histories are synchronized in time.
It can be used to allow the histories to be sampled at the same time instant.

There are two possible synchronization methods, specified through the \xmlNode{syncMethod} node. If the
\xmlNode{syncMethod} is \xmlString{grid}, a \xmlNode{numberOfSamples} node is specified,
which yields an equally-spaced grid of time points. The output values for these points will be linearly derived
using nearest sampled time points, and the new HistorySet will contain only the new grid points.

The other methods are used by specifying \xmlNode{syncMethod} as \xmlString{all}, \xmlString{min}, or
\xmlString{max}. For \xmlString{all}, the postprocessor will iterate through the
existing histories, collect all the time points used in any of them, and use these as the new grid on which to
establish histories, retaining all the exact original values and interpolating linearly where necessary.
In the event of \xmlString{min} or \xmlString{max}, the postprocessor will find the smallest or largest time
history, respectively, and use those time values as nodes to interpolate between.

\ppType{HistorySetSync}{HistorySetSync}

In the \xmlNode{PostProcessor} input block, the following XML sub-nodes are required,
independent of the \xmlAttr{subType} specified:

\begin{itemize}
\item \xmlNode{pivotParameter}, \xmlDesc{string, required field}, ID of the temporal variable
\item \xmlNode{extension}, \xmlDesc{string, required field}, type of extension when the sync process goes outside the boundaries of the history (zeroed or extended)
\item \xmlNode{syncMethod}, \xmlDesc{string, required field}, synchronization strategy to employ (see
description above). Options are \xmlString{grid}, \xmlString{all}, \xmlString{max}, \xmlString{min}.
\item \xmlNode{numberOfSamples}, \xmlDesc{integer, optional field}, required if \xmlNode{syncMethod} is
\xmlString{grid}, number of new time samples
\end{itemize}
Loading