Surrogate Plugin ML/AI Documentation (#1181)

* Initial documentation for keras_nn plugin

* Fix doc issues

* Make docs generic to include pytorch and sci-kit learn

* Add note mentioning planned updates

---------

Co-authored-by: Ludovico Bianchi <lbianchi@lbl.gov>

docs/source/chapt_surrogates/reference.rst

@@ -31,6 +31,15 @@ The following surrogate modeling tools are currently available:
    Neural Networks. It contains special features specifically designed
    for working with CFDs.
 
+-  Keras Neural Networks (keras_nn) – Surrogate models leveraging neural networks in
+   TensorFlow Keras, a high-level machine learning library that supports training sequential or interconnected graph-based models.
+
+-  PyTorch Neural Networks (pytorch_nn) – Surrogate models leveraging neural networks in
+   PyTorch, an optimized tensor library for deep learning that supports training sequential or interconnected graph-based models.
+
+-  Scikit-learn Neural Networks (scikit_nn) – Surrogate models leveraging neural networks in
+   Scikit-learn, a predictive data analysis and surrogate modeling library that supports multi-layer regression and classification.
+
 Data Selection
 --------------
 

docs/source/chapt_surrogates/tutorial/index.rst

@@ -9,3 +9,4 @@ Tutorial
     bssanova
     uq_model
     flowsheet
+    neural_networks

docs/source/chapt_surrogates/tutorial/neural_networks.rst

@@ -0,0 +1,66 @@
+.. _(sec.surrogate.nn):
+
+Tutorial 6: Neural Networks
+===========================
+
+This tutorial covers the TensorFlow Keras Neural Network surrogate modeling method via the plugin "keras_nn". The Surrogates module also supports
+PyTorch and Scikit-learn Neural Network surrogate modeling methods, which follow the workflow below using the "pytorch_nn" and "scikit_nn" plugins, respectively.
+
+More information on TensorFlow Keras model building is described by :ref:`(Wu et al. 2020)<Wu_2020>`.
+Users may follow the recommended workflow to install and use TensorFlow in a Python environment, as described in the TensorFlow documentation:
+https://www.tensorflow.org/install.
+
+Users may obtain a great deal of usage standards and best practices information as described in the PyTorch documentation:
+https://pytorch.org/docs/stable/index.html.
+
+Users may find further information on the Scikit-learn package in the documentation:
+https://scikit-learn.org/stable/index.html and further information on deep learning capabilities as well:
+https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor.
+
+The FOQUS file for this tutorial is **Simple_flow.foqus**, and 
+this file is located in: **examples/tutorial_files/Flowsheets/Tutorial_4**
+
+.. note:: |examples_reminder_text|
+
+The Python package `tensorflow` must be installed to use this surrogate plugin.
+Before starting this tutorial, you will need to install the package by referring to
+the installation instructions in the TensorFlow documentation.
+
+#. Load the "Simple_flow.foqus" file from the examples/tutorial_files/Flowsheets/Tutorial_4 folder.
+
+#. Click the **Surrogates** button from the Home window (Figure
+   :ref:`fig.kerasnn.settings`).
+
+#. Select “keras_nn” in the **Tool** drop-down list. For PyTorch, select "pytorch_nn". For Scikit-learn, select "scikit_nn".
+
+#. Select the **Variables** tab and select the desired input and output variables. Note that "graph.error" should not be selected.
+
+#. Select the **Method Settings** tab.
+
+#. Set “Initial Data Filter” to “All”.
+
+#. Set "n_hidden" to "1", "n_neurons" to "12", "layer_act" to "relu", and "out_act" to "sigmoid".
+
+#. Ensure there are values present for "numpy_seed", "random_seed", and "tensorflow_seed" so results are reproducible.
+
+#. Set "epoch" to "500", "verbose" to "0" (i.e. epoch results will not print during training) and ensure "output_file" has a file name of the form "user_ml_ai_models/[NAME].keras". NAME may be changed, if desired. The model will be saved to the working directory folder user_ml_ai_models. For PyTorch, the file extension should be ".pt". For Scikit-learn, the file extension should be ".pkl".
+
+#. Click the **Run** icon (Figure
+   :ref:`fig.kerasnn.settings`).
+
+.. figure:: ../figs/kerasnn_settings.svg
+   :alt: Keras Neural Network Session Set Up
+   :name: fig.kerasnn.settings
+
+   Keras Neural Network Session Set Up
+
+10. The execution window will automatically display. While the regression training is finished running, TensorFlow Keras will display some information on the model size and shape.
+
+11. When the run completes, the script produces a saved model file which is compatible with the Machine Learning & Artificial Intelligence Plugin. (See Section :ref:`mlaiplugin`.)
+
+In the future, these plugins will support additional items yielding enhanced surrogate modeling capabilities.
+
+- Expanded data normalization options
+- Offline data loading from a file, e.g. CSV, Excel, PSUADE
+- Support for regression of multi-output neural networks
+- Automatic regression parameter selection during model training from user-supplied parameter options