Libs update (#668)

* trying new library versions (manually update if script not run)

* matplotlib 2 update regolding, values were indistinguishable

* added optional PIL install, also fixed docs for raven code interface

* improving compatability

* removing tensorflow for now due to Windows not liking py27

* added default location for windows

* verbosity improvement

* improving string check on est conda env os

* fixed inclusions

* new feature: skip if os

* renamed namespace variable

* disabled netCDF4 until we can update libraries

* updated pip list

* found agreement between h5py and netcdf4

* regolded text-based files with increased precision printing

* finished updating manual, disabled windows check on always-mismatch test

* attempting to make conda definitions work by environment var

* added conda_envs_path var to qsub command

* new and improved install script

* passthrough conda defs, maybe we need ravenrc for this

* reading RC file for loading

* install instructions updated

* specified path

* added condition for RC file not existing in read mode

* cleaned up raven_qsub_command

* added os verbosity

* changed setup raven libs to establish conda env everywhere

* improved crow path

* activation on load-only

* load env before make docs

* load libs before pylint check

* increased verbosity

* fail est_conda_env script on any failures

* preventing concurrent locking of hdf5 file

* increased run_tests verbosity

* removing deploment target variable as a test

* reverted activation change

* better exiting and erroring in est conda env

* put back in macosx deployment target

* attempting to prevent cython activation for making hit

* added verbosity to run_tests

* trying to avoid cython

* build script to use instead of make

* updates to build_raven

* build_raven now does both Set python environment and Build Raven steps

* more careful failing of scripts, also improved pylint checking

* loaded libraries in qsub test

* no remaking libraries in build_raven for now

* added module load to qsub command

* changed integration number for comparison stats to 5e4 to avoid seg fault in scipy.integrate.simps

* pinned numpy version

* undoing spacing/precision changes from numpy 14 to numpy 12 in text tests

* some libraries appear to be missing for mpi in raven-devel, adding moose-dev-gcc as a test to qsub command

* conda activations in qsub command scripts

* added one module for qsub, also nailed down version of hdf5 for rw issues

* fixed mpi run test to use new module

* prevent removal failure from exiting, perhaps

* review comments

* cleanup, working now with raven_framework instead of Driver

Makefile

@@ -32,6 +32,11 @@ endif
 
 CURR_DIR    := $(CURDIR)
 
+# touch hit.cpp to make sure its time stamp is different than hit.pyx
+## this is not a clean solution, but hopefully it prevents asking to use cython
+CYTHON_AVOIDANCE_ACTION=$(shell touch $(MOOSE_DIR)/framework/contrib/hit/hit.cpp)
+
+
 # framework
 #include $(FRAMEWORK_DIR)/build.mk
 #include $(FRAMEWORK_DIR)/moose.mk

amsc.mk

@@ -7,5 +7,5 @@ AMSC_MODULE = $(MY_DIR)/src/contrib/_amsc.so
 amsc :: $(AMSC_MODULE)
 $(AMSC_MODULE) : $(MY_DIR)/src/contrib/amsc.i $(AMSC_srcfiles)
 	@echo "Building "$@"..."
-	(cd $(MY_DIR) && unset CXX CC && if test `uname` = Darwin; then MACOSX_DEPLOYMENT_TARGET=10.9; export MACOSX_DEPLOYMENT_TARGET; fi && . $(MY_DIR)/scripts/setup_raven_libs && python ./setup.py build_ext build install --install-platlib=./framework/contrib/AMSC)
+	(cd $(MY_DIR) && unset CXX CC && . $(MY_DIR)/scripts/establish_conda_env.sh --load && python ./setup.py build_ext build install --install-platlib=./framework/contrib/AMSC)
 	@echo "Done"

build_framework

@@ -9,7 +9,7 @@ else
 fi
 SCRIPT_DIR=`(cd $SCRIPT_DIRNAME; pwd)`
 
-source $SCRIPT_DIR/scripts/setup_raven_libs
+source $SCRIPT_DIR/scripts/establish_conda_env.sh --quiet --load
 
 #These don't always work for setup.py, so safer to unset them.
 unset CXX CC

build_raven

@@ -0,0 +1,76 @@
+#!/bin/bash
+
+# fail if anything here fails
+set -e
+
+RAVEN_BUILD_DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+echo Making RAVEN ...
+
+# defaults
+MAKE_ARGS=" "
+LIBS_MODE=0 # 0 is "create", 1 is "load"
+#### FUTURE WORK:
+####   combine "make" and "establish_conda_env.sh" into one installation command
+ECE_ARGS=" "
+
+# read arguments
+while test $# -gt 0
+do
+  case "$1" in
+     --coverage)
+       MAKE_ARGS="${MAKE_ARGS} coverage=true graceful=true"
+       LIBS_MODE=1
+       ;;
+     --jobs) MAKE_ARGS="-j $1 ${MAKE_ARGS}"
+#    --help)     ECE_ARGS="${ECE_ARGS} --help"; ;;
+#    --optional) ECE_ARGS="${ECE_ARGS} --optional"; ;;
+#    --no-clean) ECE_ARGS="${ECE_ARGS} --no-clean"; ;;
+#    --conda-defs)
+#      shift
+#      ECE_ARGS="${ECE_ARGS} --conda-defs $1"
+#      ;;
+#    --raven-libs-name)
+#      shift
+#      export RAVEN_LIBS_NAME="$1"
+#      ;;
+  esac
+  shift
+done
+#### END FUTURE WORK
+
+if [[ $LIBS_MODE == 0 ]]; then
+  #### FUTURE WORK:
+  # clear .ravenrc
+  #if [[ -f "$RAVEN_BUILD_DIR/.ravenrc" ]]; then
+  #  rm "$RAVEN_BUILD_DIR/.ravenrc"
+  #fi
+
+  # combining install script for python libs and make?
+  #echo Establishing Python libraries ...
+  #. scripts/establish_conda_env.sh --install ${ECE_ARGS}
+
+  # identify python
+  #echo python: `which python`
+  #### END FUTURE WORK
+  ## remove when FUTURE WORK is in place
+  echo Establishing Python libraries ...
+  . scripts/establish_conda_env.sh --load ${ECE_ARGS}
+
+  # clean up to assure library matching
+  echo Cleaning old build ...
+  make clean
+
+  # set up hit.cpp so that it has a newer timestamp than hit.pyx
+  touch moose/framework/contrib/hit/hit.cpp
+else
+  echo Establishing Python libraries ...
+  . scripts/establish_conda_env.sh --load ${ECE_ARGS}
+fi
+
+
+echo Making ...
+# make
+make ${MAKE_ARGS}
+
+echo ... done!

check_py_coverage.sh

@@ -48,7 +48,7 @@ cd tests/framework
 #coverage help run
 FRAMEWORK_DIR=`(cd ../../framework && pwd)`
 
-source $SCRIPT_DIR/scripts/setup_raven_libs
+source $SCRIPT_DIR/scripts/establish_conda_env.sh --quiet --load
 
 EXTRA="--rcfile=$FRAMEWORK_DIR/../tests/framework/.coveragerc --source=$FRAMEWORK_DIR -a --omit=$FRAMEWORK_DIR/contrib/*"
 export COVERAGE_FILE=`pwd`/.coverage

crow/crow_python_modules.mk

@@ -14,11 +14,11 @@ $(CROW_DIR)/install/crow_modules/_distribution1Dpy2.so $(CROW_DIR)/install/crow_
 .INTERMEDIATE: setup_py2_intermediate
 
 setup_py2_intermediate : $(CROW_DIR)/crow_modules/distribution1Dpy2.i $(CROW_DIR)/crow_modules/interpolationNDpy2.i $(CROW_DIR)/crow_modules/randomENGpy2.i $(DISTRIBUTION_SOURCE) $(INTERPOLATION_SOURCE) $(RANDOM_CLASS_SOURCE)
-	(cd $(CROW_DIR) && unset CXX CC && CFLAGS="$$CFLAGS $(COVERAGE_COMPILE_EXTRA)" && LDFLAGS="$$LDFLAGS $(COVERAGE_LINK_EXTRA)" && export CFLAGS LDFLAGS &&if test `uname` = Darwin; then MACOSX_DEPLOYMENT_TARGET=10.9; export MACOSX_DEPLOYMENT_TARGET; fi && . ./scripts/setup_raven_libs && python ./setup.py build_ext build install --install-platlib=./install)
+	(cd $(CROW_DIR) && unset CXX CC && CFLAGS="$$CFLAGS $(COVERAGE_COMPILE_EXTRA)" && LDFLAGS="$$LDFLAGS $(COVERAGE_LINK_EXTRA)" && export CFLAGS LDFLAGS &&if test `uname` = Darwin; then MACOSX_DEPLOYMENT_TARGET=10.9; export MACOSX_DEPLOYMENT_TARGET; fi && . ../scripts/establish_conda_env.sh --load && python ./setup.py build_ext build install --install-platlib=./install)
 
 $(CROW_DIR)/install/crow_modules/_distribution1Dpy3.so $(CROW_DIR)/install/crow_modules/_interpolationNDpy3.so $(CROW_DIR)/install/crow_modules/_randomENGpy3.so : setup_py3_intermediate
 
 .INTERMEDIATE: setup_py3_intermediate
 
 setup_py3_intermediate : $(CROW_DIR)/crow_modules/distribution1Dpy3.i $(CROW_DIR)/crow_modules/interpolationNDpy3.i $(CROW_DIR)/crow_modules/randomENGpy3.i $(DISTRIBUTION_SOURCE) $(INTERPOLATION_SOURCE) $(RANDOM_CLASS_SOURCE)
-	(cd $(CROW_DIR) && unset CXX CC && if test `uname` = Darwin; then MACOSX_DEPLOYMENT_TARGET=10.9; export MACOSX_DEPLOYMENT_TARGET; fi && . ./scripts/setup_raven_libs && python3 ./setup3.py build_ext build install --install-platlib=./install)
+	(cd $(CROW_DIR) && unset CXX CC && if test `uname` = Darwin; then MACOSX_DEPLOYMENT_TARGET=10.9; export MACOSX_DEPLOYMENT_TARGET; fi && . ../scripts/establish_conda_env.sh --load && python3 ./setup3.py build_ext build install --install-platlib=./install)

developer_tools/pylint_check

@@ -1,8 +1,14 @@
 #!/bin/bash
 
+# fail if any of these commands fail
+set -e
+
 SCRIPT_DIRNAME=`dirname $0`
 SCRIPT_DIR=`(cd $SCRIPT_DIRNAME; pwd)`
 
 cd $SCRIPT_DIR/..
 
+# activate raven libraries
+source $SCRIPT_DIR/../scripts/establish_conda_env.sh --load --quiet
+
 pylint --disable=all --enable=missing-docstring --enable=no-absolute-import --enable=old-division --enable=print-statement --enable=file-builtin --enable=multiple-statements framework/

doc/make_docs.sh

@@ -22,12 +22,14 @@ done
 
 rm -Rvf pdfs
 
+# load raven libraries
+source ../scripts/establish_conda_env.sh --load --quiet
 
 # add custom, collective inputs to TEXINPUTS
 #
 # Since on Windows we use MikTeX (which is a native Windows program), the TEXTINPUTS variable used i
-#   to tell the LaTeX processor where to look for .sty files must be set using Windows-style paths 
-#   (not the Unix-style ones used on other platforms).  This also means semi-colons need to be used 
+#   to tell the LaTeX processor where to look for .sty files must be set using Windows-style paths
+#   (not the Unix-style ones used on other platforms).  This also means semi-colons need to be used
 #   to separate terms instead of the Unix colon.
 #
 if [ "$(expr substr $(uname -s) 1 5)" == "MINGW" ]

doc/user_manual/HowToRun.tex

@@ -17,9 +17,8 @@ \section{Running RAVEN}
 raven_framework <inputFileName.xml>
 \end{lstlisting}
 
-Alternatively, the \texttt{Driver.py} script can be directly used.  The RAVEN driver is contained in the folder ``\texttt{raven/framework}.'' In this case, the command is:
-
-\begin{lstlisting}[language=bash]
-python raven/framework/Driver.py <inputFileName.xml>
-\end{lstlisting}
-
+Using \texttt{raven\_framework} is the recommended way to run RAVEN.  In the event bypassing the typical
+environment loading and checks is desired, it can also be run via
+the \texttt{Driver.py} script using python, with the input file as argument.  However, this is not
+recommended, as it will use whatever default versions of Python and other libraries are discovered, rather
+than the matching libraries set up during installation.

doc/user_manual/Installation/clone.tex

@@ -0,0 +1,101 @@
+\subsection{Installing RAVEN}
+\label{sec:clone raven}
+
+Once the RAVEN dependencies have been installed  and conda is present
+(see section \ref{sec:install overview}), the rest of RAVEN can be installed.
+
+The installation of RAVEN involves the following steps:
+\begin{itemize}
+  \item Obtain the source code,
+  \item Install the prerequisite Python libraries using conda,
+  \item Compile
+\end{itemize}
+
+
+
+
+\subsubsection{Obtaining RAVEN Source Code}
+RAVEN is hosted publicly as a \texttt{Git} repo on \texttt{GitHub}
+and can be viewed at \url{https://github.com/idaholab/raven/wiki}.
+In the event that access to \texttt{GitHub} is impossible, contact the user list and other arrangements may be
+possible.  In general, however, using the git repository assures the most consistent usage and update process.
+
+To clone RAVEN, navigate in a terminal to the desired destination, for example \texttt{~/projects}.  Then run
+the commands
+\begin{lstlisting}[language=bash]
+git clone https://github.com/idaholab/raven.git
+cd raven
+git submodule update --init
+\end{lstlisting}
+This will obtain RAVEN as well as other submodules that RAVEN uses.  In the future, whenever we declare a path
+starting with \texttt{raven/}, we refer to the cloned directory.
+
+
+
+
+\subsubsection{Installing Python Libraries}
+RAVEN depends heavily on Python, and uses conda to maintain a separate environment to prevent conflicts with
+other Python library installations.  This separate environment is called \texttt{raven\_libraries}.
+
+In order to establish this environment, navigate to \texttt{raven}, then
+\begin{lstlisting}[language=bash]
+cd scripts
+./establish\_conda\_env.sh --install
+\end{lstlisting}
+Assure that there are no errors in this process, then continue to compiling RAVEN.
+
+\nb If \texttt{conda} is not installed in the default location, then the path to the conda definitions
+needs to be provided, for example
+\begin{lstlisting}[language=bash]
+cd scripts
+./establish\_conda\_env.sh --install --conda-defs /path/to/miniconda2/etc/profile.d/conda.sh
+\end{lstlisting}
+replacing \texttt{/path/to} with the install path for \texttt{conda}.
+
+
+
+
+\subsubsection{Compiling RAVEN}
+Once Python libraries are established and the source code present, navigate to \texttt{raven} and run
+\begin{lstlisting}[language=bash]
+./build_raven
+\end{lstlisting}
+This will compile several dependent libraries.  This step has the highest potential for revealing problems in
+the operating system setup, particularly for Windows.  See troubleshooting on the \wiki for help sorting out
+difficulties.
+
+
+
+
+\subsubsection{Testing RAVEN}
+\label{sec:testing raven}
+To test the installation of RAVEN, navigate to \texttt{raven}, then run the command
+\begin{lstlisting}[language=bash]
+./run_tests -j2
+\end{lstlisting}
+where \texttt{-j2} signifies running with 2 processors.  If more processors are available, this can be
+increased, but using all or more than all of the available processes can slow down the testing dramatically.
+This command runs RAVEN's regression tests, analytic tests, and unit tests.  The number of tests changes
+frequently as the code's needs change, and the time taken to run the tests depends strongly on the number of
+processors and processor speed.
+
+At the end of the tests, a number passed, skipped, and failing will be reported.  Having some skipped tests is
+expected; RAVEN has many tests that apply only to particular configurations or codes that are not present on
+all machines.  However, no tests should fail; if there are problems, consult the troubleshooting section on
+the \wiki.
+
+
+\subsubsection{Updating RAVEN}
+RAVEN updates frequently, and new features are added while bugs are fixed on a regular basis.  To update
+RAVEN, navigate to \texttt{raven}, then run the commands
+\begin{lstlisting}[language=bash]
+git pull
+./scripts/establish_conda_env.sh --install
+./build_raven
+\end{lstlisting}
+
+
+\subsubsection{In-use Testing}
+
+At any time, tests can be checked by re-running the installation tests as
+described in Section \ref{sec:testing raven}.

doc/user_manual/Installation/conda.tex

@@ -0,0 +1,12 @@
+\subsection{Conda: Python Dependencies}
+\label{sec:install conda}
+
+The standard installation procedure for RAVEN includes using Miniconda (often simply referred to as
+\emph{conda}) to install the Python libraries required to run RAVEN.  If conda cannot be made available on an
+operating system, refer to the wiki (listed above) for alternatives.  To install miniconda, follow the
+instructions for your operating system at \url{https://conda.io/miniconda.html}.
+
+\nb Since RAVEN currently relies
+on some Python 2.7-only libraries, make sure to install the 2.7 64-bit version of miniconda.
+
+Once conda is installed, proceed to installing RAVEN itself (section \ref{sec:clone raven}).

doc/user_manual/Installation/linux.tex

@@ -0,0 +1,21 @@
+\subsection{Linux Ubuntu Installation}
+\label{sec:install ubunutu}
+The following instructions are for installing RAVEN on a Linux machine running Ubuntu 16.04 or greater.  Some
+explanations of alternatives for other Linux distributions may be provided on the \wiki.
+
+To install the prerequisite packages, the following terminal command should be executed (note this requires
+administrative privileges):
+
+\begin{lstlisting}[language=bash]
+ sudo apt-get install libtool git python-dev swig g++
+\end{lstlisting}
+
+\paragraph{Optional LateX installation}
+Optionally, if you want to be able to edit and rebuild the manuals, you can
+install \TeX~Live and its related packages:
+\begin{lstlisting}[language=bash]
+  sudo apt-get install texlive-latex-base \
+  texlive-extra-utils texlive-latex-extra texlive-math-extra
+\end{lstlisting}
+
+Once the above are installed, proceed with installing conda (see section \ref{sec:install conda}).

doc/user_manual/Installation/macosx.tex

@@ -0,0 +1,29 @@
+\subsection{Mac OSX Installation}
+\label{sec:install mac}
+
+When using an Apple Macintosh computer, software dependencies are met
+by following steps:
+\begin{itemize}
+  \item Install the XCode command line tools from Apple,
+  \item Install the XQuartz  X-Window system server,
+\end{itemize}
+
+\subsubsection{Installing XCode Command Line Tools}
+
+The XCode command line tools package from Apple Computer provides the C++
+compilers and git source code control tools needed to obtain and build RAVEN.
+It is freely available from the Apple store. In order to obtain it the following command should be launched in an open terminal:
+\begin{lstlisting}[language=bash]
+ xcode-select --install
+\end{lstlisting}
+
+\subsubsection{Installing XQuartz}
+XQuartz is an implementation of the X Server for the Mac OSX operating system.
+XQuartz is freely available on the web and can be downloaded from the link
+ \url{https://dl.bintray.com/xquartz/downloads/XQuartz-2.7.9.dmg}.
+\\After downloaded, install the package.
+
+With XCode and XQuartz installed, continue on to install conda (see section \ref{sec:install conda}).
+
+\nb While \texttt{gcc} and
+\texttt{git} are also required, they are installed by default in the OSX system.

doc/user_manual/Installation/main.tex

@@ -0,0 +1,12 @@
+% definitions
+
+% content
+\input{Installation/overview.tex}
+
+\input{Installation/linux.tex}
+\input{Installation/macosx.tex}
+\input{Installation/windows.tex}
+
+\input{Installation/conda.tex}
+
+\input{Installation/clone.tex}

doc/user_manual/Installation/overview.tex

@@ -0,0 +1,22 @@
+\section{Installation}
+\subsection{Overview}
+\label{sec:install overview}
+
+The installation of the RAVEN code is a straightforward procedure;
+depending on the usage purpose and machine architecture, the
+installation process slightly differs.
+
+In the following sections, the recommended installation procedure is outlined.  For alternatives, we encourage
+checking the \wiki.  The machines on which
+RAVEN is tested and developed, however, use the standard installation procedures outlined below.
+
+The installation process will involve three steps:
+\begin{itemize}
+  \item Installing prerequisites, which depends on your operating system;
+  \item Installing conda;
+  \item Installing RAVEN.
+\end{itemize}
+
+Depending on your operating system (Windows in section \ref{sec:install windows}, MacOSX in section
+\ref{sec:install mac}, Ubuntu Linux in section \ref{sec:install ubunutu}), follow the instructions for installing prerequisites, then continue with
+installing conda (section \ref{sec:install conda}), and then installing RAVEN (section \ref{sec:clone raven}).