Libs update #668

PaulTalbot-INL · 2018-05-29T20:21:24Z

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

Closes #666 and moves toward addressing #664

What are the significant changes in functionality due to this change request?

Updates matplotlib to 2.1.1 and xarray to 0.10.3
Updates documentation for raven code interface
Adds optional install command for installing PIL (and adds framework for others)
Continues to extend conda establish script to include more checks and searches to establish conda env

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

1. Review all computer code.
2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True.
6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
8. If an analytic test is changed/added is the the analytic documentation updated/added?

alfoa

approved.... Can you modify the installation wiki to suggest to run the script instead of the conda command?

alfoa · 2018-06-06T03:50:11Z

It would be worth to run the estrablish_conda_script.sh in the make file as well.
@PaulTalbot-INL @derekstucki @wangcj05 @mandd what do you think?

joshua-cogliati-inl · 2018-06-06T13:33:16Z

Just out of curiosity, how hard is it not to use conda? As in if you have to compile Python from scratch to support something that you can't use a binary on, it is much easier to use PIP since that comes with Python. Does estrablish_conda_script.sh have a work around to disable it?

alfoa · 2018-06-06T14:18:58Z

@joshua-cogliati-inl I remember that PIP had problems in Windows in the past (but those problems might have been fixed). How do you think PIP is going to solve the problems we are currently facing?

joshua-cogliati-inl · 2018-06-06T14:29:48Z

Hm, glancing at establish_conda_script.sh it looks like if it does not find a conda command, it just uses the other methods that raven has, so PIP should still work (you just get a complaint every time) So my question is answered since there is a workaround.

Let me just say that running scripts to change the environment in a makefile is a challenge since you have to do it for every target, or call a second make. It might be simpler to just revive build_framework script and just always have make call it.

PaulTalbot-INL · 2018-06-06T14:56:59Z

Given Josh's comments, I think running the library establishment script by default in "make", with an option to opt out via an argument, probably makes a lot of sense.

It also might make sense sometime in the future to investigate using a pip environment with library "pipenv" as an alternative for systems that really don't want conda. So far I don't think we've had that request, but I could see it coming up.

I'll go ahead and modify the install in the docs and wiki. As far as I know, though, the Windows and Falcon test machines do not use the library script, which we should fix if we reasonably can.

…bs-update

…created

moosebuild · 2018-06-07T14:13:49Z

Job Mingw Test on 2d1808a : canceled by @PaulTalbot-INL

keeping device clear

moosebuild · 2018-06-07T16:56:16Z

Job Test mac on 87438f1 : invalidated by @PaulTalbot-INL

trying after removing secondary user environment

moosebuild · 2018-06-07T16:58:56Z

Job Test mac on 87438f1 : invalidated by @PaulTalbot-INL

again

moosebuild · 2018-06-07T17:02:24Z

Job Test mac on 87438f1 : invalidated by @PaulTalbot-INL

testing library removal after adding clean

PaulTalbot-INL · 2018-07-12T17:12:36Z

Okay, there are differences.

One-at-a-time has the following that at-once does not:

cairo 1.14.12
libgcc 7.2.0
libgfortran 3.0.0
libiconv 1.14
openblas 0.2.14
pixman 0.34.0
pycairo 1.13.3
Note that while libgcc and libgfortran are missing from at-once, they both have libgcc-ng and libgfortran-ng.

The following versions are different between them (one-at-a-time | at-once):

curl 7.49.0 | 7.60.0
fontconfig 2.12.4 | 2.12.6
libxml2 2.9.4 | 2.9.8
libxslt 1.1.29 | 1.1.32
lxml 4.1.1 | 4.2.2
numpy 1.11.2 | 1.14.2
qt 5.6.2 h974d657_12 | hd25b39d_14

I'll start poking through those difference and see if I can nail down when the fault starts occurring.

PaulTalbot-INL · 2018-07-12T17:32:56Z

Numpy 1.12.1 is the latest version of numpy I tried that did not result in the fault. 1.13.1 and 1.42.2 both had the issue (on falcon only), so I pinned numpy to 1.12.1.

…ests

…ose-dev-gcc as a test to qsub command

…bs-update

moosebuild · 2018-07-13T18:06:44Z

Job Test qsubs on 309059b : invalidated by @PaulTalbot-INL

cleared old hdf5 file, see h5py/h5py#712 for possibly-connected problem

…sues

wangcj05 · 2018-07-13T22:58:20Z

qsub failed for the following reason:

3764041.service2
Return code: 0
PASS mpiqsub_limitnode
rm: cannot remove `/home/moosetest/civet/build_0/raven/framework/test_qsub.[eo]*': No such file or directory
ERROR: exiting with code 1

@PaulTalbot-INL Do you have any idea on that?

wangcj05 · 2018-07-13T22:59:43Z

Makefile

@@ -32,6 +32,11 @@ endif

 CURR_DIR    := $(CURDIR)

+# touch hit.cpp to make sure its time stamp is different than hit.pyx
+## this is not a clean solution, but hopefully it prevents asking to use cython
+CYTHON_AVOIDANCE_ACTION=$(shell touch $(MOOSE_DIR)/framework/contrib/hit/hit.cpp)


Could you add fixme or todo for the comments?

We don't have an official use for TODO or FIXME; on the contrary, it's been suggested we do neither. The use of either of these right now is to flag developers that may want to edit the lines being commented. I don't think adding TODO or FIXME is a worthwhile change unless we define a real usage for them in our standards.

wangcj05 · 2018-07-13T23:01:21Z

build_raven

+MAKE_ARGS=" "
+LIBS_MODE=0 # 0 is "create", 1 is "load"
+#### FUTURE WORK:
+####   combine "make" and "establish_conda_env.sh" into one installation command


wangcj05 · 2018-07-13T23:03:44Z

build_raven

+#    --raven-libs-name)
+#      shift
+#      export RAVEN_LIBS_NAME="$1"
+#      ;;


The comments lines are for the future work?

Yes, if it turns out we don't need to environment variable, it would be nice to work all through the command line in the future; however, this option didn't work for an undetermined reason previously.

wangcj05 · 2018-07-13T23:04:02Z

build_raven

+#### END FUTURE WORK
+
+if [[ $LIBS_MODE == 0 ]]; then
+  #### FUTURE WORK:


wangcj05 · 2018-07-13T23:07:41Z

crow/crow_python_modules.mk

-	(cd $(CROW_DIR) && unset CXX CC && CFLAGS="$$CFLAGS $(COVERAGE_COMPILE_EXTRA)" && LDFLAGS="$$LDFLAGS $(COVERAGE_LINK_EXTRA)" && export CFLAGS LDFLAGS &&if test `uname` = Darwin; then MACOSX_DEPLOYMENT_TARGET=10.9; export MACOSX_DEPLOYMENT_TARGET; fi && . ./scripts/setup_raven_libs && python ./setup.py build_ext build install --install-platlib=./install)
+	(cd $(CROW_DIR) && unset CXX CC && CFLAGS="$$CFLAGS $(COVERAGE_COMPILE_EXTRA)" && LDFLAGS="$$LDFLAGS $(COVERAGE_LINK_EXTRA)" && export CFLAGS LDFLAGS &&if test `uname` = Darwin; then MACOSX_DEPLOYMENT_TARGET=10.9; export MACOSX_DEPLOYMENT_TARGET; fi && . ../scripts/establish_conda_env.sh --load && python ./setup.py build_ext build install --install-platlib=./install)
+	# without mac deployment target
+	# (cd $(CROW_DIR) && unset CXX CC && CFLAGS="$$CFLAGS $(COVERAGE_COMPILE_EXTRA)" && LDFLAGS="$$LDFLAGS $(COVERAGE_LINK_EXTRA)" && export CFLAGS LDFLAGS &&. ../scripts/establish_conda_env.sh --load && python ./setup.py build_ext build install --install-platlib=./install)


Do you need these commented lines?

These can be removed, I think. I'll do that.

wangcj05 · 2018-07-13T23:33:50Z

scripts/establish_conda_env.sh

+        echo ... \>\> If this is not the desirable path, rerun with argument --conda-defs \[path\] or remove the entry from raven/.ravenrc file.
+      fi
+    fi
+	fi


not properly indented

Indenting is not part of the bash syntax, and we don't have a standard for bash yet.

wangcj05 · 2018-07-13T23:43:30Z

tests/cluster_tests/run_mpi_test.sh

-module load raven-devel-gcc
+module load raven-devel
+module load MVAPICH2/2.0.1-GCC-4.9.2
+source activate raven_libraries


Is conda activate available?

Defining the "conda" functions within a script is challenging (see establish_conda_env.sh) and often isn't worth it. For now, using "source activate" is more efficient; hopefully by Conda 4.6 (or later) the conda devs figure out how they want us to do things; right now both are acceptable.

wangcj05 · 2018-07-13T23:45:06Z

tests/framework/unit_tests/DataObjects/TestDataSet.py

-netname = 'DataSetUnitTest.nc'
-data.write(netname,style='netcdf',format='NETCDF4') # WARNING this will fail if netCDF4 not installed
-checkTrue('Wrote to netcdf',os.path.isfile(netname))
+# NOTE: due to a cool little seg fault error in netCDF4 versions less than 1.3.1, we cannot test it currently.


add TODO or FIXME here

wangcj05 · 2018-07-13T23:45:27Z

tests/framework/unit_tests/DataObjects/TestHistorySet.py

-netname = 'HistorySetUnitTest.nc'
-data.write(netname,style='netcdf',format='NETCDF4') # WARNING this will fail if netCDF4 not installed
-checkTrue('Wrote to netcdf',os.path.isfile(netname))
+# NOTE: due to a cool little seg fault error in netCDF4 versions less than 1.3.1, we cannot test it currently.


same as previous comment

wangcj05 · 2018-07-13T23:45:38Z

tests/framework/unit_tests/DataObjects/TestPointSet.py

-netname = 'PointSetUnitTest.nc'
-data.write(netname,style='netcdf',format='NETCDF4') # WARNING this will fail if netCDF4 not installed
-checkTrue('Wrote to netcdf',os.path.isfile(netname))
+# NOTE: due to a cool little seg fault error in netCDF4 versions less than 1.3.1, we cannot test it currently.


PaulTalbot-INL · 2018-07-16T19:21:35Z

While it appears the linux test is hanging on the GitHub report, it has passed on civet. Note there was a disruption on GitHub today for about an hour that may have caused this disconnect.

wangcj05 · 2018-07-16T19:25:43Z

All the tests are green except the GitHub issue mentioned by @PaulTalbot-INL, and the checklist is satisfied.

PaulTalbot-INL added 3 commits May 29, 2018 12:08

trying new library versions (manually update if script not run)

f65cb12

matplotlib 2 update regolding, values were indistinguishable

1b21980

added optional PIL install, also fixed docs for raven code interface

8bb23bd

PaulTalbot-INL requested a review from alfoa May 30, 2018 13:57

alfoa approved these changes Jun 6, 2018

View reviewed changes

PaulTalbot-INL and others added 9 commits June 6, 2018 12:35

stash, working on installation updates

2f68d22

update libraries with tensorflow

e1f7559

Merge branch 'libs-update' of github.com:PaulTalbot-INL/raven into li…

3049f9b

…bs-update

edit

953740f

improving compatability

39801a2

Merge branch 'libs-update' of github.com:PaulTalbot-INL/raven into li…

cc061e9

…bs-update

removing tensorflow for now due to Windows not liking py27

e9336b6

installation docs still undergoing; os-dependent establish conda env …

4d9fafd

…created

added default location for windows

2d1808a

PaulTalbot-INL changed the title ~~Libs update~~ [WIP] Libs update Jun 7, 2018

PaulTalbot-INL added 4 commits June 7, 2018 08:35

verbosity improvement

7b13f5d

improving string check on est conda env os

9363195

fixed inclusions

7105c11

new feature: skip if os

87438f1

PaulTalbot-INL changed the title ~~[WIP] Libs update~~ Libs update Jun 7, 2018

renamed namespace variable

738f7be

pinned numpy version

e3b235a

PaulTalbot-INL added 4 commits July 13, 2018 08:53

undoing spacing/precision changes from numpy 14 to numpy 12 in text t…

730905e

…ests

some libraries appear to be missing for mpi in raven-devel, adding mo…

2d456a3

…ose-dev-gcc as a test to qsub command

Merge branch 'libs-update' of github.com:PaulTalbot-INL/raven into li…

602e5e8

…bs-update

conda activations in qsub command scripts

309059b

PaulTalbot-INL added 4 commits July 13, 2018 12:07

clearing database storage

7b1ef48

stashing for devel checks

272629e

added one module for qsub, also nailed down version of hdf5 for rw is…

78ba72f

…sues

fixed mpi run test to use new module

f8733df

wangcj05 self-requested a review July 13, 2018 22:58

wangcj05 requested changes Jul 13, 2018

View reviewed changes

PaulTalbot-INL added 8 commits July 16, 2018 08:51

prevent removal failure from exiting, perhaps

06871a7

review comments

b1f9233

remaining review comments

8d4ce41

working locally, attempting to fix

2d5b1c6

iterating

7685502

iterating

cd0f15b

iterating

acf7aca

cleanup, working now with raven_framework instead of Driver

f475b14

PaulTalbot-INL mentioned this pull request Jul 16, 2018

Outdated RAVEN code interface docs #666

Closed

10 tasks

wangcj05 approved these changes Jul 16, 2018

View reviewed changes

wangcj05 merged commit 6737843 into idaholab:devel Jul 16, 2018

moosebuild mentioned this pull request Jul 16, 2018

Raven runs Raven on a cluster #684

Merged

8 tasks

PaulTalbot-INL deleted the libs-update branch April 8, 2021 20:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Libs update #668

Libs update #668

PaulTalbot-INL commented May 29, 2018 •

edited by wangcj05

Loading

alfoa left a comment

alfoa commented Jun 6, 2018 •

edited

Loading

joshua-cogliati-inl commented Jun 6, 2018

alfoa commented Jun 6, 2018 •

edited

Loading

joshua-cogliati-inl commented Jun 6, 2018 •

edited

Loading

PaulTalbot-INL commented Jun 6, 2018 •

edited

Loading

moosebuild commented Jun 7, 2018

moosebuild commented Jun 7, 2018

moosebuild commented Jun 7, 2018

moosebuild commented Jun 7, 2018

PaulTalbot-INL commented Jul 12, 2018

PaulTalbot-INL commented Jul 12, 2018

moosebuild commented Jul 13, 2018

wangcj05 commented Jul 13, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL Jul 14, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL Jul 14, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL Jul 14, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL Jul 14, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL Jul 14, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL Jul 14, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL Jul 14, 2018

PaulTalbot-INL Jul 14, 2018

wangcj05 Jul 13, 2018

wangcj05 Jul 13, 2018

wangcj05 Jul 13, 2018

PaulTalbot-INL commented Jul 16, 2018 •

edited

Loading

wangcj05 commented Jul 16, 2018

Libs update #668

Libs update #668

Conversation

PaulTalbot-INL commented May 29, 2018 • edited by wangcj05 Loading

Pull Request Description

What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)

What are the significant changes in functionality due to this change request?

For Change Control Board: Change Request Review

alfoa left a comment

Choose a reason for hiding this comment

alfoa commented Jun 6, 2018 • edited Loading

joshua-cogliati-inl commented Jun 6, 2018

alfoa commented Jun 6, 2018 • edited Loading

joshua-cogliati-inl commented Jun 6, 2018 • edited Loading

PaulTalbot-INL commented Jun 6, 2018 • edited Loading

moosebuild commented Jun 7, 2018

moosebuild commented Jun 7, 2018

moosebuild commented Jun 7, 2018

moosebuild commented Jun 7, 2018

PaulTalbot-INL commented Jul 12, 2018

PaulTalbot-INL commented Jul 12, 2018

moosebuild commented Jul 13, 2018

wangcj05 commented Jul 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PaulTalbot-INL commented Jul 16, 2018 • edited Loading

wangcj05 commented Jul 16, 2018

PaulTalbot-INL commented May 29, 2018 •

edited by wangcj05

Loading

alfoa commented Jun 6, 2018 •

edited

Loading

alfoa commented Jun 6, 2018 •

edited

Loading

joshua-cogliati-inl commented Jun 6, 2018 •

edited

Loading

PaulTalbot-INL commented Jun 6, 2018 •

edited

Loading

PaulTalbot-INL commented Jul 16, 2018 •

edited

Loading