CMOR tests

[hot007]

Hi there,

We're trying to start CMORising ACCESS model output, and hitting some walls that are almost certainly in our post-processing code, but we're trying to debug them at the moment. As I installed the CMOR library I thought maybe I should check that CMOR is behaving as expected, by running the embedded unit tests. I couldn't find any doco on how you're meant to do that, so I've run "python -m pytest ." in the cmor library folder (containing the Test directory). I had to copy a bunch of table files and tas_sample.nc in after various errors, initially there was cmor.write errors, once all files were present I get a segmentation fault. This occurred for both CMOR 3.3.3 and 3.4.
I have tried running this with up to 8GB memory and unlimited stacksize so I don't imagine it's a memory problem now, can you please point me to any instructions on how to run the tests from a conda install and whether we'd expect them to pass?

Thanks,
Claire

[durack1]

Thanks @hot007, can you provide a little more context as to how you have configured CMOR - did you install via conda, from source etc etc?

It'd also be useful to know/see snippets of the segfaults and any CMOR log history.

[hot007]

Hi Paul,

Yep, installed via conda on NCI's Raijin as per the instructions https://cmor.llnl.gov/mydoc_cmor3_conda/ leveraging an existing climate community conda set up so that it's appropriately HDF/netCDF/udunits aware (afaik).

I don't think I can give much useful info re output - once it stopped giving errors about missing files all I get is the segfault. I'm not sure how to set a cmor log file when just calling tests (help?) but please find attached some portion of stdout from when it was producing errors, followed by me copying in some remaining missing files, trying again and exceeding job memory (4GB), trying again with 8GB and simply getting a segfault during test collection.
CMOR-dump.txt

[hot007]

I don't know whether this is worth mentioning, but note that when it was producing output it seems like it was having issues in cmor.write; that is also where we're experiencing issues with our ACCESS post-processor, so I'd like to be sure we can run tests that rely on cmor.write.

[doutriaux1]

@hot007 looks like cmor cannot find the tables. I believe the examples look for it in the Tables directory right under where you run them from. Did you clone the cmip6-cmor-tables? IF you create a symlink to their repo's Tables directory. @mauzey1 can you please follow up on this? Thanks.

[mauzey1]

The best way I have gotten to run the unit tests is to clone the CMOR repo, and install cmor with testsrunner.

git clone git@github.com:PCMDI/cmor.git
cd cmor
conda create -n cmor-3.4.0 -c cdat/label/nightly -c cdat -c conda-forge cmor cdms2 testsrunner
python run_tests.py -v2 -H -n1 Test/test_python_CMIP6_CV_*.py

I tried running run_tests.py in the same git clone directory but with the tests stored in the conda environment. It gave these errors for every tests:

Executing nosetests -s /home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/cmor/Test/test_python_CMIP6_CV_badgridgr.py in /home/mauzey1/git/cmor
E
======================================================================
ERROR: test suite for <module 'cmor.Test.test_python_CMIP6_CV_badgridgr' from '/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/cmor/Test/test_python_CMIP6_CV_badgridgr.py'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/nose/util.py", line 469, in try_run
    return func(obj)
  File "/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/cmor/pywrapper.py", line 877, in setup
    if not isinstance(inpath, str) and not os.path.exists(inpath):
  File "/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/genericpath.py", line 26, in exists
    os.stat(path)
TypeError: coercing to Unicode: need string or buffer, module found
----------------------------------------------------------------------
Ran 0 tests in 0.002s
FAILED (errors=1)
FAILED...cmd: ['nosetests', '-s', '/home/mauzey1/anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/cmor/Test/test_python_CMIP6_CV_badgridgr.py']

Even if I copied the Tables directory, Test/CMOR_input_example.json, and run_tests.py to the conda environment folder it would still produce those errors.

If I use pytest instead of run_tests.py, all of the CMIP6_CV tests will pass except for test_python_CMIP6_CV_terminate_signal.py

====================================================== FAILURES =======================================================
____________________________________________ TestCase.testTerminateSignal _____________________________________________

self = <cmor.Test.test_python_CMIP6_CV_terminate_signal.TestCase testMethod=testTerminateSignal>

    def testTerminateSignal(self):
>       self.assertEqual(cmor.get_terminate_signal(), -999)
E       AssertionError: 15 != -999

../../anaconda2/envs/cmor-3.4.0/lib/python2.7/site-packages/cmor/Test/test_python_CMIP6_CV_terminate_signal.py:29: AssertionError

[hot007]

Thanks for the suggestions @mauzey1. I will try again with the testrunner env.

In the meantime, I upped my memory request to 32GB and managed to get output without segfaults.
Many of the tests fail because they depend on data files which are not in the repo.

I'm thinking maybe a lot of the tests are simply not viable in a general environment, I'm not sure where I'd get these files from (they don't look like something that'll be available through Synda), so I thought I'd look for tests that have cmor.write issues, as that's what I'm trying to rule out, and run them specifically. However even when a test seems to depend on general data (e.g. data/tas_sample.nc), I'm still finding similar problems. e.g.

$ python -m pytest Test/test_python_user_interface_03.py
========================= test session starts ============================
platform linux2 -- Python 2.7.15, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cmor, inifile:
collected 0 items / 1 errors                                                   

============================= ERRORS ==============================
____________ ERROR collecting Test/test_python_user_interface_03.py ____________
Test/test_python_user_interface_03.py:88: in <module>
    data, data_ordered = read_input(var, order=o)
Test/test_python_user_interface_03.py:22: in read_input
    f = cdms2.open(os.path.join(dpth, "%s_sample.nc" % var))
../cdms2/dataset.py:487: in openDataset
    return CdmsFile(path, mode, mpiBarrier=CdMpi)
../cdms2/dataset.py:1253: in __init__
    raise CDMSError('Cannot open file %s (%s)' % (path, err))
E   CDMSError: Cannot open file /short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cmor/data/tas_sample.nc (No error)
------------------------------- Captured stdout --------------------------------
Done importing
Testing var: tas
	ordering: tyx...
------------------------------- Captured stderr --------------------------------
CDMS system error: No such file or directory
CDMS I/O error: Opening file /short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cmor/data/tas_sample.nc
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
======================= 1 error in 2.05 seconds ========================

Put the sample file in and rerun:

$ mv data/tas_sample.nc.mv data/tas_sample.nc 
(CMOR3-4) [ct5255@raijin1 cmor]$ python -m pytest Test/test_python_user_interface_03.py
======================== test session starts ==========================
platform linux2 -- Python 2.7.15, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cmor, inifile:
collected 0 items / 1 errors                                                   

============================= ERRORS ============================
____________ ERROR collecting Test/test_python_user_interface_03.py ____________
Test/test_python_user_interface_03.py:98: in <module>
    f = cdms2.open(fn)
../cdms2/dataset.py:487: in openDataset
    return CdmsFile(path, mode, mpiBarrier=CdMpi)
../cdms2/dataset.py:1253: in __init__
    raise CDMSError('Cannot open file %s (%s)' % (path, err))
E   CDMSError: Cannot open file /software/cmor3/cmor/CMIP6/CMIP6/ISMIP6/PCMDI/PCMDI-test-1-0/piControl-withism/r11i1p1f1/Amon/tas/gr/v20190204/tas_Amon_PCMDI-test-1-0_piControl-withism_r11i1p1f1_gr_197901-199605.nc (Variable not found)
------------------------------- Captured stdout --------------------------------
Done importing
Testing var: tas
	ordering: tyx...
(206, 45, 72)
0 units: days since 1979-1-1 0 15.5
------------------------------- Captured stderr --------------------------------
! ------
! CMOR is now closed.
! ------
! During execution we encountered:
!   8 Warning(s)
!   0 Error(s)
! ------
! Please review them.
! ------
! 
CDMS system error: No such file or directory
CDMS I/O error: Opening file /software/cmor3/cmor/CMIP6/CMIP6/ISMIP6/PCMDI/PCMDI-test-1-0/piControl-withism/r11i1p1f1/Amon/tas/gr/v20190204/tas_Amon_PCMDI-test-1-0_piControl-withism_r11i1p1f1_gr_197901-199605.nc
========================= warnings summary =========================
/short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cdms2/axis.py:825
  /short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cdms2/axis.py:825: UserWarning: genutil module not present, was not able to determine if axis is level based on units
    "genutil module not present, was not able to determine if axis is level based on units")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
==================== 1 warnings, 1 error in 2.07 seconds =====================

We seem to have lots of hardcoded paths in these tests for files that aren't in the data directory.

So, maybe I should open a separate issue for this question, but can you tell me how we could confirm that cmor.write is behaving as expected? Ideally, can you tell me what is a test that should write lat, lon, time and data (e.g. tas) that we could modify to try on our locally produced data, with the assumption that the unit tests do work as expected?
Thanks!

[mauzey1]

Here is a branch of CMOR with a correction made to the NetCDF path in Test/test_python_user_interface_03.py
https://github.com/PCMDI/cmor/tree/fixed_user_interface_03_test
https://github.com/PCMDI/cmor/blob/fixed_user_interface_03_test/Test/test_python_user_interface_03.py

[hot007]

Okay, that code looks better... at the risk of making @durack1 say "told you so" (#206) though, I'm now getting the "genutil module not present" warning. I don't imagine it's fatal but the only output from pytest I then get is the warning, no info about tests passed or failed or if anything was done (collected 0 items?). Maybe this is expected behaviour, but I'm guessing not? Is there a particular way I should try to install genutil? This feels like a red herring but not sure what to do with it...

python -m pytest Test/test_python_user_interface_03.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.15, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
rootdir: /short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cmor, inifile:
collected 0 items
=============================== warnings summary ===============================
/short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cdms2/axis.py:825
/short/p66/ct5255/conda/envs/CMOR3-4/lib/python2.7/site-packages/cdms2/axis.py:825: UserWarning: genutil module not present, was not able to determine if axis is level based on units
"genutil module not present, was not able to determine if axis is level based on units")
-- Docs: https://docs.pytest.org/en/latest/warnings.html
========================== 1 warnings in 3.88 seconds ==========================

[durack1]

@hot007 glad you’re up and running. I personally think genutol should be added to the env recipe as a dependency, but @doutriaux1 seems opposed to this, no idea why..

@taylor13 @mauzey1 @dnadeau4

[hot007]

I've installed genutil via cdat-lite, I'm not sure that was the right choice though, as the test seems to require "cdms2 interface" which google implies is UVCDAT but that in principle requires its own conda env. How is this test usually run?? Sorry to be such a pain, I'm just not sure what additional dependencies are required for testing.

[durack1]

@hot007 conda install -c conda-forge genutil should work, kinda depends how you installed CMOR though.. from source it sounds..

[hot007]

I am in a conda env so can do... interestingly that wants to downgrade CMOR to 3.3.3. Dunno if that's a bug...

[durack1]

@hot007, cdms2 was just updated, and the dependency chain is working its way through, @dnadeau4 could provide a little more background

[taylor13]

Just to note, that I support the notion that CMOR should not need CDMS or UVCDAT or genutil as dependencies. It was (originally) designed as a "writer", not a "reader" and interacts directly with the netCDF library (and that library has dependencies on hdf and other packages). The fewer the dependencies, the simpler the build, and the fewer problems will be encountered. CMOR does rely on udunits, so that is a dependency.

It is possible that PrePARE has added features that require other dependencies. Perhaps @dnadeau4 or @doutriaux1 or @mauzey1 can advise.

[mauzey1]

Although cdms2 is used by the CMIP6 CV Python tests, the tests with the pattern Test/test_python_CMIP6_CV*.py, I don't see the genutil warning showing up when running them like it does with test_python_user_interface_03.py.

The reason why pytest collected 0 items from test_python_user_interface_03.py is due to the test not being in a function with a name that starts with "test". Fox example, https://github.com/PCMDI/cmor/blob/master/Test/test_python_CMIP6_CV_badinstitutionID.py

Currently, we are using the test_python_CMIP6_CV*.py tests for our Python unit tests on CircleCI and Travis. These tests are run with the TestsRunner module. https://github.com/CDAT/TestsRunner

conda create -n cmor-3.4.0 -c cdat/label/nightly -c cdat -c conda-forge cmor cdms2 testsrunner
python run_tests.py -v2 -H -n1 Test/test_python_CMIP6_CV_*.py

[hot007]

Okay, thanks @mauzey1 that's good to know that the test_python_CMIP6 tests are what you're using with CI. I'll give them a go and see if I have more luck. cheers.

[hot007]

Thanks @mauzey1 , makes sense to run the same tests you're using for CircleCI/Travis. I had some multiprocessing.pool hangs with run_tests but seemed otherwise okay, and all clear/as expected with pytest. So, the good news is I'm going to assume that means our installation is all good, this issue can be closed. As you'll have seen in #440 we tracked down what the bug we were seeing was and it certainly seems like it's not related to an installation problem. Thanks all for your assistance!