Replace cdms2 with netCDF4 in PrePARE and Python tests #493

mauzey1 · 2019-05-24T18:39:39Z

Resolves #490

This PR replaces the use of cdms2 in PrePARE and some of the Python tests with netCDF4.

Note: The MacOS CircleCI checks failed due to an error when creating the conda environment.

MD5MismatchError: Conda detected a mismatch between the expected content and downloaded content
for url 'https://repo.anaconda.com/pkgs/main/osx-64/libcxx-4.0.1-h579ed51_0.tar.bz2'.
  download saved to: /Users/distiller/project/workspace/test_macos_cmor/miniconda/pkgs/libcxx-4.0.1-h579ed51_0.tar.bz2
  expected md5 sum: 28471619a3a79da80d9b5747f1ec0221
  actual md5 sum: 291e61684d014641092020bdb1acb096

However, the changes that triggered the failed tests (9f56aa6) didn't change the CircleCI configuration script. The previous commit (8ee7112) that did change the CircleCI and Travis scripts was passing tests.

…ecking data types, and removed the index from _FillValue

doutriaux1 · 2019-05-24T18:45:44Z

@muryanto1 this is the same error as you. I think a package got corrupted on their server

doutriaux1 · 2019-05-24T18:46:14Z

@mauzey1 can we have some test with cdms and some w/o like that we test both ways

doutriaux1 · 2019-05-24T18:46:55Z

LibCV/PrePARE/PrePARE.py

@@ -457,15 +457,15 @@ def ControlVocab(self, ncfile, variable=None, print_all=True):
        for attr in ['branch_time_in_child', 'branch_time_in_parent']:
            if attr in list(self.dictGbl.keys()):
                self.set_double_value(attr)
-                if not isinstance(self.dictGbl[attr], numpy.float64):
+                if not numpy.issubdtype(self.dictGbl[attr], numpy.float64):


doutriaux1 · 2019-05-24T18:47:36Z

LibCV/PrePARE/PrePARE.py

@@ -535,7 +535,7 @@ def ControlVocab(self, ncfile, variable=None, print_all=True):
        # -------------------------------------------------------------------
        varid = cmip6_cv.setup_variable(variable_cmor_entry,
                                        self.dictVar['units'],
-                                        self.dictVar['_FillValue'][0],


is this due to the switch from cdms2 to netcdf?

cdms2 was returning values as numpy ndarrays even if they were just one value. netCDF4 returns numpy scalars for single values.

mauzey1 · 2019-05-29T16:53:18Z

@doutriaux1 Should we include cdms2 along side netCDF4, or just use cdms2 in the testing environment for cdms tests?

mauzey1 · 2019-05-29T23:01:51Z

@doutriaux1 The tests that I previously modified to use netCDF4 instead of cdms2 now have test cases for both. cdms2 is installed in the testing environment but not for the build.

…on_toomany_tables.py, and commented out deletion of CMIP6 folder

…issing tests in Makefile, and placed the "2Gb" in a separate section.

mauzey1 · 2019-06-04T18:26:36Z

@doutriaux1
The test test_python_max_variables.py is failing on the MacOS checks. It's giving this error:

HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140735957398464:
  #000: H5F.c line 444 in H5Fcreate(): unable to create file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1364 in H5F__create(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1559 in H5F_open(): unable to open file: name = 'CMIP6/CMIP6/ISMIP6/PCMDI/PCMDI-test-1-0/piControl-withism/r3i1p1f1/Amon/ta/gn/v20190531/ta_Amon_PCMDI-test-1-0_piControl-withism_r3i1p1f1_gnaG4OFK13537.nc', tent_flags = 13
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #004: H5FDsec2.c line 346 in H5FD_sec2_open(): unable to open file: name = 'CMIP6/CMIP6/ISMIP6/PCMDI/PCMDI-test-1-0/piControl-withism/r3i1p1f1/Amon/ta/gn/v20190531/ta_Amon_PCMDI-test-1-0_piControl-withism_r3i1p1f1_gnaG4OFK13537.nc', errno = 24, error message = 'Too many open files', flags = 13, o_flags = 602
    major: File accessibilty
    minor: Unable to open file

I suspect it is due to it creating a lot of open file handles faster than it can close them.

The Python 3.7 MacOS check is getting an error at line 153 in Test/test_python_joerg_3.py.

cmor/Test/test_python_joerg_3.py

Lines 148 to 154 in 36351d7

    
               file1 = cmor.close(ivar1, True) 
        
               file2 = cmor.close(ivar2, True) 
        
               print('File1:', file1) 
        
               print('File2:', file2) 
        
               cmor.close() 
        
           print(cmor.close(ivar1, True)) 
        
           cmor.close()

It produces this error:

Traceback (most recent call last):
  File "Test/test_python_joerg_3.py", line 153, in <module>
    print(cmor.close(ivar1, True))
  File "/Users/distiller/project/workspace/test_macos_cmor/miniconda/envs/py3.7/lib/python3.7/site-packages/CMOR-3.4.0-py3.7-macosx-10.9-x86_64.egg/cmor/pywrapper.py", line 944, in close
    return _cmor.close(var_id, 1, 0)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaf in position 1: invalid start byte
make[1]: *** [test_a_python] Error 1
make: *** [test_python] Error 2
Exited with code 2

It seems that Python 3.7 on OSX is having an issue with CMOR returning a file name when closing a variable that was already closed.

mauzey1 · 2019-06-04T20:43:49Z

@doutriaux1 Looking at the function cmor_close_variable, the parts that set the parameter file_name are skipped if the variable is closed. This will cause _cmor.close in pywrapper.py to return an uninitialized string.

Should we set file_name to an empty string if the variable is closed? Or, should we throw an error or warning if attempting to close an already closed variable?

edit: If preserve is not NULL, then the variable's file name should still exist otherwise it gets set to an empty string. So we should be passing the file name even if the variable is closed.

mauzey1 · 2019-06-05T18:41:08Z

@taylor13 @durack1 What should be the behavior of cmor_close(var_id) if var_id is for an already closed variable, or if CMOR is closed? Warnings or errors? Should it still return a file name for the variable if it was closed with preserve set to true?

taylor13 · 2019-06-05T18:59:55Z

In this case I think a "Warning" should be raised (not an "error"). If "preserve" is set to true, I think is would be nice to return the file name (even if the file had already been closed), but if this is difficult to do, then don't bother. [We should probably document the behavior in the CMOR documentation.

mauzey1 · 2019-06-06T00:18:24Z

I'm not sure about how to handle the test_python_max_variables.py error. It seems to stem from an open file descriptor limit on CircleCI's MacOS. It runs fine on my Macbook. All it seems to be doing is creating, writing, and closing a variable 600 times. I will comment it out of the Makefile for the time being.

mauzey1 · 2019-06-11T16:30:49Z

@doutriaux1 Do you think this pull request can be approved? Should I put the changes for the inclusion of the Python tests in another PR?

doutriaux1 · 2019-06-11T16:50:07Z

@mauzey1 how does make test knows about which tests need cdms2 and which do not?

mauzey1 · 2019-06-11T17:19:42Z

@doutriaux1 There are some Python tests, such as test_python_history.py, that check if cdms2 is installed. If not installed then the test will exit.

There are similar checks for cdms2 in the CMIP6 CV tests. The code below will skip the test case if cdms2 is not installed.

cmor/Test/test_python_CMIP6_CV_trackingNoprefix.py

Lines 64 to 66 in 0d7618a

    
           @unittest.skipIf(pkgutil.find_loader("cdms2") is None, "cdms2 not installed") 
        
           def testCMIP6_CDMS2(self): 
        
               import cdms2

Since cdms2 is a Python libary, should we only rerun the Python tests after installing cdms2?

doutriaux1 · 2019-06-19T19:41:06Z

@mauzey1 looks good.

mauzey1 added 3 commits May 23, 2019 17:13

Replace cdms2 with netCDF4 in PrePARE and CMIP6 CV Python tests

4edbe5c

Replaced cdms2 with netcdf4 in CircleCI and Travis scripts

8ee7112

Fixed integer check, replaced isinstance with numpy.issubdtype for ch…

9f56aa6

…ecking data types, and removed the index from _FillValue

mauzey1 requested review from doutriaux1 and durack1 May 24, 2019 18:39

mauzey1 changed the title ~~Replace cdms2 with netCDF4~~ Replace cdms2 with netCDF4 in PrePARE and Python tests May 24, 2019

doutriaux1 reviewed May 24, 2019

View reviewed changes

Add cdms2 unit tests

db393d8

mauzey1 added 13 commits May 30, 2019 17:40

Allow type to be unicode for strings

f566cfa

Change PyBytes_AsString to PyUnicode_AsUTF8

c8b1578

Accept numpy.int64 and numpy.float64 values in Python wrapper

2452daa

Use time1 for double singleton test

12cfd7e

fixed file path; fixed time axis values for test_1hr

e517ade

check if cdms2 is installed before running test

1b89f52

Use byte strings for types in test_python_direct_calls.py

cbd14ec

make values integers

d00c824

Change print for Python 3

10c18a3

fixed creation of values array in test_python_joerg_tim2_clim_02.py

ab0f3d5

Fixed error "ValueError: can't have unbuffered text I/O" in test_pyth…

f8c57b8

…on_toomany_tables.py, and commented out deletion of CMIP6 folder

Adding Python tests to continuous integration checks. Commented out m…

daed92c

…issing tests in Makefile, and placed the "2Gb" in a separate section.

Remove cdms2 from environment creation step in travis script

36351d7

mauzey1 mentioned this pull request Jun 5, 2019

Python tests in Makefile are not being used in CI test suite #497

Closed

Remove cmor.close lines at end of test_python_joerg_3.py

049c759

mauzey1 added 3 commits June 5, 2019 17:27

Disable test_python_max_variables.py in Makefile

d262743

Disable test_python_max_variables_2.py in Makefile

50be160

Remove check for cdms2 from tests that don't need it.

0d7618a

mauzey1 mentioned this pull request Jun 11, 2019

Add libnetcdf 4.6.2 to the run requirements of meta.yaml.in #489

Closed

mauzey1 mentioned this pull request Jun 12, 2019

Throw exception if not enough data is passed to cmor.write in Python #500

Merged

doutriaux1 approved these changes Jun 19, 2019

View reviewed changes

mauzey1 merged commit b8e0bfc into master Jun 19, 2019

doutriaux1 deleted the 490_replace_cdms2_with_netcdf4 branch June 19, 2019 19:50

mauzey1 mentioned this pull request Jun 19, 2019

Add netcdf4 to the conda environment creation step PCMDI/cmor3_documentation#67

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace cdms2 with netCDF4 in PrePARE and Python tests #493

Replace cdms2 with netCDF4 in PrePARE and Python tests #493

mauzey1 commented May 24, 2019

doutriaux1 commented May 24, 2019

doutriaux1 commented May 24, 2019

doutriaux1 May 24, 2019

doutriaux1 May 24, 2019

mauzey1 May 24, 2019

mauzey1 commented May 29, 2019

mauzey1 commented May 29, 2019

mauzey1 commented Jun 4, 2019

mauzey1 commented Jun 4, 2019 •

edited

Loading

mauzey1 commented Jun 5, 2019

taylor13 commented Jun 5, 2019

mauzey1 commented Jun 6, 2019

mauzey1 commented Jun 11, 2019

doutriaux1 commented Jun 11, 2019

mauzey1 commented Jun 11, 2019

doutriaux1 commented Jun 19, 2019

Replace cdms2 with netCDF4 in PrePARE and Python tests #493

Replace cdms2 with netCDF4 in PrePARE and Python tests #493

Conversation

mauzey1 commented May 24, 2019

doutriaux1 commented May 24, 2019

doutriaux1 commented May 24, 2019

doutriaux1 May 24, 2019

Choose a reason for hiding this comment

doutriaux1 May 24, 2019

Choose a reason for hiding this comment

mauzey1 May 24, 2019

Choose a reason for hiding this comment

mauzey1 commented May 29, 2019

mauzey1 commented May 29, 2019

mauzey1 commented Jun 4, 2019

mauzey1 commented Jun 4, 2019 • edited Loading

mauzey1 commented Jun 5, 2019

taylor13 commented Jun 5, 2019

mauzey1 commented Jun 6, 2019

mauzey1 commented Jun 11, 2019

doutriaux1 commented Jun 11, 2019

mauzey1 commented Jun 11, 2019

doutriaux1 commented Jun 19, 2019

mauzey1 commented Jun 4, 2019 •

edited

Loading