Update cmor3 for python3

[markcollier]

Just to mention that I have been able to successfully run cmor3 with a new anaconda installation of software (python=2.7) - I plan to create a python=3.6 version of my cmor3 based software removing packages that only exist in python2*, this means removing dependancies on packages like cdms2 that I think are only python2* currently (some packages are only currently supported in python3 like xesmf that I need to utilise for regridding).
It was easy to setup my cmor3 input file with sensible values and no issues with items like license last time. Previously I had to establish my own python environment, but now anaconda appears to be flexible enough for most applications.
Thanks for the great work.

[doutriaux1]

@markcollier we now have all packages (except cmor3 python maybe) available in python3.

try:
Linux:

conda create -q -n py3 -c nesii/channel/dev-esmf  -c uvcdat/label/nightly -c conda-forge -c uvcdat "python>3" "cdms2>2.12.2017" cdat_info udunits2 nose flake8 genutil "matplotlib<2.1" vtk-cdat mesalib image-compare "dv3d>2.12.2017" "cdutil>2.12.2017" nbsphinx easydev vcs

Mac:

conda create -q -n py3 -c uvcdat/label/nightly -c nadeau1  -c conda-forge -c uvcdat "cdms2>2.12.2017" nose flake8 "python>3" cdat_info udunits2 mesalib nose image-compare flake8 "matplotlib<2.1" numpy=1.13 image-compare genutil vtk-cdat "dv3d>2.12.2017" "cdutil>2.12.2017" vcs

Remove packages you don't really need like vcs sphinx etc...

[dnadeau4]

@markcollier thanks a lot for you comment!

We have ESMF with Python 3/2 integrated in CDMS, similar to XESMF, but I think CDMS is more powerful and allows regridding in ways that xesmf does not. I have a lot of tests in the CDMS directory. In CDMS you can generate your own grid, cartesian or spherical. You can mask your data. I have an example where you can regrid the model mask land/ocean you fit you regridded data.

Let me know if you want some help on you file, I will be happy to give you support.

https://github.com/UV-CDAT/cdms/tree/master/tests

[dnadeau4]

I am still working on cdms documentation, but here is the draft.
http://cdms.readthedocs.io/en/cdmsdocsmerge/

[markcollier]

I have tried installing cdms2 using conda with python=3.6 but it always wants to downgrade to python=2.7
Is there some way to overcome this, maybe I am looking at wrong documentation? This is why I tried to remove cdms2 from my software.

[dnadeau4]

@doutriaux1 do you know why this is happening? Python 3 was done last September. Do you have a special channel under conda for it?

[dnadeau4]

@markcollier It seems that cdms with python3 is under the channel uvcdat/label/nightly

These are the channels needed. (not nesii is for ESMF python3)

- uvcdat/label/nightly
- nesii/channel/dev-esmf
- conda-forge
- defaults

[dnadeau4]

The documentations is here. I am writing a new chapter for ESMF, but the SCRIP one is very similar.
http://cdms.readthedocs.io/en/cdmsdocsmerge/

[doutriaux1]

@markcollier did you try the command I posted above? They come straight from our test suite.

[markcollier]

Yes it worked, I modified it to leave our some packages that seemed unnecessary. I could also install xemsf on top of it plus some other packages at environment creation time. I'll keep playing around with it and adding in new packages, but I usually clone the environment for testing to ensure I don't break an installation. Didn't need to explicitly specify cmor as I suspect it is in one of the standard packages...

[markcollier]

do you expect to have cmor3 in python? I had a some mixed environments and it was working for the wrong reason. I would really like to move to python3 asap.

[dnadeau4]

@markcollier cmor3 won't be available for python3. I would have to change all the C AP* and CMIP6 has already started to roll.

[markcollier]

Yes and seems like you don't have a whole lot of coding support, pity the CMIP6 community cannot support a software developer for activities like this. Python2 will not be "supported" beyond 2020 and this will still be an important period of the CMIP6 delivery: https://pythonclock.org/
For me personally CMOR is an important part of the file generation process as the output files are so useful regardless of CMIP6 - for example, our group has no formal participation in CMIP6 (and I was made redundant in the group developing models for CMIP6 participation) but I have strongly advocated for our files to be rewritten in CMIP6 format for all the obvious reasons. However, I have to maintain and python2 & 3 environment solely for the need of CMOR...it would be good if we could find resources to hire someone to port the library before 2020 so that python coders like me can ween themselves off python2 entirely...lucky for anaconda and the ability to develop independent software environs.

[taylor13]

Thanks, Mark, for your ongoing interest. I too believe that by the end of this year it would be good to port CMOR to python 3. Perhaps this summer we could get a summer student to do this? Does that seem like a good idea @dnadeau4 ?

[dnadeau4]

I was thinking of havingCMOR with Python 3 for CMIP7.

It should not be too difficult now that we have the experience of CDMS to Python 3. That was never in the plans to have CMOR in python 3 for CMIP6.

A summer student seems like a good idea.

[markcollier]

I could also investigate whether one of the Australian groups might be able to put some $ up to pay for some of this work to be undertaken. It could be a scientist from Australia/U.S. or elsewhere with the potential to visit PCMDI and elsewhere to facilitate a more featured product. David Karoly is now based in Aspendale and sits on a prominent position that might be able to investigate funding. I think CMIP7 is so far off (and python2 end-of-life is 2020, I assume this means the end of 2020 rather than beginning? Which would mean 2.75 years left) although the development can start whenever there is funding to support it. I assume it would be necessary to hire a computer scientist with some experience (and enthusiasm) working on earth science problems?

[taylor13]

thanks for this suggestion. We'll discuss and be in touch. Good to know that Dave K. may be able to vouch for the importance of this.