pydap support is broken for recent releases of pydap (and on Python 3)

[taeold]

Hello!

I was hoping to spend some of my end-of-the-year free time picking up low-hanging fruits in xarray, and while trying to clean up tests failures as mentioned in #1109, I've come across an issue using engine='pydap'.

Specifically, I can't get engine='pydap' to work at all:

>>> import xarray

;; opening with default backend works ('netcdf4' I think?)
>>> xarray.open_dataset("http://test.opendap.org/opendap/hyrax/data/nc/123bears.nc")
<xarray.Dataset>
Dimensions:...
Coordinates:...
Data variables:...
Attributes:...

;; but using 'pydap'...
>>> xarray.open_dataset("http://test.opendap.org/opendap/hyrax/data/nc/123bears.nc", engine='pydap')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "xarray/backends/api.py", line 306, in open_dataset
    return maybe_decode_store(store, lock)
  File "xarray/backends/api.py", line 227, in maybe_decode_store
    drop_variables=drop_variables)
  File "xarray/conventions.py", line 941, in decode_cf
    vars, attrs = obj.load()
  File "xarray/backends/common.py", line 100, in load
    for k, v in iteritems(self.get_variables()))
  File "xarray/backends/pydap_.py", line 72, in get_variables
    for k, v in self.ds.iteritems())
  File "/Users/daniel.lee/anaconda/envs/test_env/lib/python2.7/site-packages/pydap/model.py", line 301, in __getattr__
    return DapType.__getattr__(self, attr)
  File "/Users/daniel.lee/anaconda/envs/test_env/lib/python2.7/site-packages/pydap/model.py", line 188, in __getattr__
    % (self.__class__, attr))
AttributeError: '<class 'pydap.model.DatasetType'>' object has no attribute 'iteritems'

latest xarray + latest pydap (3.2.0)

From my best attempt at looking over the xarray code and playing with pydap.client, I'm beginning to suspect that engine='pydap' no longer working in xarray (for instance - as noted in the exception msg above - you cannot call iteritems() on pydap.model.DatasetType but xarray insists on doing so).

Is what I'm seeing and suggesting correct? If so, is xarray still committed to supporting pydap as an engine (and be a worthwhile things for a new contributors to work on)?

[jhamman]

@taeold - can you see if you can open the test dataset via pydap directly?

[shoyer]

This looks like a Python 2 vs 3 thing. (iteritems() went away in Python 3.)

Until it's latest release (within the last month), pydap didn't support Python 3. Now it does (yay!) but we haven't updated xarray for the latest pydap. Switching iteritems() to items() should be a pretty easy fix.

Contributions here would definitely be welcome! We even have some pydap specific tests, but they aren't run by default in continuous integration because they require accessing an external server (see PydapTest in test_backends.py) which has been unreliable in the past.

Somewhat related is pydap/pydap#33, which will allow us to run a pydap server as part of our test suite.

[taeold]

Thanks for your replies!

@jhamman Yes - pydap can open the file without issue:

In [7]: pydap.client.open_url("http://test.opendap.org/opendap/hyrax/data/nc/123bears.nc")
Out[7]: <DatasetType with children 'bears', 'order', 'shot', 'aloan', 'cross', 'i', 'j', 'l'>

@shoyer I set up the testing environment using this (currenlty failing) recipe (python 2.7)! I think the issue might not be python2/3 compatibility, but that the datatype returned from the pydap client (DatasetType) is not quite a dictionary - it doesn't define iteritems() or items().

I'll dig a bit more deeper - maybe an upstream change in pydap is reasonable?

[shoyer]

@taeold pydap has new maintainers, I recommend raising an issue to get their perspective. For what it's worth, I think they probably should make their DatasetType inherit from collections.Mapping to ensure it satisfies the full mapping API.

[shoyer]

See #1439 for a fix