Loading data with pydap engine fails due to outdated/missing dependency

[observingClouds]

What happened?

Hi,

I was trying to load a dataset with the pydap engine, but unfortunately it failed due to a mis-match of arguments in xarray's pydap backend and the used version of pydap (3.2.2).

What did you expect to happen?

Retrieving a nice dataset 😄

Minimal Complete Verifiable Example

import xarray as xr
xr.open_dataset('http://test.opendap.org/dap/data/nc/coads_climatology.nc', engine='pydap')

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

TypeError                                 Traceback (most recent call last)
<ipython-input-4-36e60ce43c69> in <module>
----> 1 xr.open_dataset('http://test.opendap.org/dap/data/nc/coads_climatology.nc', engine='pydap')

~/mambaforge/envs/how_to_eurec4a/lib/python3.8/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, *args, **kwargs)
    493 
    494     overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 495     backend_ds = backend.open_dataset(
    496         filename_or_obj,
    497         drop_variables=drop_variables,

~/mambaforge/envs/how_to_eurec4a/lib/python3.8/site-packages/xarray/backends/pydap_.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, application, session, output_grid, timeout, verify, user_charset)
    163     ):
    164 
--> 165         store = PydapDataStore.open(
    166             url=filename_or_obj,
    167             application=application,

~/mambaforge/envs/how_to_eurec4a/lib/python3.8/site-packages/xarray/backends/pydap_.py in open(cls, url, application, session, output_grid, timeout, verify, user_charset)
    112             user_charset = "ascii"
    113 
--> 114         ds = pydap.client.open_url(
    115             url=url,
    116             application=application,

TypeError: open_url() got an unexpected keyword argument 'verify'

Anything else we need to know?

The root-cause of this issue seems to be a missing dependency.

With dfaedb2 the verify-argument has been added to xarray/backends/pydap_.py, which is supported by pydap since late 2018 (pydap/pydap#112). However, only recently a new release of pydap has been published and incorporates these changes now.

Version 3.3.0, released on 1. Feb. 2022 (includes verify)
Version 3.2.2 released on 25. May 2017 (has no verify)

Unfortunately, version 3.3.0 is not yet available on pypi and only on conda-forge.

I couldn't find any pins or limitations on versions in e.g. requirements.txt or setup.cfg for non-core dependencies. Should this dependency be introduced somewhere? At least https://github.com/pydata/xarray/blob/e02b1c3f6d18c7afcdf4f78cf3463652b4cc96c9/ci/requirements/min-all-deps.yml needs to be updated I guess.

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:06:49) [Clang 12.0.1 ] python-bits: 64 OS: Darwin OS-release: 21.4.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: None LOCALE: (None, 'UTF-8') libhdf5: 1.12.1 libnetcdf: 4.8.1

xarray: 2022.3.0
pandas: 1.4.2
numpy: 1.22.3
scipy: 1.8.0
netCDF4: 1.5.8
pydap: installed
h5netcdf: None
h5py: None
Nio: None
zarr: 2.11.1
cftime: 1.6.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.04.1
distributed: 2022.4.1
matplotlib: 3.5.1
cartopy: 0.20.2
seaborn: None
numbagg: None
fsspec: 2022.5.0
cupy: None
pint: None
sparse: None
setuptools: 61.1.1
pip: 22.0.4
conda: None
pytest: 7.1.2
IPython: 7.29.0
sphinx: 4.5.0

[dcherian]

Thanks @observingClouds. A PR adding a pydap version check to pydap_.py would be very welcome!