Opening .dods with OPeNDAP clients

[xaviernogueira]

Hi again @abkfenris.

So I was testing OPeNDAP clients with my Catalog-To-Xpublish repo which uses xpublish-opendap. There seems to be some issues with certain clients/techniques for opening the data.

For reference the testing URL I am using looks like this: "http://127.0.0.1:8000/conus404-daily/datasets/zarr-s3/opendap?ACETLSM[100:150][100:300][100:300]" where "zarr-s3" is the dataset name.

What works:

• xarray.open_dataset(DODS_URL, engine='netcdf4'): This works as expected.
• netCDF4.Dataset(DODS_URL): This works as expected.

What doesn't work:

• xarray.open_dataset(DODS_URL, engine='h5netcdf'): This fails with an "OSError: [Errno 404] Not Found" error. Looking at the logging it appears that the xarray backend splits to root (i.e., "http://127.0.0.1:8000/") from the rest of the URL adding an extra "/", leading to a 404.
• xarray.open_dataset(DODS_URL, engine='pydap'): This works as expected at first, but when you try and access the underlying values (i.e., dataset[VARIABLE].values) further requests are made to the server that lead to a "IncompleteRead" error. I suspect this could be due to the altered URL that is pinged by pydap of the form "http://.../opendap.dods?ACETLSM.ACETLSM[100:1:50][100:1:200][100:1:200]".

How to address this may be out of the scope of this plugin, but I wanted to mention this for potential inclusion in documentation before others run into similar issues.

[abkfenris]

Bummer. I'm guessing we have probably a slight incompatibility with the underlying engines. I feel like this might take some reading of the OPeNDAP spec (dap4), and maybe comparing it against other implementations.

We may need to use something like pytest-xprocess so that we can test the various engines directly rather than only making mock requests.

[xaviernogueira]

By intercepting the Xarray backend logging I can see that it is not an issue with the OPeNDAP spec but rather Xarray lazy loading and how it pings my locally API iteratively to load in chunks. The issue seems to be the URL it expects to ping, not the structure of the DODs binary (at least from what I see).

Regardless I agree adding test cases for the clients/engines one might use makes sense.

[abkfenris]

Ok, I've got a PR with tests that replicates what you've found: #20