add new backend api documentation #4810

aurghs · 2021-01-14T15:41:50Z

add backend documentation
rename store_spec in filename_or_objin backend entrypoint method guess_can_open
Related Update Documentation for backend Implementation #4803

doc/internals.rst

dcherian

I have mostly minor suggestions.

One high-level comment is that it would be nice to have an example implementation with pseudo-code. That would make it easier to understand.

class YourBackendArray(BackendEntrypoint):
    def open_dataset(...):
        ds = ...
 	    # example built-in decoder use
        return ds

    open_dataset_parameters = ...

    def guess_can_open(filename_obj):
        # do checks
        return True

dcherian · 2021-02-03T18:16:17Z

doc/internals.rst

+
+- Create a class that inherits from Xarray py:class:`~xarray.backend.commonBackendEntrypoint`
+- Implement the method ``open_dataset`` that returns an instance of :py:class:`~xarray.Dataset`
+- Declare such a class as an external plugin in your setup.py.


Can we link to the entrypoints documentation here.

Suggested change

- Declare such a class as an external plugin in your setup.py.

- Declare such a class as an external plugin in your ``setup.py``.

EDIT: I see it's linked below. Won't hurt to add a link here also.

doc/internals.rst

mathause

I also think that an example would go a long way. It could also be a good idea to point to the simplest of the implemented backends.

I tried to look at some of the backends that are now available, e.g. pydap and one thing that's missing from the description here is the DataStore.

Not for here but I did also not find much on what a LazilyOuterIndexedArray is etc...

doc/internals.rst

keewis

I agree with the need for examples, but overall I think this is pretty well structured. In case it is of use, I find this pretty helpful: https://documentation.divio.com/how-to-guides/

I think it would be useful to have a comparison between the old and new API, but I'm not sure if the old API ever was part of the public API (if not this is not necessary).

BackendArray, CachingFileManager, LazilyOuterIndexedArray, and LazilyVectorizedIndexedArray are not yet public API (I think?), so if we want to change that I think we should make sure their purpose is properly documented (it looks like that's what you were going to do, though).

This makes internals a fairly long document, so we should think about splitting this into different pages (not in this PR). I think @andersy005 started something like that in #4835.

keewis · 2021-02-03T22:22:41Z

doc/internals.rst

+to integrate any code in Xarray; all you need to do is approaching the
+following steps:
+
+- Create a class that inherits from Xarray py:class:`~xarray.backend.commonBackendEntrypoint`


Suggested change

- Create a class that inherits from Xarray py:class:`~xarray.backend.commonBackendEntrypoint`

- Create a class that inherits from :py:class:`~xarray.backends.common.BackendEntrypoint`

also, I think I read somewhere that the official spelling of the package is xarray (I can't remember where, though). If that is still valid, I think we should make sure the documentation follows that.

doc/internals.rst

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Co-authored-by: keewis <keewis@users.noreply.github.com>

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Co-authored-by: keewis <keewis@users.noreply.github.com>

…cumentation-draft � Conflicts: � doc/api-hidden.rst � doc/internals.rst

jhamman · 2021-03-03T17:53:43Z

@jp-dark - Given your recent experience developing a backend using this new API, it would be great if you could provide some comments on this PR.

kmuehlbauer · 2021-03-03T18:58:03Z

@jhamman As I'm currently implementing xarray backends for reading weather radar data, I've already closely followed the adventures of @aurghs and @alexamici.

To the current state of the backend documentation I can say that it is very well written and understandable (also for non native speakers) and I could easily follow the train of thought.

From my perspective: I would not try to show extensive examples in the beginning, but would extend/rearrange documentation after first users issued problems/asked questions. So that only things get special documentation which are not immediately clear from users perspective.

This is a really remarkable extension which will boost usage of xarray. Thanks!

jp-dark · 2021-03-03T19:37:01Z

doc/internals.rst

+If you support more complex indexing as `explicit indexing` or
+`numpy indexing`, you can have a look to the implemetation of Zarr backend and Scipy backend,
+currently available in :py:mod:`~xarray.backends` module.
+


It may be useful to add a discussion here on how a raw key is transformed to an ExplicitIndexer when querying an xarray DataArray with a third-party backend. This would be especially useful for picking good test cases for querying DataArrays and debugging problems in edge-cases.

I can try to add something about it.

I had a look at what to add here. I started writing some more details, but I think It would become a little bit too long.
Today we should merge. So I suggest to merge it as it is, and I'll add later a dedicated section.

shoyer · 2021-01-14T16:18:19Z

doc/internals.rst

+
+- Implement a function that returns an instance :py:class:``Dataset``
+
+- Create a `BackendEntrypoint`` instance with your function as input.


Suggested change

- Create a `BackendEntrypoint`` instance with your function as input.

- Create a ``BackendEntrypoint`` instance with your function as input.

shoyer · 2021-01-14T16:27:11Z

doc/internals.rst

+
+**Output**
+
+```BackendEntrypoint`.open_dataset`` output shall be an instance of Xarray :py:class:`Dataset`


Suggested change

```BackendEntrypoint`.open_dataset`` output shall be an instance of Xarray :py:class:`Dataset`

``BackendEntrypoint.open_dataset`` output shall be an instance of Xarray :py:class:`Dataset`

shoyer · 2021-01-14T16:28:36Z

doc/internals.rst

+While ``open_dataset`` is mandatory, ``open_dataset_parameters`` and ``guess_can_open`` are optional.
+
+
+BackendEntrypoint.open_dataset


Consider documenting the interface as a class? That might make it easier to document (e.g., in the form of docstrings).

doc/internals.rst

Co-authored-by: Stephan Hoyer <shoyer@google.com>

doc/internals.rst

Co-authored-by: Julia Dark <github@email.jpdark.com>

…indow * upstream/master: add polyval to polyfit see also (pydata#5020) mention map_blocks in the docstring of apply_ufunc (pydata#5011) Switch backend API to v2 (pydata#4989) WIP: add new backend api documentation (pydata#4810) pin netCDF4=1.5.3 in min-all-deps (pydata#4982) fix matplotlib errors for single level discrete colormaps (pydata#4256) Adapt exception handling in CFTimeIndex.__sub__ and __rsub__ (pydata#5006) Update options.py (pydata#5000) Adjust tests to use updated pandas syntax for offsets (pydata#4537) add a combine_attrs parameter to Dataset.merge (pydata#4895) Support for dask.graph_manipulation (pydata#4965) raise on passing axis to Dataset.reduce methods (pydata#4940)

…-tasks * upstream/master: Fix regression in decoding large standard calendar times (pydata#5050) Fix sticky sidebar responsiveness on small screens (pydata#5039) Flexible indexes refactoring notes (pydata#4979) add a install xarray step to the upstream-dev CI (pydata#5044) Adds Dataset.query() method, analogous to pandas DataFrame.query() (pydata#4984) run tests on python 3.9 (pydata#5040) Add date attribute to datetime accessor (pydata#4994) 📚 New theme & rearrangement of the docs (pydata#4835) upgrade ci-trigger to the most recent version (pydata#5037) GH5005 fix documentation on open_rasterio (pydata#5021) GHA for automatically canceling previous CI runs (pydata#5025) Implement GroupBy.__getitem__ (pydata#3691) conventions: decode unsigned integers to signed if _Unsigned=false (pydata#4966) Added support for numpy.bool_ (pydata#4986) Add additional str accessor methods for DataArray (pydata#4622) add polyval to polyfit see also (pydata#5020) mention map_blocks in the docstring of apply_ufunc (pydata#5011) Switch backend API to v2 (pydata#4989) WIP: add new backend api documentation (pydata#4810) pin netCDF4=1.5.3 in min-all-deps (pydata#4982)

aurghs added 3 commits January 14, 2021 15:38

documentation first draft

7e33010

documentation update

75544c9

documentation update

c6f64cc

Illviljan reviewed Jan 16, 2021

View reviewed changes

doc/internals.rst Outdated Show resolved Hide resolved

update documentation

11fc283

jhamman changed the title ~~WIP: add e new backend documentation~~ WIP: add new backend api documentation Jan 28, 2021

aurghs added 4 commits February 2, 2021 02:17

update backend documentation

1b8ac13

incompletre draft: update Backend Documentation

e471f3a

fix

65c339d

fix syle

113197d

jhamman added topic-backends grant-czi topic-documentation labels Feb 3, 2021

dcherian reviewed Feb 3, 2021

View reviewed changes

mathause reviewed Feb 3, 2021

View reviewed changes

keewis reviewed Feb 3, 2021

View reviewed changes

aurghs and others added 14 commits February 4, 2021 04:27

Update doc/internals.rst

f9ed1d4

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Update doc/internals.rst

6b2ecd5

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Update doc/internals.rst

1958163

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Update doc/internals.rst

bba32e4

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Update doc/internals.rst

cb8d716

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Update doc/internals.rst

2adf355

Co-authored-by: keewis <keewis@users.noreply.github.com>

Update doc/internals.rst

7ec3238

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Update doc/internals.rst

fb03493

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

Update doc/internals.rst

22794d2

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Update doc/internals.rst

67d2c1f

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Update doc/internals.rst

f58f16b

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Update doc/internals.rst

6a07a7c

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Update doc/internals.rst

1285874

Co-authored-by: Mathias Hauser <mathause@users.noreply.github.com>

Update doc/internals.rst

112837d

Co-authored-by: keewis <keewis@users.noreply.github.com>

aurghs added 8 commits February 25, 2021 14:10

update documentation

e838d40

update documentation

8633e08

Merge branch 'documentation-draft' of github.com:bopen/xarray into do…

3281345

…cumentation-draft � Conflicts: � doc/api-hidden.rst � doc/internals.rst

Merge remote-tracking branch 'origin/master' into documentation-draft

992d47d

isort

e3eb56d

rename store_spec in filename_or_obj in guess_can_open

a456478

small update in backend documentation

abf60e0

small update in backend documentation

e72ce9b

jhamman mentioned this pull request Mar 3, 2021

A read-only TileDB backend #4987

Closed

jp-dark reviewed Mar 3, 2021

View reviewed changes

shoyer reviewed Mar 3, 2021

View reviewed changes

aurghs and others added 4 commits March 3, 2021 21:56

Update doc/internals.rst

7108f80

Co-authored-by: Stephan Hoyer <shoyer@google.com>

Update doc/internals.rst

e8499cd

Co-authored-by: Stephan Hoyer <shoyer@google.com>

fix backend documentation

0955c16

replace LazilyOuterIndexedArray with LazilyIndexedArray

54c202c

shoyer mentioned this pull request Mar 4, 2021

Flexible indexes refactoring notes #4979

Merged

jp-dark reviewed Mar 5, 2021

View reviewed changes

doc/internals.rst Show resolved Hide resolved

doc/internals.rst Outdated Show resolved Hide resolved

aurghs and others added 3 commits March 5, 2021 22:23

Update doc/internals.rst

3cc18d5

Co-authored-by: Julia Dark <github@email.jpdark.com>

Update doc/internals.rst

9faf5e6

Co-authored-by: Julia Dark <github@email.jpdark.com>

Fix broken doc merge

06371df

alexamici merged commit d2582c2 into pydata:master Mar 8, 2021

alexamici mentioned this pull request Mar 8, 2021

Update Documentation for backend Implementation #4803

Closed

This was referenced Mar 10, 2021

Update GINI for new XArray backends API Unidata/MetPy#1763

Closed

Update CDMRemote for new XArray backends API Unidata/siphon#341

Open

aurghs changed the title ~~WIP: add new backend api documentation~~ add new backend api documentation Mar 25, 2021

alexamici mentioned this pull request Apr 5, 2021

Rename of LazilyOuterIndexedArray to LazilyIndexedArray breaks rioxarray #5111

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add new backend api documentation #4810

add new backend api documentation #4810

aurghs commented Jan 14, 2021 •

edited

Loading

dcherian left a comment •

edited by keewis

Loading

dcherian Feb 3, 2021

mathause left a comment

keewis left a comment •

edited

Loading

keewis Feb 3, 2021 •

edited

Loading

jhamman commented Mar 3, 2021

kmuehlbauer commented Mar 3, 2021

jp-dark Mar 3, 2021

aurghs Mar 4, 2021

aurghs Mar 8, 2021

shoyer Jan 14, 2021

shoyer Jan 14, 2021

shoyer Jan 14, 2021

	- Declare such a class as an external plugin in your setup.py.
	- Declare such a class as an external plugin in your ``setup.py``.

	- Create a class that inherits from Xarray py:class:`~xarray.backend.commonBackendEntrypoint`
	- Create a class that inherits from :py:class:`~xarray.backends.common.BackendEntrypoint`


		- Implement a function that returns an instance :py:class:``Dataset``

		- Create a `BackendEntrypoint`` instance with your function as input.

	- Create a `BackendEntrypoint`` instance with your function as input.
	- Create a ``BackendEntrypoint`` instance with your function as input.


		Output

		```BackendEntrypoint`.open_dataset`` output shall be an instance of Xarray :py:class:`Dataset`

	```BackendEntrypoint`.open_dataset`` output shall be an instance of Xarray :py:class:`Dataset`
	``BackendEntrypoint.open_dataset`` output shall be an instance of Xarray :py:class:`Dataset`

		While ``open_dataset`` is mandatory, ``open_dataset_parameters`` and ``guess_can_open`` are optional.


		BackendEntrypoint.open_dataset

add new backend api documentation #4810

add new backend api documentation #4810

Conversation

aurghs commented Jan 14, 2021 • edited Loading

dcherian left a comment • edited by keewis Loading

Choose a reason for hiding this comment

dcherian Feb 3, 2021

Choose a reason for hiding this comment

mathause left a comment

Choose a reason for hiding this comment

keewis left a comment • edited Loading

Choose a reason for hiding this comment

keewis Feb 3, 2021 • edited Loading

Choose a reason for hiding this comment

jhamman commented Mar 3, 2021

kmuehlbauer commented Mar 3, 2021

jp-dark Mar 3, 2021

Choose a reason for hiding this comment

aurghs Mar 4, 2021

Choose a reason for hiding this comment

aurghs Mar 8, 2021

Choose a reason for hiding this comment

shoyer Jan 14, 2021

Choose a reason for hiding this comment

shoyer Jan 14, 2021

Choose a reason for hiding this comment

shoyer Jan 14, 2021

Choose a reason for hiding this comment

aurghs commented Jan 14, 2021 •

edited

Loading

dcherian left a comment •

edited by keewis

Loading

keewis left a comment •

edited

Loading

keewis Feb 3, 2021 •

edited

Loading