sparse=True option for from_dataframe and from_series

[shoyer]

Fixes #3206

Example usage:

In [3]: import pandas as pd
   ...: import numpy as np
   ...: import xarray
   ...: df = pd.DataFrame({
   ...:     'w': range(10),
   ...:     'x': list('abcdefghij'),
   ...:     'y': np.arange(0, 100, 10),
   ...:     'z': np.ones(10),
   ...: }).set_index(['w', 'x', 'y'])
   ...:

In [4]: ds = xarray.Dataset.from_dataframe(df, sparse=True)

In [5]: ds.z.data
Out[5]: <COO: shape=(10, 10, 10), dtype=float64, nnz=10, fill_value=nan>

• Closes DataFrame with MultiIndex -> xarray with sparse array #3206, Closes From pandas to xarray without blowing up memory #2139
• Tests added
• Passes black . && mypy . && flake8
• Fully documented, including whats-new.rst for all changes and api.rst for new API

[max-sixty]

Amazing! This is going to allow way more usage of xarray for sparse representations of data currently confined to pandas

[shoyer]

It would be great if someone could review this. Maybe @crusaderky ?

[shoyer]

thanks for the review!

[shoyer]

@crusaderky any other comments?

[crusaderky]

I think I can get rid of the cast by changing Dataset.__getitem__. Let's merge this one first though.

[crusaderky]

... or not. Unlike @functools.single_dispatch, @typing.overload does not allow for a more specific signature followed by a more generic one :(

    @overload
    def __getitem__(self, key: Mapping) -> "Dataset":
        ...

    @overload
    def __getitem__(self, key: Hashable) -> "DataArray":
        ...

    @overload
    def __getitem__(self, key: Any) -> "Dataset":
        ...

    def __getitem__(self, key):

mypy complains:

xarray/core/dataset.py:1218: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
xarray/core/dataset.py:1222: error: Overloaded function signatures 2 and 3 overlap with incompatible return types

[shoyer]

Yeah, I tried something very similar. This is why I wrote the "TODO" note above mentioning python/mypy#7328.

[crusaderky]

Looks good to me

[max-sixty]

Great!