DEPR: SparseDataFrame and SparseSeries subclasses

[jreback]

This is slightly useful in a single dtyped case, but enough of a headache here that we should simply remove it. Ultimately we should even remove SparseSeries and just rely on the SparseArray abstraction and extension types (#19174) .

[jreback]

cc @jorisvandenbossche @TomAugspurger @wesm
cc @hexgnu
cc @Licht-T
cc @kernc

[datapythonista]

In case my use case it's useful in the discussion. I mostly (probably only) use SparseDataFrame because of pandas.get_dummies(). And after that, I'm mainly interested in having a scipy.sparse structure (for example to use it in scikit-learn). I'd bet this is the case for many pandas users.

Personally I'm +1 on getting rid of it. Then, it could be useful to have a get_dummies sparse version that returns a scipy.sparse. Not sure if in pandas or as a separate project.

[hexgnu]

I think a lot of the issue with SparseDataFrame is honestly a lack of test coverage as well as a lot of bleedover into DataFrame. For instance a lot of code assumes that something is dense and that it has length > 0. This isn't the real case, and I feel that with a flag day or two of squashing sparse bugs and writing more tests it wouldn't be so awful.

But that being said I also don't want to spend a bunch of time fixing something nobody really uses.

Also this is not obvious to me on first look:

• DataFrame can have both Series and SparseSeries in it
• SparseDataFrame can only really have SparseSeries inside of it

Personally I would think that SparseDataFrame is the only time you can have SparseSeries and DataFrame is always dense.

[hexgnu]

Also I wanted to point out that Sparse bugs aren't the biggest label category in this repo

It looks to me like reshaping, indexing, and dtypes are the big hotspots. Getting rid of sparse I don't think would bring those bug counts down.

Also I think with some TLC SparseDataFrames can be a really nice little abstraction for saving on memory consumption in certain cases.

[jreback]

@hexgnu its not about the issues, rather about having another structure to use, which leads to confusion and code complexity. Yes a pure sparse frame does lead to efficiencies, but you (w/o a lot of acrobatics) lose the heterogenity which makes a DataFrame itself so useful. I think a SparseDataFrame should exist as a separate project :> (and the primitives, IOW SparseArray/Series can still live in pandas proper).

[kernc]

As long as high-level NDFrame APIs defer actions to block primitives as far as possible, having a regular DataFrame wrapping them should work fine.

[TomAugspurger]

What are the arguments for keeping SparseDataFrame around? I'm not familiar enough to say for sure whether this is possible, but ideally we would

• Clearly document that DataFrame can hold sparse data
• Move all current sparse-specific methods to a .sparse accessor (density, to_coo). Some of these methods would error if the DataFrame isn't entirely sparse.
• Figure out how to handle default_fill_value
• Deprceate SparseDataFrame in favor of a DataFrame holding sparse arrays

What can a SparseDataFrame do that a DataFrame[SparseArray] can't?

[xpe]

I hope we'd all agree that support for sparsity is very useful. I don't know the internals of Pandas, so I won't weigh in on the particular question of "What can a SparseDataFrame do that a DataFrame[SparseArray] cannot?"

As a user, I'll share my use case and an appreciation for clear documentation.

My use case involves load_svmlight_file from scikit-learn:

from sklearn.datasets import load_svmlight_file
features, labels = load_svmlight_file(data_filename)  # sparse data
feature_labels = ['a', 'b', 'c']  # etc
df = pd.SparseDataFrame(data=features, columns=feature_labels)
df.plot(y='c')  # fails
df.to_dense().plot(y='c')  # succeeds

I'd be open to using other ways to load the sparse LibSVM data, but I don't currently know them. Any recommendations?

Having .plot() work in this case with SparseDataFrame would be nice. :)

[jorisvandenbossche]

I am also not familiar enough with the sparse dataframe to answer the question ""What can a SparseDataFrame do that a DataFrame[SparseArray] cannot?".
But, when we would take this route and do:

Deprceate SparseDataFrame in favor of a DataFrame holding sparse arrays

My main worry is: how much will this complicate our current DataFrame implementation? Will this mean adding special-casing for sparse dataframes in multiple places? ... If that is the case, I am not sure I am in favor of this.
Eg in pandas/core/sparse/frame.py, there is still a lot of custom code overriding parent frame methods (next to the sparse-specific methods like to_coo or to_dense). I don't know what all this code does, but it might serve a purpose.

Of course it might be that with the new ExtensionArray interface we could make this much cleaner, and wouldn't need such amount of special casing. But I can't assess if that will be the case or not.

Figure out how to handle default_fill_value

This is a bit similar to the the extra metadata attributes I also have in geopandas. It would be good if we can figure out something.
I suppose having it as an attribute on a "sparse dtype" object might be one possible way.

[jreback]

The main difference is this. A SDF of a single dtype can be very efficiently represented because it has a default_fill_value for the entire frame. Think of csr_matrix here. A DataFrame can already hold SparseSeries. So in theory you can simply replace every case of a SDF with a DataFrame. If you have lots of columns (like a lot), then this would be pretty inefficient.

My argument for removing this is simple. This causes an enormous amount of complexity in the codebase, and we have many non-implemented features (mainly indexing). So its pretty half-baked. Simply deprecated / removing in pandas would allow an external library to implement this properly (to allow for the efficiency argument above), while not sacrificing the occasionaly use of a SparseSeries (or even a small number of SparseSeries), e.g. what get_dummies returns (or could return); this is in fact the main case.

We would also leave this, but move the implementation out of main pandas (to an external library) this is separately maintained.

[jorisvandenbossche]

The main difference is this. A SDF of a single dtype can be very efficiently represented because it has a default_fill_value for the entire frame. Think of csr_matrix here. A DataFrame can already hold SparseSeries. So in theory you can simply replace every case of a SDF with a DataFrame. If you have lots of columns (like a lot), then this would be pretty inefficient.

As said above, not familiar with sparse code, but it seems the above is not true and that even a SparseDataFrame with a single dtype already stores the data column-by-column?

In [61]: pd.SparseDataFrame([[0,1],[1,0]])._data
Out[61]: 
BlockManager
Items: RangeIndex(start=0, stop=2, step=1)
Axis 1: RangeIndex(start=0, stop=2, step=1)
SparseBlock: slice(0, 1, 1), 1 x 2, dtype: int64
SparseBlock: slice(1, 2, 1), 1 x 2, dtype: int64

[jreback]

construct by using .to_sparse() from a DF

the method above is not consolidated by default

[jorisvandenbossche]

There is no difference:

In [63]: pd.DataFrame([[0,1],[1,0]]).to_sparse()._data
Out[63]: 
BlockManager
Items: RangeIndex(start=0, stop=2, step=1)
Axis 1: RangeIndex(start=0, stop=2, step=1)
SparseBlock: slice(0, 1, 1), 1 x 2, dtype: int64
SparseBlock: slice(1, 2, 1), 1 x 2, dtype: int64

And SparseArray is also limited to 1D I think (it gives an error if you try to pass the above 2D array to SparseArray).

[TomAugspurger]

And SparseArray is also limited to 1D I think

Ahh, that answers it then. I was thinking SparseArray was 2D.

[TomAugspurger]

Opened #23148 for creating a sparse accessor.

Have we identified anything SparseDataFrame can do that a regular DataFrame can't?

@xpe I assume that scikit-learn load_svmlight returns a spicy.sparse matrix? In my ideal world you would be able to load that with pd.DataFrame.sparse.from_coo.

Anything else?

[TomAugspurger]

I've updated the title to consider deprecating both SparseDataFrame and SparseSeries now that #22325 is in master (FYI, it'd be helpful to have people testing that on real workloads)

FWIW, when I was doing that refactor, I discovered several subtle difference between methods implemented on both Series and SparseSeries. The Series implementation was updated but the SparseSeries one lagged behind.

[jreback]

if we can't do this for 0.24, 0.25 is ok too.

[frndlytm]

Multi-Indexing SparseDataFrames is a nice feature you don't get from scipy.sparse arrays.

[TomAugspurger]

@frndlytm you'll still be able to do that with a DataFrame with sparse values

In [11]: df = pd.DataFrame({"A": [1, 0], "B": [0, 1]}, dtype="Sparse[int]", index=pd.MultiIndex.from_tuples([('A', 'a'), ('A', 'b')]))

In [12]: df.dtypes
Out[12]:
A    Sparse[int64, 0]
B    Sparse[int64, 0]
dtype: object

In [13]: df
Out[13]:
     A  B
A a  1  0
  b  0  1

or do I misunderstand?