diff --git a/doc/source/user_guide/sparse.rst b/doc/source/user_guide/sparse.rst index 20962749e2040..8fed29d7a6316 100644 --- a/doc/source/user_guide/sparse.rst +++ b/doc/source/user_guide/sparse.rst @@ -6,27 +6,27 @@ Sparse data structures ********************** -We have implemented "sparse" versions of ``Series`` and ``DataFrame``. These are not sparse -in the typical "mostly 0". Rather, you can view these objects as being "compressed" -where any data matching a specific value (``NaN`` / missing value, though any value -can be chosen) is omitted. A special ``SparseIndex`` object tracks where data has been -"sparsified". This will make much more sense with an example. All of the standard pandas -data structures have a ``to_sparse`` method: +.. note:: -.. ipython:: python - - ts = pd.Series(np.random.randn(10)) - ts[2:-2] = np.nan - sts = ts.to_sparse() - sts + ``SparseSeries`` and ``SparseDataFrame`` have been deprecated. Their purpose + is served equally well by a :class:`Series` or :class:`DataFrame` with + sparse values. See :ref:`sparse.migration` for tips on migrating. -The ``to_sparse`` method takes a ``kind`` argument (for the sparse index, see -below) and a ``fill_value``. So if we had a mostly zero ``Series``, we could -convert it to sparse with ``fill_value=0``: +Pandas provides data structures for efficiently storing sparse data. +These are not necessarily sparse in the typical "mostly 0". Rather, you can view these +objects as being "compressed" where any data matching a specific value (``NaN`` / missing value, though any value +can be chosen, including 0) is omitted. The compressed values are not actually stored in the array. .. ipython:: python - ts.fillna(0).to_sparse(fill_value=0) + arr = np.random.randn(10) + arr[2:-2] = np.nan + ts = pd.Series(pd.SparseArray(arr)) + ts + +Notice the dtype, ``Sparse[float64, nan]``. The ``nan`` means that elements in the +array that are ``nan`` aren't actually stored, only the non-``nan`` elements are. +Those non-``nan`` elements have a ``float64`` dtype. The sparse objects exist for memory efficiency reasons. Suppose you had a large, mostly NA ``DataFrame``: @@ -35,21 +35,82 @@ large, mostly NA ``DataFrame``: df = pd.DataFrame(np.random.randn(10000, 4)) df.iloc[:9998] = np.nan - sdf = df.to_sparse() - sdf - sdf.density + sdf = df.astype(pd.SparseDtype("float", np.nan)) + sdf.head() + sdf.dtypes + sdf.sparse.density As you can see, the density (% of values that have not been "compressed") is extremely low. This sparse object takes up much less memory on disk (pickled) -and in the Python interpreter. Functionally, their behavior should be nearly +and in the Python interpreter. + +.. ipython:: python + + 'dense : {:0.2f} bytes'.format(df.memory_usage().sum() / 1e3) + 'sparse: {:0.2f} bytes'.format(sdf.memory_usage().sum() / 1e3) + +Functionally, their behavior should be nearly identical to their dense counterparts. -Any sparse object can be converted back to the standard dense form by calling -``to_dense``: +.. _sparse.array: + +SparseArray +----------- + +:class:`SparseArray` is a :class:`~pandas.api.extensions.ExtensionArray` +for storing an array of sparse values (see :ref:`basics.dtypes` for more +on extension arrays). It is a 1-dimensional ndarray-like object storing +only values distinct from the ``fill_value``: .. ipython:: python - sts.to_dense() + arr = np.random.randn(10) + arr[2:5] = np.nan + arr[7:8] = np.nan + sparr = pd.SparseArray(arr) + sparr + +A sparse array can be converted to a regular (dense) ndarray with :meth:`numpy.asarray` + +.. ipython:: python + + np.asarray(sparr) + + +SparseDtype +----------- + +The :attr:`SparseArray.dtype` property stores two pieces of information + +1. The dtype of the non-sparse values +2. The scalar fill value + + +.. ipython:: python + + sparr.dtype + + +A :class:`SparseDtype` may be constructed by passing each of these + +.. ipython:: python + + pd.SparseDtype(np.dtype('datetime64[ns]')) + +The default fill value for a given NumPy dtype is the "missing" value for that dtype, +though it may be overridden. + +.. ipython:: python + + pd.SparseDtype(np.dtype('datetime64[ns]'), + fill_value=pd.Timestamp('2017-01-01')) + +Finally, the string alias ``'Sparse[dtype]'`` may be used to specify a sparse dtype +in many places + +.. ipython:: python + + pd.array([1, 0, 0, 2], dtype='Sparse[int]') .. _sparse.accessor: @@ -71,130 +132,146 @@ attributes and methods that are specific to sparse data. This accessor is available only on data with ``SparseDtype``, and on the :class:`Series` class itself for creating a Series with sparse data from a scipy COO matrix with. -.. _sparse.array: -SparseArray ------------ +.. versionadded:: 0.25.0 + +A ``.sparse`` accessor has been added for :class:`DataFrame` as well. +See :ref:`api.frame.sparse` for more. -``SparseArray`` is the base layer for all of the sparse indexed data -structures. It is a 1-dimensional ndarray-like object storing only values -distinct from the ``fill_value``: +.. _sparse.calculation: + +Sparse Calculation +------------------ + +You can apply NumPy `ufuncs `_ +to ``SparseArray`` and get a ``SparseArray`` as a result. .. ipython:: python - arr = np.random.randn(10) - arr[2:5] = np.nan - arr[7:8] = np.nan - sparr = pd.SparseArray(arr) - sparr + arr = pd.SparseArray([1., np.nan, np.nan, -2., np.nan]) + np.abs(arr) + -Like the indexed objects (SparseSeries, SparseDataFrame), a ``SparseArray`` -can be converted back to a regular ndarray by calling ``to_dense``: +The *ufunc* is also applied to ``fill_value``. This is needed to get +the correct dense result. .. ipython:: python - sparr.to_dense() + arr = pd.SparseArray([1., -1, -1, -2., -1], fill_value=-1) + np.abs(arr) + np.abs(arr).to_dense() +.. _sparse.migration: -SparseIndex objects -------------------- +Migrating +--------- -Two kinds of ``SparseIndex`` are implemented, ``block`` and ``integer``. We -recommend using ``block`` as it's more memory efficient. The ``integer`` format -keeps an arrays of all of the locations where the data are not equal to the -fill value. The ``block`` format tracks only the locations and sizes of blocks -of data. +In older versions of pandas, the ``SparseSeries`` and ``SparseDataFrame`` classes (documented below) +were the preferred way to work with sparse data. With the advent of extension arrays, these subclasses +are no longer needed. Their purpose is better served by using a regular Series or DataFrame with +sparse values instead. -.. _sparse.dtype: +.. note:: -Sparse Dtypes -------------- + There's no performance or memory penalty to using a Series or DataFrame with sparse values, + rather than a SparseSeries or SparseDataFrame. -Sparse data should have the same dtype as its dense representation. Currently, -``float64``, ``int64`` and ``bool`` dtypes are supported. Depending on the original -dtype, ``fill_value`` default changes: +This section provides some guidance on migrating your code to the new style. As a reminder, +you can use the python warnings module to control warnings. But we recommend modifying +your code, rather than ignoring the warning. -* ``float64``: ``np.nan`` -* ``int64``: ``0`` -* ``bool``: ``False`` +**Construction** -.. ipython:: python +From an array-like, use the regular :class:`Series` or +:class:`DataFrame` constructors with :class:`SparseArray` values. - s = pd.Series([1, np.nan, np.nan]) - s - s.to_sparse() +.. code-block:: python - s = pd.Series([1, 0, 0]) - s - s.to_sparse() + # Previous way + >>> pd.SparseDataFrame({"A": [0, 1]}) - s = pd.Series([True, False, True]) - s - s.to_sparse() +.. ipython:: python + + # New way + pd.DataFrame({"A": pd.SparseArray([0, 1])}) + +From a SciPy sparse matrix, use :meth:`DataFrame.sparse.from_spmatrix`, -You can change the dtype using ``.astype()``, the result is also sparse. Note that -``.astype()`` also affects to the ``fill_value`` to keep its dense representation. +.. code-block:: python + # Previous way + >>> from scipy import sparse + >>> mat = sparse.eye(3) + >>> df = pd.SparseDataFrame(mat, columns=['A', 'B', 'C']) .. ipython:: python - s = pd.Series([1, 0, 0, 0, 0]) - s - ss = s.to_sparse() - ss - ss.astype(np.float64) + # New way + from scipy import sparse + mat = sparse.eye(3) + df = pd.DataFrame.sparse.from_spmatrix(mat, columns=['A', 'B', 'C']) + df.dtypes -It raises if any value cannot be coerced to specified dtype. +**Conversion** -.. code-block:: ipython +From sparse to dense, use the ``.sparse`` accessors - In [1]: ss = pd.Series([1, np.nan, np.nan]).to_sparse() - Out[1]: - 0 1.0 - 1 NaN - 2 NaN - dtype: float64 - BlockIndex - Block locations: array([0], dtype=int32) - Block lengths: array([1], dtype=int32) +.. ipython:: python - In [2]: ss.astype(np.int64) - Out[2]: - ValueError: unable to coerce current fill_value nan to int64 dtype + df.sparse.to_dense() + df.sparse.to_coo() -.. _sparse.calculation: +From dense to sparse, use :meth:`DataFrame.astype` with a :class:`SparseDtype`. -Sparse Calculation ------------------- +.. ipython:: python + + dense = pd.DataFrame({"A": [1, 0, 0, 1]}) + dtype = pd.SparseDtype(int, fill_value=0) + dense.astype(dtype) + +**Sparse Properties** -You can apply NumPy *ufuncs* to ``SparseArray`` and get a ``SparseArray`` as a result. +Sparse-specific properties, like ``density``, are available on the ``.sparse`` accessor. .. ipython:: python - arr = pd.SparseArray([1., np.nan, np.nan, -2., np.nan]) - np.abs(arr) + df.sparse.density +**General Differences** -The *ufunc* is also applied to ``fill_value``. This is needed to get -the correct dense result. +In a ``SparseDataFrame``, *all* columns were sparse. A :class:`DataFrame` can have a mixture of +sparse and dense columns. As a consequence, assigning new columns to a ``DataFrame`` with sparse +values will not automatically convert the input to be sparse. + +.. code-block:: python + + # Previous Way + >>> df = pd.SparseDataFrame({"A": [0, 1]}) + >>> df['B'] = [0, 0] # implicitly becomes Sparse + >>> df['B'].dtype + Sparse[int64, nan] + +Instead, you'll need to ensure that the values being assigned are sparse .. ipython:: python - arr = pd.SparseArray([1., -1, -1, -2., -1], fill_value=-1) - np.abs(arr) - np.abs(arr).to_dense() + df = pd.DataFrame({"A": pd.SparseArray([0, 1])}) + df['B'] = [0, 0] # remains dense + df['B'].dtype + df['B'] = pd.SparseArray([0, 0]) + df['B'].dtype + +The ``SparseDataFrame.default_kind`` and ``SparseDataFrame.default_fill_value`` attributes +have no replacement. .. _sparse.scipysparse: Interaction with scipy.sparse ----------------------------- -SparseDataFrame -~~~~~~~~~~~~~~~ - -.. versionadded:: 0.20.0 +Use :meth:`DataFrame.sparse.from_coo` to create a ``DataFrame`` with sparse values from a sparse matrix. -Pandas supports creating sparse dataframes directly from ``scipy.sparse`` matrices. +.. versionadded:: 0.25.0 .. ipython:: python @@ -206,20 +283,18 @@ Pandas supports creating sparse dataframes directly from ``scipy.sparse`` matric sp_arr = csr_matrix(arr) sp_arr - sdf = pd.SparseDataFrame(sp_arr) - sdf + sdf = pd.DataFrame.sparse.from_spmatrix(sp_arr) + sdf.head() + sdf.dtypes All sparse formats are supported, but matrices that are not in :mod:`COOrdinate ` format will be converted, copying data as needed. -To convert a ``SparseDataFrame`` back to sparse SciPy matrix in COO format, you can use the :meth:`SparseDataFrame.to_coo` method: +To convert back to sparse SciPy matrix in COO format, you can use the :meth:`DataFrame.sparse.to_coo` method: .. ipython:: python - sdf.to_coo() + sdf.sparse.to_coo() -SparseSeries -~~~~~~~~~~~~ - -A :meth:`SparseSeries.to_coo` method is implemented for transforming a ``SparseSeries`` indexed by a ``MultiIndex`` to a ``scipy.sparse.coo_matrix``. +meth:`Series.sparse.to_coo` is implemented for transforming a ``Series`` with sparse values indexed by a :class:`MultiIndex` to a :class:`scipy.sparse.coo_matrix`. The method requires a ``MultiIndex`` with two or more levels. @@ -233,19 +308,17 @@ The method requires a ``MultiIndex`` with two or more levels. (2, 1, 'b', 0), (2, 1, 'b', 1)], names=['A', 'B', 'C', 'D']) - s - # SparseSeries - ss = s.to_sparse() + ss = s.astype('Sparse') ss -In the example below, we transform the ``SparseSeries`` to a sparse representation of a 2-d array by specifying that the first and second ``MultiIndex`` levels define labels for the rows and the third and fourth levels define labels for the columns. We also specify that the column and row labels should be sorted in the final sparse representation. +In the example below, we transform the ``Series`` to a sparse representation of a 2-d array by specifying that the first and second ``MultiIndex`` levels define labels for the rows and the third and fourth levels define labels for the columns. We also specify that the column and row labels should be sorted in the final sparse representation. .. ipython:: python - A, rows, columns = ss.to_coo(row_levels=['A', 'B'], - column_levels=['C', 'D'], - sort_labels=True) + A, rows, columns = ss.sparse.to_coo(row_levels=['A', 'B'], + column_levels=['C', 'D'], + sort_labels=True) A A.todense() @@ -256,16 +329,16 @@ Specifying different row and column labels (and not sorting them) yields a diffe .. ipython:: python - A, rows, columns = ss.to_coo(row_levels=['A', 'B', 'C'], - column_levels=['D'], - sort_labels=False) + A, rows, columns = ss.sparse.to_coo(row_levels=['A', 'B', 'C'], + column_levels=['D'], + sort_labels=False) A A.todense() rows columns -A convenience method :meth:`SparseSeries.from_coo` is implemented for creating a ``SparseSeries`` from a ``scipy.sparse.coo_matrix``. +A convenience method :meth:`Series.sparse.from_coo` is implemented for creating a ``Series`` with sparse values from a ``scipy.sparse.coo_matrix``. .. ipython:: python @@ -275,12 +348,12 @@ A convenience method :meth:`SparseSeries.from_coo` is implemented for creating a A A.todense() -The default behaviour (with ``dense_index=False``) simply returns a ``SparseSeries`` containing +The default behaviour (with ``dense_index=False``) simply returns a ``Series`` containing only the non-null entries. .. ipython:: python - ss = pd.SparseSeries.from_coo(A) + ss = pd.Series.sparse.from_coo(A) ss Specifying ``dense_index=True`` will result in an index that is the Cartesian product of the @@ -289,5 +362,14 @@ row and columns coordinates of the matrix. Note that this will consume a signifi .. ipython:: python - ss_dense = pd.SparseSeries.from_coo(A, dense_index=True) + ss_dense = pd.Series.sparse.from_coo(A, dense_index=True) ss_dense + + +.. _sparse.subclasses: + +Sparse Subclasses +----------------- + +The :class:`SparseSeries` and :class:`SparseDataFrame` classes are deprecated. Visit their +API pages for usage. diff --git a/doc/source/whatsnew/v0.25.0.rst b/doc/source/whatsnew/v0.25.0.rst index 6bcb71773183e..f9cf137285cd9 100644 --- a/doc/source/whatsnew/v0.25.0.rst +++ b/doc/source/whatsnew/v0.25.0.rst @@ -299,6 +299,32 @@ Other API Changes Deprecations ~~~~~~~~~~~~ +Sparse Subclasses +^^^^^^^^^^^^^^^^^ + +The ``SparseSeries`` and ``SparseDataFrame`` subclasses are deprecated. Their functionality is better-provided +by a ``Series`` or ``DataFrame`` with sparse values. + +**Previous Way** + +.. ipython:: python + :okwarning: + + df = pd.SparseDataFrame({"A": [0, 0, 1, 2]}) + df.dtypes + +**New Way** + +.. ipython:: python + + df = pd.DataFrame({"A": pd.SparseArray([0, 0, 1, 2])}) + df.dtypes + +The memory usage of the two approaches is identical. See :ref:`sparse.migration` for more (:issue:`19239`). + +Other Deprecations +^^^^^^^^^^^^^^^^^^ + - The deprecated ``.ix[]`` indexer now raises a more visible FutureWarning instead of DeprecationWarning (:issue:`26438`). - Deprecated the ``units=M`` (months) and ``units=Y`` (year) parameters for ``units`` of :func:`pandas.to_timedelta`, :func:`pandas.Timedelta` and :func:`pandas.TimedeltaIndex` (:issue:`16344`) - The :attr:`SparseArray.values` attribute is deprecated. You can use ``np.asarray(...)`` or @@ -306,7 +332,6 @@ Deprecations - The functions :func:`pandas.to_datetime` and :func:`pandas.to_timedelta` have deprecated the ``box`` keyword. Instead, use :meth:`to_numpy` or :meth:`Timestamp.to_datetime64` or :meth:`Timedelta.to_timedelta64`. (:issue:`24416`) - The :meth:`DataFrame.compound` and :meth:`Series.compound` methods are deprecated and will be removed in a future version (:issue:`26405`). - .. _whatsnew_0250.prior_deprecations: Removal of prior version deprecations/changes diff --git a/pandas/core/arrays/sparse.py b/pandas/core/arrays/sparse.py index b0236cb393c1c..ecc06db2bd07b 100644 --- a/pandas/core/arrays/sparse.py +++ b/pandas/core/arrays/sparse.py @@ -2014,9 +2014,9 @@ def from_coo(cls, A, dense_index=False): from pandas.core.sparse.scipy_sparse import _coo_to_sparse_series from pandas import Series - result = _coo_to_sparse_series(A, dense_index=dense_index) - # SparseSeries -> Series[sparse] - result = Series(result.values, index=result.index, copy=False) + result = _coo_to_sparse_series(A, dense_index=dense_index, + sparse_series=False) + result = Series(result.array, index=result.index, copy=False) return result diff --git a/pandas/core/frame.py b/pandas/core/frame.py index 963da247fcaa5..5957b23535350 100644 --- a/pandas/core/frame.py +++ b/pandas/core/frame.py @@ -1930,13 +1930,13 @@ def to_sparse(self, fill_value=None, kind='block'): >>> type(df) - >>> sdf = df.to_sparse() - >>> sdf + >>> sdf = df.to_sparse() # doctest: +SKIP + >>> sdf # doctest: +SKIP 0 1 0 NaN NaN 1 1.0 NaN 2 NaN 1.0 - >>> type(sdf) + >>> type(sdf) # doctest: +SKIP """ from pandas.core.sparse.api import SparseDataFrame diff --git a/pandas/core/generic.py b/pandas/core/generic.py index 76c73fc40977c..87db069d94893 100644 --- a/pandas/core/generic.py +++ b/pandas/core/generic.py @@ -5589,7 +5589,7 @@ def ftypes(self): 3 float64:dense dtype: object - >>> pd.SparseDataFrame(arr).ftypes + >>> pd.SparseDataFrame(arr).ftypes # doctest: +SKIP 0 float64:sparse 1 float64:sparse 2 float64:sparse diff --git a/pandas/core/series.py b/pandas/core/series.py index 55b5bdcbf53f4..8fb6ad3e3ccc5 100644 --- a/pandas/core/series.py +++ b/pandas/core/series.py @@ -1586,7 +1586,6 @@ def to_sparse(self, kind='block', fill_value=None): SparseSeries Sparse representation of the Series. """ - # TODO: deprecate from pandas.core.sparse.series import SparseSeries values = SparseArray(self, kind=kind, fill_value=fill_value) diff --git a/pandas/core/sparse/frame.py b/pandas/core/sparse/frame.py index d21a809d7246d..fa3cd781eaf88 100644 --- a/pandas/core/sparse/frame.py +++ b/pandas/core/sparse/frame.py @@ -28,6 +28,13 @@ from pandas.core.sparse.series import SparseSeries _shared_doc_kwargs = dict(klass='SparseDataFrame') +depr_msg = """\ +SparseDataFrame is deprecated and will be removed in a future version. +Use a regular DataFrame whose columns are SparseArrays instead. + +See http://pandas.pydata.org/pandas-docs/stable/\ +user_guide/sparse.html#migrating for more. +""" class SparseDataFrame(DataFrame): @@ -35,6 +42,10 @@ class SparseDataFrame(DataFrame): DataFrame containing sparse floating point data in the form of SparseSeries objects + .. deprectaed:: 0.25.0 + + Use a DataFrame with sparse values instead. + Parameters ---------- data : same types as can be passed to DataFrame or scipy.sparse.spmatrix @@ -56,6 +67,7 @@ class SparseDataFrame(DataFrame): def __init__(self, data=None, index=None, columns=None, default_kind=None, default_fill_value=None, dtype=None, copy=False): + warnings.warn(depr_msg, FutureWarning, stacklevel=2) # pick up the defaults from the Sparse structures if isinstance(data, SparseDataFrame): if index is None: diff --git a/pandas/core/sparse/scipy_sparse.py b/pandas/core/sparse/scipy_sparse.py index 40b4452caa8dc..7630983421ff9 100644 --- a/pandas/core/sparse/scipy_sparse.py +++ b/pandas/core/sparse/scipy_sparse.py @@ -116,14 +116,32 @@ def _sparse_series_to_coo(ss, row_levels=(0, ), column_levels=(1, ), return sparse_matrix, rows, columns -def _coo_to_sparse_series(A, dense_index=False): +def _coo_to_sparse_series(A, dense_index: bool = False, + sparse_series: bool = True): """ Convert a scipy.sparse.coo_matrix to a SparseSeries. - Use the defaults given in the SparseSeries constructor. + + Parameters + ---------- + A : scipy.sparse.coo.coo_matrix + dense_index : bool, default False + sparse_series : bool, default True + + Returns + ------- + Series or SparseSeries """ + from pandas import SparseDtype + s = Series(A.data, MultiIndex.from_arrays((A.row, A.col))) s = s.sort_index() - s = s.to_sparse() # TODO: specify kind? + if sparse_series: + # TODO(SparseSeries): remove this and the sparse_series keyword. + # This is just here to avoid a DeprecationWarning when + # _coo_to_sparse_series is called via Series.sparse.from_coo + s = s.to_sparse() # TODO: specify kind? + else: + s = s.astype(SparseDtype(s.dtype)) if dense_index: # is there a better constructor method to use here? i = range(A.shape[0]) diff --git a/pandas/core/sparse/series.py b/pandas/core/sparse/series.py index eac59e2c0f5eb..e4f8579a398dd 100644 --- a/pandas/core/sparse/series.py +++ b/pandas/core/sparse/series.py @@ -32,9 +32,24 @@ optional_labels='', optional_axis='') +depr_msg = """\ +SparseSeries is deprecated and will be removed in a future version. +Use a Series with sparse values instead. + + >>> series = pd.Series(pd.SparseArray(...)) + +See http://pandas.pydata.org/pandas-docs/stable/\ +user_guide/sparse.html#migrating for more. +""" + + class SparseSeries(Series): """Data structure for labeled, sparse floating point data + .. deprectaed:: 0.25.0 + + Use a Series with sparse values instead. + Parameters ---------- data : {array-like, Series, SparseSeries, dict} @@ -60,6 +75,7 @@ class SparseSeries(Series): def __init__(self, data=None, index=None, sparse_index=None, kind='block', fill_value=None, name=None, dtype=None, copy=False, fastpath=False): + warnings.warn(depr_msg, FutureWarning, stacklevel=2) # TODO: Most of this should be refactored and shared with Series # 1. BlockManager -> array # 2. Series.index, Series.name, index, name reconciliation diff --git a/pandas/tests/arrays/sparse/test_accessor.py b/pandas/tests/arrays/sparse/test_accessor.py index 676f578dd2acc..370d222c1ab4e 100644 --- a/pandas/tests/arrays/sparse/test_accessor.py +++ b/pandas/tests/arrays/sparse/test_accessor.py @@ -101,3 +101,21 @@ def test_density(self): res = df.sparse.density expected = 0.75 assert res == expected + + @pytest.mark.parametrize("dtype", ['int64', 'float64']) + @pytest.mark.parametrize("dense_index", [True, False]) + @td.skip_if_no_scipy + def test_series_from_coo(self, dtype, dense_index): + import scipy.sparse + + A = scipy.sparse.eye(3, format='coo', dtype=dtype) + result = pd.Series.sparse.from_coo(A, dense_index=dense_index) + index = pd.MultiIndex.from_tuples([(0, 0), (1, 1), (2, 2)]) + expected = pd.Series(pd.SparseArray(np.array([1, 1, 1], dtype=dtype)), + index=index) + if dense_index: + expected = expected.reindex( + pd.MultiIndex.from_product(index.levels) + ) + + tm.assert_series_equal(result, expected) diff --git a/pandas/tests/arrays/sparse/test_arithmetics.py b/pandas/tests/arrays/sparse/test_arithmetics.py index b2f254a556603..eb3af4e6dea73 100644 --- a/pandas/tests/arrays/sparse/test_arithmetics.py +++ b/pandas/tests/arrays/sparse/test_arithmetics.py @@ -8,6 +8,7 @@ import pandas.util.testing as tm +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseArrayArithmetics: _base = np.array diff --git a/pandas/tests/arrays/sparse/test_array.py b/pandas/tests/arrays/sparse/test_array.py index aa364870c7e60..659f2b97485a9 100644 --- a/pandas/tests/arrays/sparse/test_array.py +++ b/pandas/tests/arrays/sparse/test_array.py @@ -215,6 +215,7 @@ def test_scalar_with_index_infer_dtype(self, scalar, dtype): assert exp.dtype == dtype @pytest.mark.parametrize("fill", [1, np.nan, 0]) + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_sparse_series_round_trip(self, kind, fill): # see gh-13999 arr = SparseArray([np.nan, 1, np.nan, 2, 3], @@ -231,6 +232,7 @@ def test_sparse_series_round_trip(self, kind, fill): tm.assert_sp_array_equal(arr, res) @pytest.mark.parametrize("fill", [True, False, np.nan]) + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_sparse_series_round_trip2(self, kind, fill): # see gh-13999 arr = SparseArray([True, False, True, True], dtype=np.bool, @@ -1098,6 +1100,7 @@ def test_npoints(self): assert arr.npoints == 1 +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestAccessor: @pytest.mark.parametrize('attr', [ diff --git a/pandas/tests/dtypes/test_common.py b/pandas/tests/dtypes/test_common.py index c48fae5c26301..c7a62dfe77c37 100644 --- a/pandas/tests/dtypes/test_common.py +++ b/pandas/tests/dtypes/test_common.py @@ -15,6 +15,10 @@ from pandas.core.sparse.api import SparseDtype import pandas.util.testing as tm +ignore_sparse_warning = pytest.mark.filterwarnings( + "ignore:Sparse:FutureWarning" +) + # EA & Actual Dtypes def to_ea_dtypes(dtypes): @@ -146,6 +150,7 @@ def test_is_object(): @pytest.mark.parametrize("check_scipy", [ False, pytest.param(True, marks=td.skip_if_no_scipy) ]) +@ignore_sparse_warning def test_is_sparse(check_scipy): assert com.is_sparse(pd.SparseArray([1, 2, 3])) assert com.is_sparse(pd.SparseSeries([1, 2, 3])) @@ -158,6 +163,7 @@ def test_is_sparse(check_scipy): @td.skip_if_no_scipy +@ignore_sparse_warning def test_is_scipy_sparse(): from scipy.sparse import bsr_matrix assert com.is_scipy_sparse(bsr_matrix([1, 2, 3])) @@ -529,6 +535,7 @@ def test_is_bool_dtype(): @pytest.mark.parametrize("check_scipy", [ False, pytest.param(True, marks=td.skip_if_no_scipy) ]) +@ignore_sparse_warning def test_is_extension_type(check_scipy): assert not com.is_extension_type([1, 2, 3]) assert not com.is_extension_type(np.array([1, 2, 3])) @@ -595,8 +602,6 @@ def test_is_offsetlike(): (pd.DatetimeIndex([1, 2]).dtype, np.dtype('=M8[ns]')), (' msgpack diff --git a/pandas/tests/io/test_pickle.py b/pandas/tests/io/test_pickle.py index 6acf54ab73b2d..b115a08d3b0d3 100644 --- a/pandas/tests/io/test_pickle.py +++ b/pandas/tests/io/test_pickle.py @@ -196,6 +196,7 @@ def legacy_pickle(request, datapath): # --------------------- # tests # --------------------- +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_pickles(current_pickle_data, legacy_pickle): if not is_platform_little_endian(): pytest.skip("known failure on non-little endian") @@ -206,6 +207,7 @@ def test_pickles(current_pickle_data, legacy_pickle): compare(current_pickle_data, legacy_pickle, version) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_round_trip_current(current_pickle_data): def python_pickler(obj, path): diff --git a/pandas/tests/io/test_pytables.py b/pandas/tests/io/test_pytables.py index d8f3283ae5c4c..8b5907b920cca 100644 --- a/pandas/tests/io/test_pytables.py +++ b/pandas/tests/io/test_pytables.py @@ -50,6 +50,7 @@ ignore_natural_naming_warning = pytest.mark.filterwarnings( "ignore:object name:tables.exceptions.NaturalNameWarning" ) +ignore_sparse = pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") # contextmanager to ensure the file cleanup @@ -2243,6 +2244,7 @@ def test_series(self): self._check_roundtrip(ts3, tm.assert_series_equal, check_index_type=False) + @ignore_sparse def test_sparse_series(self): s = tm.makeStringSeries() @@ -2259,6 +2261,7 @@ def test_sparse_series(self): self._check_roundtrip(ss3, tm.assert_series_equal, check_series_type=True) + @ignore_sparse def test_sparse_frame(self): s = tm.makeDataFrame() @@ -2597,6 +2600,7 @@ def test_overwrite_node(self): tm.assert_series_equal(store['a'], ts) + @ignore_sparse def test_sparse_with_compression(self): # GH 2931 @@ -3741,6 +3745,7 @@ def test_start_stop_multiple(self): expected = df.loc[[0], ['foo', 'bar']] tm.assert_frame_equal(result, expected) + @ignore_sparse def test_start_stop_fixed(self): with ensure_clean_store(self.path) as store: diff --git a/pandas/tests/reshape/test_reshape.py b/pandas/tests/reshape/test_reshape.py index 552d71ac4fc38..283814d2375b1 100644 --- a/pandas/tests/reshape/test_reshape.py +++ b/pandas/tests/reshape/test_reshape.py @@ -13,6 +13,7 @@ from pandas.util.testing import assert_frame_equal +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestGetDummies: @pytest.fixture diff --git a/pandas/tests/series/test_api.py b/pandas/tests/series/test_api.py index 316c193e96a85..9b4f1f5fd0fe5 100644 --- a/pandas/tests/series/test_api.py +++ b/pandas/tests/series/test_api.py @@ -122,6 +122,7 @@ def test_sort_index_name(self): result = self.ts.sort_index(ascending=False) assert result.name == self.ts.name + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_to_sparse_pass_name(self): result = self.ts.to_sparse() assert result.name == self.ts.name @@ -194,9 +195,12 @@ def test_constructor_dict_timedelta_index(self): ) self._assert_series_equal(result, expected) + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_from_array_deprecated(self): - with tm.assert_produces_warning(FutureWarning): + # multiple FutureWarnings, so can't assert stacklevel + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False): self.series_klass.from_array([1, 2, 3]) def test_sparse_accessor_updates_on_inplace(self): diff --git a/pandas/tests/series/test_combine_concat.py b/pandas/tests/series/test_combine_concat.py index ed5cf2d6b2c51..f11595febf6ed 100644 --- a/pandas/tests/series/test_combine_concat.py +++ b/pandas/tests/series/test_combine_concat.py @@ -211,6 +211,7 @@ def test_combine_first_dt_tz_values(self, tz_naive_fixture): exp = pd.Series(exp_vals, name='ser1') assert_series_equal(exp, result) + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_concat_empty_series_dtypes(self): # booleans diff --git a/pandas/tests/series/test_missing.py b/pandas/tests/series/test_missing.py index 11ad238eecd77..77b43c1414f77 100644 --- a/pandas/tests/series/test_missing.py +++ b/pandas/tests/series/test_missing.py @@ -780,6 +780,7 @@ def test_series_fillna_limit(self): expected[:3] = np.nan assert_series_equal(result, expected) + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_sparse_series_fillna_limit(self): index = np.arange(10) s = Series(np.random.randn(10), index=index) @@ -787,7 +788,8 @@ def test_sparse_series_fillna_limit(self): ss = s[:2].reindex(index).to_sparse() # TODO: what is this test doing? why are result an expected # the same call to fillna? - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): # TODO: release-note fillna performance warning result = ss.fillna(method='pad', limit=5) expected = ss.fillna(method='pad', limit=5) @@ -797,7 +799,8 @@ def test_sparse_series_fillna_limit(self): assert_series_equal(result, expected) ss = s[-2:].reindex(index).to_sparse() - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): result = ss.fillna(method='backfill', limit=5) expected = ss.fillna(method='backfill') expected = expected.to_dense() @@ -805,13 +808,15 @@ def test_sparse_series_fillna_limit(self): expected = expected.to_sparse() assert_series_equal(result, expected) + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_sparse_series_pad_backfill_limit(self): index = np.arange(10) s = Series(np.random.randn(10), index=index) s = s.to_sparse() result = s[:2].reindex(index, method='pad', limit=5) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): expected = s[:2].reindex(index).fillna(method='pad') expected = expected.to_dense() expected[-3:] = np.nan @@ -819,13 +824,15 @@ def test_sparse_series_pad_backfill_limit(self): assert_series_equal(result, expected) result = s[-2:].reindex(index, method='backfill', limit=5) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): expected = s[-2:].reindex(index).fillna(method='backfill') expected = expected.to_dense() expected[:3] = np.nan expected = expected.to_sparse() assert_series_equal(result, expected) + @pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_series_pad_backfill_limit(self): index = np.arange(10) s = Series(np.random.randn(10), index=index) diff --git a/pandas/tests/series/test_subclass.py b/pandas/tests/series/test_subclass.py index aef5ccf535add..563a94f4588cb 100644 --- a/pandas/tests/series/test_subclass.py +++ b/pandas/tests/series/test_subclass.py @@ -1,4 +1,5 @@ import numpy as np +import pytest import pandas as pd from pandas import SparseDtype @@ -39,6 +40,7 @@ def test_subclass_unstack(self): tm.assert_frame_equal(res, exp) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeriesSubclassing: def test_subclass_sparse_slice(self): diff --git a/pandas/tests/sparse/frame/test_analytics.py b/pandas/tests/sparse/frame/test_analytics.py index 95c1c8c453d0a..ae97682f297ad 100644 --- a/pandas/tests/sparse/frame/test_analytics.py +++ b/pandas/tests/sparse/frame/test_analytics.py @@ -5,6 +5,7 @@ from pandas.util import testing as tm +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") @pytest.mark.xfail(reason='Wrong SparseBlock initialization (GH#17386)') def test_quantile(): # GH 17386 @@ -22,6 +23,7 @@ def test_quantile(): tm.assert_sp_series_equal(result, sparse_expected) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") @pytest.mark.xfail(reason='Wrong SparseBlock initialization (GH#17386)') def test_quantile_multi(): # GH 17386 diff --git a/pandas/tests/sparse/frame/test_apply.py b/pandas/tests/sparse/frame/test_apply.py index b5ea0a5c90e1a..afb54a9fa6264 100644 --- a/pandas/tests/sparse/frame/test_apply.py +++ b/pandas/tests/sparse/frame/test_apply.py @@ -37,17 +37,21 @@ def fill_frame(frame): index=frame.index) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_apply(frame): applied = frame.apply(np.sqrt) assert isinstance(applied, SparseDataFrame) tm.assert_almost_equal(applied.values, np.sqrt(frame.values)) # agg / broadcast - with tm.assert_produces_warning(FutureWarning): + # two FutureWarnings, so we can't check stacklevel properly. + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False): broadcasted = frame.apply(np.sum, broadcast=True) assert isinstance(broadcasted, SparseDataFrame) - with tm.assert_produces_warning(FutureWarning): + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False): exp = frame.to_dense().apply(np.sum, broadcast=True) tm.assert_frame_equal(broadcasted.to_dense(), exp) @@ -56,15 +60,18 @@ def test_apply(frame): frame.to_dense().apply(nanops.nansum).to_sparse()) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_apply_fill(fill_frame): applied = fill_frame.apply(np.sqrt) assert applied['A'].fill_value == np.sqrt(2) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_apply_empty(empty): assert empty.apply(np.sqrt) is empty +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_apply_nonuq(): orig = DataFrame([[1, 2, 3], [4, 5, 6], [7, 8, 9]], index=['a', 'a', 'c']) @@ -88,12 +95,14 @@ def test_apply_nonuq(): # tm.assert_series_equal(res.to_dense(), exp) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_applymap(frame): # just test that it works result = frame.applymap(lambda x: x * 2) assert isinstance(result, SparseDataFrame) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_apply_keep_sparse_dtype(): # GH 23744 sdf = SparseDataFrame(np.array([[0, 1, 0], [0, 0, 0], [0, 0, 1]]), diff --git a/pandas/tests/sparse/frame/test_frame.py b/pandas/tests/sparse/frame/test_frame.py index 67e50a733b2c6..050526aecd2bb 100644 --- a/pandas/tests/sparse/frame/test_frame.py +++ b/pandas/tests/sparse/frame/test_frame.py @@ -19,6 +19,12 @@ from pandas.tseries.offsets import BDay +def test_deprecated(): + with tm.assert_produces_warning(FutureWarning): + pd.SparseDataFrame({"A": [1, 2]}) + + +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseDataFrame(SharedWithSparse): klass = SparseDataFrame @@ -668,7 +674,8 @@ def test_append(self, float_frame): a = float_frame.iloc[:5, :3] b = float_frame.iloc[5:] - with tm.assert_produces_warning(FutureWarning, check_stacklevel=False): + with tm.assert_produces_warning(FutureWarning, check_stacklevel=False, + raise_on_extra_warnings=False): # Stacklevel is set for pd.concat, not append appended = a.append(b) tm.assert_sp_frame_equal(appended.iloc[:, :3], float_frame.iloc[:, :3], @@ -683,12 +690,12 @@ def test_append(self, float_frame): "A": [None, None, 2, 3], "D": [None, None, 5, None], }, index=a.index | b.index, columns=['B', 'C', 'A', 'D']) - with tm.assert_produces_warning(None): + with tm.assert_produces_warning(None, raise_on_extra_warnings=False): appended = a.append(b, sort=False) tm.assert_frame_equal(appended, expected) - with tm.assert_produces_warning(None): + with tm.assert_produces_warning(None, raise_on_extra_warnings=False): appended = a.append(b, sort=True) tm.assert_sp_frame_equal(appended, expected[['A', 'B', 'C', 'D']], @@ -809,7 +816,8 @@ def test_sparse_frame_pad_backfill_limit(self): result = sdf[:2].reindex(index, method='pad', limit=5) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): expected = sdf[:2].reindex(index).fillna(method='pad') expected = expected.to_dense() expected.values[-3:] = np.nan @@ -818,7 +826,8 @@ def test_sparse_frame_pad_backfill_limit(self): result = sdf[-2:].reindex(index, method='backfill', limit=5) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): expected = sdf[-2:].reindex(index).fillna(method='backfill') expected = expected.to_dense() expected.values[:3] = np.nan @@ -831,10 +840,12 @@ def test_sparse_frame_fillna_limit(self): sdf = df.to_sparse() result = sdf[:2].reindex(index) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): result = result.fillna(method='pad', limit=5) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): expected = sdf[:2].reindex(index).fillna(method='pad') expected = expected.to_dense() expected.values[-3:] = np.nan @@ -842,10 +853,12 @@ def test_sparse_frame_fillna_limit(self): tm.assert_frame_equal(result, expected) result = sdf[-2:].reindex(index) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): result = result.fillna(method='backfill', limit=5) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): expected = sdf[-2:].reindex(index).fillna(method='backfill') expected = expected.to_dense() expected.values[:3] = np.nan @@ -1280,6 +1293,7 @@ def test_default_fill_value_with_no_data(self): tm.assert_frame_equal(expected, result) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseDataFrameArithmetic: def test_numeric_op_scalar(self): @@ -1309,6 +1323,7 @@ def test_comparison_op_scalar(self): tm.assert_frame_equal(res.to_dense(), df != 0) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseDataFrameAnalytics: def test_cumsum(self, float_frame): diff --git a/pandas/tests/sparse/frame/test_to_csv.py b/pandas/tests/sparse/frame/test_to_csv.py index 8b2c1b951fdfe..0dda6b5cbbdae 100644 --- a/pandas/tests/sparse/frame/test_to_csv.py +++ b/pandas/tests/sparse/frame/test_to_csv.py @@ -5,6 +5,7 @@ from pandas.util import testing as tm +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseDataFrameToCsv: fill_values = [np.nan, 0, None, 1] diff --git a/pandas/tests/sparse/frame/test_to_from_scipy.py b/pandas/tests/sparse/frame/test_to_from_scipy.py index a80a51a66017e..269d67976b567 100644 --- a/pandas/tests/sparse/frame/test_to_from_scipy.py +++ b/pandas/tests/sparse/frame/test_to_from_scipy.py @@ -19,6 +19,7 @@ @pytest.mark.parametrize('fill_value', [None, 0, np.nan]) @pytest.mark.parametrize('dtype', [bool, int, float, np.uint16]) @ignore_matrix_warning +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_from_to_scipy(spmatrix, index, columns, fill_value, dtype): # GH 4343 # Make one ndarray and from it one sparse matrix, both to be used for @@ -69,6 +70,7 @@ def test_from_to_scipy(spmatrix, index, columns, fill_value, dtype): @pytest.mark.parametrize('fill_value', [None, 0, np.nan]) # noqa: F811 @ignore_matrix_warning @pytest.mark.filterwarnings("ignore:object dtype is not supp:UserWarning") +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_from_to_scipy_object(spmatrix, fill_value): # GH 4343 dtype = object @@ -117,6 +119,7 @@ def test_from_to_scipy_object(spmatrix, fill_value): @ignore_matrix_warning +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_from_scipy_correct_ordering(spmatrix): # GH 16179 arr = np.arange(1, 5).reshape(2, 2) @@ -136,6 +139,7 @@ def test_from_scipy_correct_ordering(spmatrix): @ignore_matrix_warning +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_from_scipy_fillna(spmatrix): # GH 16112 arr = np.eye(3) @@ -169,6 +173,7 @@ def test_from_scipy_fillna(spmatrix): tm.assert_sp_frame_equal(sdf, expected) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_index_names_multiple_nones(): # https://github.com/pandas-dev/pandas/pull/24092 sparse = pytest.importorskip("scipy.sparse") diff --git a/pandas/tests/sparse/series/test_series.py b/pandas/tests/sparse/series/test_series.py index 004a382f9067c..b8d0fab1debbd 100644 --- a/pandas/tests/sparse/series/test_series.py +++ b/pandas/tests/sparse/series/test_series.py @@ -21,6 +21,11 @@ from pandas.tseries.offsets import BDay +def test_deprecated(): + with tm.assert_produces_warning(FutureWarning): + pd.SparseSeries([0, 1]) + + def _test_data1(): # nan-based arr = np.arange(20, dtype=float) @@ -55,6 +60,7 @@ def _test_data2_zero(): return arr, index +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeries(SharedWithSparse): series_klass = SparseSeries @@ -532,10 +538,13 @@ def _compare(idx): exp = pd.Series(np.repeat(nan, 5)) tm.assert_series_equal(sp.take([0, 1, 2, 3, 4]), exp.to_sparse()) - with tm.assert_produces_warning(FutureWarning): + # multiple FutureWarnings, can't check stacklevel + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False): sp.take([1, 5], convert=True) - with tm.assert_produces_warning(FutureWarning): + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False): sp.take([1, 5], convert=False) def test_numpy_take(self): @@ -1032,6 +1041,7 @@ def test_memory_usage_deep(self, deep, fill_value): assert sparse_usage < dense_usage +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseHandlingMultiIndexes: def setup_method(self, method): @@ -1062,6 +1072,7 @@ def test_round_trip_preserve_multiindex_names(self): @pytest.mark.filterwarnings( "ignore:the matrix subclass:PendingDeprecationWarning" ) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeriesScipyInteraction: # Issue 8048: add SparseSeries coo methods @@ -1253,13 +1264,15 @@ def test_concat_different_fill(self): sparse1 = pd.SparseSeries(val1, name='x', kind=kind) sparse2 = pd.SparseSeries(val2, name='y', kind=kind, fill_value=0) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse1, sparse2]) exp = pd.concat([pd.Series(val1), pd.Series(val2)]) exp = pd.SparseSeries(exp, kind=kind) tm.assert_sp_series_equal(res, exp) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse2, sparse1]) exp = pd.concat([pd.Series(val2), pd.Series(val1)]) exp = pd.SparseSeries(exp, kind=kind, fill_value=0) @@ -1285,13 +1298,15 @@ def test_concat_different_kind(self): sparse1 = pd.SparseSeries(val1, name='x', kind='integer') sparse2 = pd.SparseSeries(val2, name='y', kind='block', fill_value=0) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse1, sparse2]) exp = pd.concat([pd.Series(val1), pd.Series(val2)]) exp = pd.SparseSeries(exp, kind='integer') tm.assert_sp_series_equal(res, exp) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse2, sparse1]) exp = pd.concat([pd.Series(val2), pd.Series(val1)]) exp = pd.SparseSeries(exp, kind='block', fill_value=0) @@ -1425,6 +1440,7 @@ def _dense_series_compare(s, f): tm.assert_series_equal(result.to_dense(), dense_result) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeriesAnalytics: def setup_method(self, method): @@ -1484,16 +1500,20 @@ def test_deprecated_numpy_func_call(self): for func in funcs: for series in ('bseries', 'zbseries'): with tm.assert_produces_warning(FutureWarning, - check_stacklevel=False): + check_stacklevel=False, + raise_on_extra_warnings=False): getattr(np, func)(getattr(self, series)) with tm.assert_produces_warning(FutureWarning, - check_stacklevel=False): + check_stacklevel=False, + raise_on_extra_warnings=False): getattr(getattr(self, series), func)() def test_deprecated_reindex_axis(self): # https://github.com/pandas-dev/pandas/issues/17833 - with tm.assert_produces_warning(FutureWarning) as m: + # Multiple FutureWarnings, can't check stacklevel + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False) as m: self.bseries.reindex_axis([0, 1, 2]) assert 'reindex' in str(m[0].message) @@ -1502,6 +1522,7 @@ def test_deprecated_reindex_axis(self): 'datetime_type', (np.datetime64, pd.Timestamp, lambda x: datetime.strptime(x, '%Y-%m-%d'))) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_constructor_dict_datetime64_index(datetime_type): # GH 9456 dates = ['1984-02-19', '1988-11-06', '1989-12-03', '1990-03-15'] @@ -1513,6 +1534,7 @@ def test_constructor_dict_datetime64_index(datetime_type): tm.assert_sp_series_equal(result, expected) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_to_sparse(): # https://github.com/pandas-dev/pandas/issues/22389 arr = pd.SparseArray([1, 2, None, 3]) @@ -1521,12 +1543,14 @@ def test_to_sparse(): tm.assert_sp_array_equal(result.values, arr, check_kind=False) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_constructor_mismatched_raises(): msg = "Length of passed values is 2, index implies 3" with pytest.raises(ValueError, match=msg): SparseSeries([1, 2], index=[1, 2, 3]) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_block_deprecated(): s = SparseSeries([1]) with tm.assert_produces_warning(FutureWarning): diff --git a/pandas/tests/sparse/test_combine_concat.py b/pandas/tests/sparse/test_combine_concat.py index dff5b51d7a967..ed29f24ae677f 100644 --- a/pandas/tests/sparse/test_combine_concat.py +++ b/pandas/tests/sparse/test_combine_concat.py @@ -35,6 +35,7 @@ def test_uses_first_kind(self, kind): assert result.kind == kind +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeriesConcat: @pytest.mark.parametrize('kind', [ @@ -82,14 +83,16 @@ def test_concat_different_fill(self): sparse1 = pd.SparseSeries(val1, name='x', kind=kind) sparse2 = pd.SparseSeries(val2, name='y', kind=kind, fill_value=0) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse1, sparse2]) exp = pd.concat([pd.Series(val1), pd.Series(val2)]) exp = pd.SparseSeries(exp, kind=kind) tm.assert_sp_series_equal(res, exp) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse2, sparse1]) exp = pd.concat([pd.Series(val2), pd.Series(val1)]) @@ -176,6 +179,7 @@ def test_concat_sparse_dense(self, kind): tm.assert_series_equal(res, exp) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseDataFrameConcat: def setup_method(self, method): @@ -245,12 +249,14 @@ def test_concat_different_fill_value(self): sparse = self.dense1.to_sparse() sparse2 = self.dense2.to_sparse(fill_value=0) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse, sparse2]) exp = pd.concat([self.dense1, self.dense2]).to_sparse() tm.assert_sp_frame_equal(res, exp, consolidate_block_indices=True) - with tm.assert_produces_warning(PerformanceWarning): + with tm.assert_produces_warning(PerformanceWarning, + raise_on_extra_warnings=False): res = pd.concat([sparse2, sparse]) exp = pd.concat([self.dense2, self.dense1]).to_sparse(fill_value=0) exp._default_fill_value = np.nan @@ -260,9 +266,15 @@ def test_concat_different_columns_sort_warns(self): sparse = self.dense1.to_sparse() sparse3 = self.dense3.to_sparse() - with tm.assert_produces_warning(FutureWarning): + # stacklevel is wrong since we have two FutureWarnings, + # one for depr, one for sorting. + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False, + raise_on_extra_warnings=False): res = pd.concat([sparse, sparse3]) - with tm.assert_produces_warning(FutureWarning): + with tm.assert_produces_warning(FutureWarning, + check_stacklevel=False, + raise_on_extra_warnings=False,): exp = pd.concat([self.dense1, self.dense3]) exp = exp.to_sparse() diff --git a/pandas/tests/sparse/test_format.py b/pandas/tests/sparse/test_format.py index 47e457e6ed519..37c2acc587cf6 100644 --- a/pandas/tests/sparse/test_format.py +++ b/pandas/tests/sparse/test_format.py @@ -1,4 +1,5 @@ import numpy as np +import pytest from pandas.compat import is_platform_32bit, is_platform_windows @@ -9,6 +10,7 @@ use_32bit_repr = is_platform_windows() or is_platform_32bit() +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeriesFormatting: @property @@ -105,6 +107,7 @@ def test_sparse_int(self): assert result == exp +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseDataFrameFormatting: def test_sparse_frame(self): diff --git a/pandas/tests/sparse/test_groupby.py b/pandas/tests/sparse/test_groupby.py index ba5c0c4b18b3d..7abc1530618b8 100644 --- a/pandas/tests/sparse/test_groupby.py +++ b/pandas/tests/sparse/test_groupby.py @@ -5,6 +5,7 @@ import pandas.util.testing as tm +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseGroupBy: def setup_method(self, method): @@ -59,6 +60,7 @@ def test_aggfuncs(self): @pytest.mark.parametrize("fill_value", [0, np.nan]) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_groupby_includes_fill_value(fill_value): # https://github.com/pandas-dev/pandas/issues/5078 df = pd.DataFrame({'a': [fill_value, 1, fill_value, fill_value], diff --git a/pandas/tests/sparse/test_indexing.py b/pandas/tests/sparse/test_indexing.py index e388d05fe112d..21c303fa2a064 100644 --- a/pandas/tests/sparse/test_indexing.py +++ b/pandas/tests/sparse/test_indexing.py @@ -6,6 +6,7 @@ import pandas.util.testing as tm +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeriesIndexing: def setup_method(self, method): @@ -454,6 +455,7 @@ def tests_indexing_with_sparse(self, kind, fill): s.iloc[indexer] +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseSeriesMultiIndexing(TestSparseSeriesIndexing): def setup_method(self, method): @@ -599,6 +601,7 @@ def test_reindex(self): assert sparse is not res +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestSparseDataFrameIndexing: def test_getitem(self): @@ -976,6 +979,7 @@ def test_reindex_fill_value(self): tm.assert_sp_frame_equal(res, exp) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestMultitype: def setup_method(self, method): diff --git a/pandas/tests/sparse/test_pivot.py b/pandas/tests/sparse/test_pivot.py index 98e16259d25d1..48d0719bc7f2b 100644 --- a/pandas/tests/sparse/test_pivot.py +++ b/pandas/tests/sparse/test_pivot.py @@ -1,9 +1,11 @@ import numpy as np +import pytest import pandas as pd import pandas.util.testing as tm +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") class TestPivotTable: def setup_method(self, method): diff --git a/pandas/tests/sparse/test_reshape.py b/pandas/tests/sparse/test_reshape.py index 6830e40ce6533..37ec0bba2621d 100644 --- a/pandas/tests/sparse/test_reshape.py +++ b/pandas/tests/sparse/test_reshape.py @@ -15,12 +15,14 @@ def multi_index3(): return pd.MultiIndex.from_tuples([(0, 0), (1, 1), (2, 2)]) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_sparse_frame_stack(sparse_df, multi_index3): ss = sparse_df.stack() expected = pd.SparseSeries(np.ones(3), index=multi_index3) tm.assert_sp_series_equal(ss, expected) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_sparse_frame_unstack(sparse_df): mi = pd.MultiIndex.from_tuples([(0, 0), (1, 0), (1, 2)]) sparse_df.index = mi @@ -33,6 +35,7 @@ def test_sparse_frame_unstack(sparse_df): tm.assert_numpy_array_equal(unstacked_df.values, unstacked_sdf.values) +@pytest.mark.filterwarnings("ignore:Sparse:FutureWarning") def test_sparse_series_unstack(sparse_df, multi_index3): frame = pd.SparseSeries(np.ones(3), index=multi_index3).unstack() diff --git a/setup.cfg b/setup.cfg index c0833c5609bea..68d042ecfc4b8 100644 --- a/setup.cfg +++ b/setup.cfg @@ -70,9 +70,9 @@ doctest_optionflags = NORMALIZE_WHITESPACE IGNORE_EXCEPTION_DETAIL addopts = --strict-data-files xfail_strict = True filterwarnings = + error:Sparse:FutureWarning error:The SparseArray:FutureWarning - [coverage:run] branch = False omit = */tests/*