Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DEPR: MultiIndex.to_hierarchical, labels #29766

Merged
merged 10 commits into from
Nov 25, 2019
Merged

DEPR: MultiIndex.to_hierarchical, labels #29766

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
2d3e009
DEPR: MultiIndex.to_hierarchical, labels
jbrockmendel Nov 21, 2019
68ea192
Update doc/source/whatsnew/v1.0.0.rst
jbrockmendel Nov 21, 2019
042d394
Merge branch 'master' of https://github.com/pandas-dev/pandas into de…
jbrockmendel Nov 21, 2019
e22ba3f
update docs
jbrockmendel Nov 21, 2019
a98c259
Merge branch 'depr-mi' of github.com:jbrockmendel/pandas into depr-mi
jbrockmendel Nov 21, 2019
0a0de52
Bump pyarrow min version to 0.12.0
jbrockmendel Nov 22, 2019
5e0aae4
add Changed column
jbrockmendel Nov 22, 2019
fecb434
Merge branch 'master' of https://github.com/pandas-dev/pandas into de…
jbrockmendel Nov 22, 2019
c2e9f14
bump numpy for the py36 macos azure build
jbrockmendel Nov 22, 2019
bb15c47
Merge branch 'master' of https://github.com/pandas-dev/pandas into de…
jbrockmendel Nov 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ci/deps/azure-macos-36.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ dependencies:
- matplotlib=2.2.3
- nomkl
- numexpr
- numpy=1.13.3
- numpy=1.14
- openpyxl
- pyarrow
- pyarrow>=0.12.0
- pytables
- python-dateutil==2.6.1
- pytz
Expand Down
2 changes: 1 addition & 1 deletion ci/deps/azure-windows-36.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ dependencies:
- numexpr
- numpy=1.15.*
- openpyxl
- pyarrow
- pyarrow>=0.12.0
- pytables
- python-dateutil
- pytz
Expand Down
1 change: 0 additions & 1 deletion doc/redirects.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -828,7 +828,6 @@ generated/pandas.MultiIndex.sortlevel,../reference/api/pandas.MultiIndex.sortlev
generated/pandas.MultiIndex.swaplevel,../reference/api/pandas.MultiIndex.swaplevel
generated/pandas.MultiIndex.to_flat_index,../reference/api/pandas.MultiIndex.to_flat_index
generated/pandas.MultiIndex.to_frame,../reference/api/pandas.MultiIndex.to_frame
generated/pandas.MultiIndex.to_hierarchical,../reference/api/pandas.MultiIndex.to_hierarchical
generated/pandas.notna,../reference/api/pandas.notna
generated/pandas.notnull,../reference/api/pandas.notnull
generated/pandas.option_context,../reference/api/pandas.option_context
Expand Down
2 changes: 1 addition & 1 deletion doc/source/getting_started/install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ matplotlib 2.2.2 Visualization
openpyxl 2.4.8 Reading / writing for xlsx files
pandas-gbq 0.8.0 Google Big Query access
psycopg2 PostgreSQL engine for sqlalchemy
pyarrow 0.9.0 Parquet and feather reading / writing
pyarrow 0.12.0 Parquet and feather reading / writing
pymysql 0.7.11 MySQL engine for sqlalchemy
pyreadstat SPSS files (.sav) reading
pytables 3.4.2 HDF5 reading / writing
Expand Down
1 change: 0 additions & 1 deletion doc/source/reference/indexing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,6 @@ MultiIndex components

MultiIndex.set_levels
MultiIndex.set_codes
MultiIndex.to_hierarchical
MultiIndex.to_flat_index
MultiIndex.to_frame
MultiIndex.is_lexsorted
Expand Down
103 changes: 54 additions & 49 deletions doc/source/whatsnew/v1.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,62 +240,62 @@ The following methods now also correctly output values for unobserved categories
Increased minimum versions for dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some minimum supported versions of dependencies were updated (:issue:`29723`).
Some minimum supported versions of dependencies were updated (:issue:`29766`, :issue:`29723`).
If installed, we now require:

+-----------------+-----------------+----------+
| Package | Minimum Version | Required |
+=================+=================+==========+
| numpy | 1.13.3 | X |
+-----------------+-----------------+----------+
| pytz | 2015.4 | X |
+-----------------+-----------------+----------+
| python-dateutil | 2.6.1 | X |
+-----------------+-----------------+----------+
| bottleneck | 1.2.1 | |
+-----------------+-----------------+----------+
| numexpr | 2.6.2 | |
+-----------------+-----------------+----------+
| pytest (dev) | 4.0.2 | |
+-----------------+-----------------+----------+
+-----------------+-----------------+----------+---------+
| Package | Minimum Version | Required | Changed |
+=================+=================+==========+=========+
| numpy | 1.13.3 | X | |
+-----------------+-----------------+----------+---------+
| pytz | 2015.4 | X | |
+-----------------+-----------------+----------+---------+
| python-dateutil | 2.6.1 | X | |
+-----------------+-----------------+----------+---------+
| bottleneck | 1.2.1 | | |
+-----------------+-----------------+----------+---------+
| numexpr | 2.6.2 | | |
+-----------------+-----------------+----------+---------+
| pytest (dev) | 4.0.2 | | |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you update this to 5.0.1 with an X next time you change something in whatsnew, cc @datapythonista

+-----------------+-----------------+----------+---------+

For `optional libraries <https://dev.pandas.io/docs/install.html#dependencies>`_ the general recommendation is to use the latest version.
The following table lists the lowest version per library that is currently being tested throughout the development of pandas.
Optional libraries below the lowest tested version may still work, but are not considered supported.

+-----------------+-----------------+
| Package | Minimum Version |
+=================+=================+
| beautifulsoup4 | 4.6.0 |
+-----------------+-----------------+
| fastparquet | 0.3.2 |
+-----------------+-----------------+
| gcsfs | 0.2.2 |
+-----------------+-----------------+
| lxml | 3.8.0 |
+-----------------+-----------------+
| matplotlib | 2.2.2 |
+-----------------+-----------------+
| openpyxl | 2.4.8 |
+-----------------+-----------------+
| pyarrow | 0.9.0 |
+-----------------+-----------------+
| pymysql | 0.7.1 |
+-----------------+-----------------+
| pytables | 3.4.2 |
+-----------------+-----------------+
| scipy | 0.19.0 |
+-----------------+-----------------+
| sqlalchemy | 1.1.4 |
+-----------------+-----------------+
| xarray | 0.8.2 |
+-----------------+-----------------+
| xlrd | 1.1.0 |
+-----------------+-----------------+
| xlsxwriter | 0.9.8 |
+-----------------+-----------------+
| xlwt | 1.2.0 |
+-----------------+-----------------+
+-----------------+-----------------+---------+
| Package | Minimum Version | Changed |
+=================+=================+=========+
| beautifulsoup4 | 4.6.0 | |
+-----------------+-----------------+---------+
| fastparquet | 0.3.2 | X |
+-----------------+-----------------+---------+
| gcsfs | 0.2.2 | |
+-----------------+-----------------+---------+
| lxml | 3.8.0 | |
+-----------------+-----------------+---------+
| matplotlib | 2.2.2 | |
+-----------------+-----------------+---------+
| openpyxl | 2.4.8 | |
+-----------------+-----------------+---------+
| pyarrow | 0.12.0 | X |
+-----------------+-----------------+---------+
| pymysql | 0.7.1 | |
+-----------------+-----------------+---------+
| pytables | 3.4.2 | |
+-----------------+-----------------+---------+
| scipy | 0.19.0 | |
+-----------------+-----------------+---------+
| sqlalchemy | 1.1.4 | |
+-----------------+-----------------+---------+
| xarray | 0.8.2 | |
+-----------------+-----------------+---------+
| xlrd | 1.1.0 | |
+-----------------+-----------------+---------+
| xlsxwriter | 0.9.8 | |
+-----------------+-----------------+---------+
| xlwt | 1.2.0 | |
+-----------------+-----------------+---------+

See :ref:`install.dependencies` and :ref:`install.optional_dependencies` for more.

Expand Down Expand Up @@ -389,6 +389,11 @@ or ``matplotlib.Axes.plot``. See :ref:`plotting.formatters` for more.
- :func:`core.internals.blocks.make_block` no longer accepts the "fastpath" keyword(:issue:`19265`)
- :meth:`Block.make_block_same_class` no longer accepts the "dtype" keyword(:issue:`19434`)
- Removed the previously deprecated :meth:`ExtensionArray._formatting_values`. Use :attr:`ExtensionArray._formatter` instead. (:issue:`23601`)
- Removed the previously deprecated :meth:`MultiIndex.to_hierarchical` (:issue:`21613`)
- Removed the previously deprecated :attr:`MultiIndex.labels`, use :attr:`MultiIndex.codes` instead (:issue:`23752`)
- Removed the previously deprecated "labels" keyword from the :class:`MultiIndex` constructor, use "codes" instead (:issue:`23752`)
- Removed the previously deprecated :meth:`MultiIndex.set_labels`, use :meth:`MultiIndex.set_codes` instead (:issue:`23752`)
- Removed the previously deprecated "labels" keyword from :meth:`MultiIndex.set_codes`, :meth:`MultiIndex.copy`, :meth:`MultiIndex.drop`, use "codes" instead (:issue:`23752`)
- Removed support for legacy HDF5 formats (:issue:`29787`)
- :func:`read_excel` removed support for "skip_footer" argument, use "skipfooter" instead (:issue:`18836`)
- :meth:`DataFrame.to_records` no longer supports the argument "convert_datetime64" (:issue:`18902`)
Expand Down
2 changes: 1 addition & 1 deletion pandas/compat/_optional.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"odfpy": "1.3.0",
"openpyxl": "2.4.8",
"pandas_gbq": "0.8.0",
"pyarrow": "0.9.0",
"pyarrow": "0.12.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@datapythonista here we need to update pytest

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification, I found it. It's already implemented in #29782

"pytables": "3.4.2",
"pytest": "5.0.1",
"s3fs": "0.3.0",
Expand Down
6 changes: 6 additions & 0 deletions pandas/core/indexes/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,12 @@ def _new_Index(cls, d):
from pandas.core.indexes.period import _new_PeriodIndex

return _new_PeriodIndex(cls, **d)

if issubclass(cls, ABCMultiIndex):
if "labels" in d and "codes" not in d:
# GH#23752 "labels" kwarg has been replaced with "codes"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have an issue about removing this code at some point, what are the pre-conditions for that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the pre-conditions for that?

not wanting/needing to support sufficiently-old pickles

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you clarify, meaning < 0.23.4? later? (if you can pls put in the issue)

d["codes"] = d.pop("labels")
simonjayhawkins marked this conversation as resolved.
Show resolved Hide resolved

return cls.__new__(cls, **d)


Expand Down
94 changes: 4 additions & 90 deletions pandas/core/indexes/multi.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from pandas._libs.hashtable import duplicated_int64
from pandas.compat.numpy import function as nv
from pandas.errors import PerformanceWarning, UnsortedIndexError
from pandas.util._decorators import Appender, cache_readonly, deprecate_kwarg
from pandas.util._decorators import Appender, cache_readonly

from pandas.core.dtypes.common import (
ensure_int64,
Expand Down Expand Up @@ -229,9 +229,7 @@ class MultiIndex(Index):
of the mentioned helper methods.
"""

_deprecations = Index._deprecations | frozenset(
["labels", "set_labels", "to_hierarchical"]
)
_deprecations = Index._deprecations | frozenset()

# initialize to zero-length tuples to make everything work
_typ = "multiindex"
Expand All @@ -244,7 +242,6 @@ class MultiIndex(Index):
# --------------------------------------------------------------------
# Constructors

@deprecate_kwarg(old_arg_name="labels", new_arg_name="codes")
def __new__(
cls,
levels=None,
Expand Down Expand Up @@ -813,15 +810,6 @@ def set_levels(self, levels, level=None, inplace=False, verify_integrity=True):
def codes(self):
return self._codes

@property
def labels(self):
warnings.warn(
(".labels was deprecated in version 0.24.0. Use .codes instead."),
FutureWarning,
stacklevel=2,
)
return self.codes

def _set_codes(
self, codes, level=None, copy=False, validate=True, verify_integrity=False
):
Expand Down Expand Up @@ -854,23 +842,6 @@ def _set_codes(
self._tuples = None
self._reset_cache()

def set_labels(self, labels, level=None, inplace=False, verify_integrity=True):
warnings.warn(
(
".set_labels was deprecated in version 0.24.0. "
"Use .set_codes instead."
),
FutureWarning,
stacklevel=2,
)
return self.set_codes(
codes=labels,
level=level,
inplace=inplace,
verify_integrity=verify_integrity,
)

@deprecate_kwarg(old_arg_name="labels", new_arg_name="codes")
def set_codes(self, codes, level=None, inplace=False, verify_integrity=True):
"""
Set new codes on MultiIndex. Defaults to returning
Expand Down Expand Up @@ -947,7 +918,6 @@ def set_codes(self, codes, level=None, inplace=False, verify_integrity=True):
if not inplace:
return idx

@deprecate_kwarg(old_arg_name="labels", new_arg_name="codes")
def copy(
self,
names=None,
Expand Down Expand Up @@ -981,7 +951,8 @@ def copy(
"""
name = kwargs.get("name")
names = self._validate_names(name=name, names=names, deep=deep)

if "labels" in kwargs:
raise TypeError("'labels' argument has been removed; use 'codes' instead")
Comment on lines +954 to +955
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally i think we should just get kwargs out of the signature of this function, but im fine with adding a test for this in the interim

if deep:
from copy import deepcopy

Expand Down Expand Up @@ -1700,62 +1671,6 @@ def to_frame(self, index=True, name=None):
result.index = self
return result

def to_hierarchical(self, n_repeat, n_shuffle=1):
"""
Return a MultiIndex reshaped to conform to the
shapes given by n_repeat and n_shuffle.

.. deprecated:: 0.24.0

Useful to replicate and rearrange a MultiIndex for combination
with another Index with n_repeat items.

Parameters
----------
n_repeat : int
Number of times to repeat the labels on self.
n_shuffle : int
Controls the reordering of the labels. If the result is going
to be an inner level in a MultiIndex, n_shuffle will need to be
greater than one. The size of each label must divisible by
n_shuffle.

Returns
-------
MultiIndex

Examples
--------
>>> idx = pd.MultiIndex.from_tuples([(1, 'one'), (1, 'two'),
(2, 'one'), (2, 'two')])
>>> idx.to_hierarchical(3)
MultiIndex([(1, 'one'),
(1, 'one'),
(1, 'one'),
(1, 'two'),
(1, 'two'),
(1, 'two'),
(2, 'one'),
(2, 'one'),
(2, 'one'),
(2, 'two'),
(2, 'two'),
(2, 'two')],
)
"""
levels = self.levels
codes = [np.repeat(level_codes, n_repeat) for level_codes in self.codes]
# Assumes that each level_codes is divisible by n_shuffle
codes = [x.reshape(n_shuffle, -1).ravel(order="F") for x in codes]
names = self.names
warnings.warn(
"Method .to_hierarchical is deprecated and will "
"be removed in a future version",
FutureWarning,
stacklevel=2,
)
return MultiIndex(levels=levels, codes=codes, names=names)

def to_flat_index(self):
"""
Convert a MultiIndex to an Index of Tuples containing the level values.
Expand Down Expand Up @@ -2148,7 +2063,6 @@ def repeat(self, repeats, axis=None):
def where(self, cond, other=None):
raise NotImplementedError(".where is not supported for MultiIndex operations")

@deprecate_kwarg(old_arg_name="labels", new_arg_name="codes")
def drop(self, codes, level=None, errors="raise"):
"""
Make new MultiIndex with passed list of codes deleted
Expand Down
10 changes: 1 addition & 9 deletions pandas/io/feather_format.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
""" feather-format compat """

from distutils.version import LooseVersion

from pandas.compat._optional import import_optional_dependency

from pandas import DataFrame, Int64Index, RangeIndex
Expand Down Expand Up @@ -96,15 +94,9 @@ def read_feather(path, columns=None, use_threads=True):
-------
type of object stored in file
"""
pyarrow = import_optional_dependency("pyarrow")
import_optional_dependency("pyarrow")
from pyarrow import feather

path = _stringify_path(path)

if LooseVersion(pyarrow.__version__) < LooseVersion("0.11.0"):
int_use_threads = int(use_threads)
if int_use_threads < 1:
int_use_threads = 1
return feather.read_feather(path, columns=columns, nthreads=int_use_threads)

return feather.read_feather(path, columns=columns, use_threads=bool(use_threads))
2 changes: 1 addition & 1 deletion pandas/tests/extension/arrow/test_bool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from pandas.tests.extension import base
import pandas.util.testing as tm

pytest.importorskip("pyarrow", minversion="0.10.0")
pytest.importorskip("pyarrow", minversion="0.12.0")

from .arrays import ArrowBoolArray, ArrowBoolDtype # isort:skip

Expand Down
2 changes: 1 addition & 1 deletion pandas/tests/extension/arrow/test_string.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import pandas as pd

pytest.importorskip("pyarrow", minversion="0.10.0")
pytest.importorskip("pyarrow", minversion="0.12.0")

from .arrays import ArrowStringDtype # isort:skip

Expand Down
Loading