Fix to _get_nearest_indexer for pydata/xarray#3751 #32905

spencerkclark · 2020-03-22T14:57:14Z

@jbrockmendel thanks for encouraging me to submit a PR. I'm totally open to any suggested changes; this is just a sketch of something that would work for our needs and hopefully not break anything in pandas.

To those not familiar, some changes introduced in #31511 broke some functionality of an Index subclass we define in xarray. More details/discussion can be found in pydata/xarray#3751 and pydata/xarray#3764, but the gist of the issue is that for better or for worse, we have been relying on left and right distances in _get_nearest_indexer being determined using NumPy arrays rather than Index objects (as they were prior to #31511).

One way to fix this is to override _get_nearest_indexer within our Index subclass, which we've temporarily done in pydata/xarray#3764, but this potentially fragile. It would be great if we could find a solution here.

tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

ci/deps/travis-36-cov.yaml

spencerkclark · 2020-03-22T16:48:34Z

The test I added fails in CI, because of another issue, which I proposed a downstream fix for in pydata/xarray#3874 (comment).

pandas/core/indexes/base.py

pandas/tests/test_downstream.py

pandas/core/indexes/base.py

jbrockmendel · 2020-03-22T22:39:35Z

Looks like CFTImeIndex.__new__ needs to instantiate _cache; for now i guess the test will have to monkeypatch

spencerkclark · 2020-03-22T23:09:20Z

It occurred to me we could write a more direct test that circumvents the _cache issue for now; that way we won't need to temporarily monkeypatch.

jbrockmendel · 2020-03-23T01:21:09Z

Looks like test failures are happening because we are calling _filter_indexer_tolerance inconsistently (in master, unrelated to this PR).

To fix this, I think we need to go back to your way of doing it: pass target._values to _filter_indexer_tolerance, and we'll have to update the annotation

spencerkclark · 2020-03-23T11:21:29Z

Cool, that seemed to do it; looks like things are all green now. Let me know if you have any further comments.

jreback · 2020-03-23T11:42:14Z

ci/deps/azure-36-locale.yaml

@@ -15,6 +15,7 @@ dependencies:

  # pandas dependencies
  - beautifulsoup4
+  - cftime  # Needed for downstream xarray.CFTimeIndex test


pls move these near xarray

jreback · 2020-03-23T11:42:50Z

ci/deps/azure-macos-36.yaml

@@ -14,6 +14,7 @@ dependencies:
  # pandas dependencies
  - beautifulsoup4
  - bottleneck
+  - cftime  # Needed for downstream xarray.CFTimeIndex test


these should not be in every CI yaml
rather u skip the test if if it’s not installed

Yes, I understand; I was just following @jbrockmendel's recommendation. Which CI yaml should I put this in?

remove from all
except the environment.yaml

pandas/tests/test_downstream.py

spencerkclark · 2020-03-25T12:32:41Z

I think this may be ready now. Does this require a what's new entry? If so, I notice this was tagged for a 1.0.4 release. Should I create a new what's new file for that?

jreback

impl and tests look good. can you add a whatsnew in 1.1 (I am not sure we are actually doing a 1.0.4). put in bug fixes indexing. you can reference this PR number is fine (if you want to link to the xarray issue that's ok too).

spencerkclark · 2020-03-26T11:04:27Z

Great, thanks @jreback. I added a what's new entry. Let me know if you'd like me to tweak it at all. I appreciate your and @jbrockmendel's help with this.

jreback · 2020-03-26T12:47:09Z

doc/source/whatsnew/v1.1.0.rst

@@ -310,6 +310,7 @@ Indexing
 - Bug in :meth:`DataFrame.iloc.__setitem__` on a :class:`DataFrame` with duplicate columns incorrectly setting values for all matching columns (:issue:`15686`, :issue:`22036`)
 - Bug in :meth:`DataFrame.loc:` and :meth:`Series.loc` with a :class:`DatetimeIndex`, :class:`TimedeltaIndex`, or :class:`PeriodIndex` incorrectly allowing lookups of non-matching datetime-like dtypes (:issue:`32650`)
 - Bug in :meth:`Series.__getitem__` indexing with non-standard scalars, e.g. ``np.dtype`` (:issue:`32684`)
+- :meth:`Index._get_nearest_indexer` now internally uses NumPy arrays or ExtensionArrays to compute left and right distances.  This preserves the fix to :issue:`26683` while maintaining compatibility with downstream :class:`Index` subclasses, e.g. xarray's CFTimeIndex (:issue:`32905`). See also `pydata/xarray#3751 <https://github.com/pydata/xarray/issues/3751>`_.


don't reference an internal routine, rather only user visible things. You can just mention xarray.

jreback · 2020-03-26T13:45:45Z

lgtm ping on green.

spencerkclark · 2020-03-26T14:29:27Z

Many thanks @jreback; looks like things are green.

jreback · 2020-03-26T22:59:55Z

thanks @spencerkclark very nice! keep em coming!

…exer for pydata/xarray#3751)

…ydata/xarray#3751) (#34025) Co-authored-by: Spencer Clark <spencerkclark@gmail.com>

Possible _get_nearest_indexer fix

2687e34

spencerkclark commented Mar 22, 2020

View reviewed changes

ci/deps/travis-36-cov.yaml Outdated Show resolved Hide resolved

spencerkclark mentioned this pull request Mar 22, 2020

more upstream-dev cftime failures pydata/xarray#3751

Closed

isort

7b4458d

jbrockmendel reviewed Mar 22, 2020

View reviewed changes

pandas/core/indexes/base.py Outdated Show resolved Hide resolved

jbrockmendel reviewed Mar 22, 2020

View reviewed changes

pandas/tests/test_downstream.py Outdated Show resolved Hide resolved

jorisvandenbossche added this to the 1.0.4 milestone Mar 22, 2020

spencerkclark added 5 commits March 22, 2020 16:36

Address initial review comments

5998295

target = target._values

97bfabe

Add comments to environment yaml files

8034e55

Remove unused import

5abefa3

Fix mypy error

fc2aebf

jbrockmendel reviewed Mar 22, 2020

View reviewed changes

pandas/core/indexes/base.py Show resolved Hide resolved

spencerkclark added 2 commits March 22, 2020 19:02

Update _filter_indexer_tolerance and write a better test

063488a

Remove unneeded noqa again

2d4bc9b

Fix typo

1c467c3

Revert back to previous _filter_indexer_tolerance solution

addb4e1

jreback requested changes Mar 23, 2020

View reviewed changes

Move cftime to proper CI environment

764439e

jreback added the Dependencies Required and optional dependencies label Mar 23, 2020

spencerkclark added 2 commits March 23, 2020 08:46

Use preferred pandas skipping logic; regenerate requirements-dev.txt

c04fe3c

Merge branch 'master' into possible-cftime-nearest-fix

389064c

spencerkclark changed the title ~~Possible fix to _get_nearest_indexer for pydata/xarray#3751~~ Fix to _get_nearest_indexer for pydata/xarray#3751 Mar 25, 2020

jreback removed this from the 1.0.4 milestone Mar 26, 2020

jreback added this to the 1.1 milestone Mar 26, 2020

jreback requested changes Mar 26, 2020

View reviewed changes

spencerkclark added 2 commits March 26, 2020 06:45

Merge branch 'master' into possible-cftime-nearest-fix

7dda7b1

Add a what's new entry

3c68605

jreback requested changes Mar 26, 2020

View reviewed changes

spencerkclark and others added 2 commits March 26, 2020 08:54

Remove reference to Index internals

b795f4c

Merge branch 'master' into possible-cftime-nearest-fix

689077d

jreback approved these changes Mar 26, 2020

View reviewed changes

jreback merged commit 883379c into pandas-dev:master Mar 26, 2020

spencerkclark mentioned this pull request Apr 8, 2020

Add workflow to run model with supplied nudging tendency ai2cm/fv3net#215

Merged

spencerkclark deleted the possible-cftime-nearest-fix branch April 9, 2020 01:07

jorisvandenbossche modified the milestones: 1.1, 1.0.4 May 5, 2020

simonjayhawkins added the Still Needs Manual Backport label May 6, 2020

simonjayhawkins pushed a commit to simonjayhawkins/pandas that referenced this pull request May 6, 2020

Backport PR pandas-dev#32905 on branch 1.0.x (Fix to _get_nearest_ind…

3c838ae

…exer for pydata/xarray#3751)

simonjayhawkins mentioned this pull request May 6, 2020

Backport PR #32905 on branch 1.0.x (Fix to _get_nearest_indexer for p… #34025

Merged

simonjayhawkins removed the Still Needs Manual Backport label May 6, 2020

simonjayhawkins added a commit that referenced this pull request May 6, 2020

Backport PR #32905 on branch 1.0.x (Fix to _get_nearest_indexer for p…

217413a

…ydata/xarray#3751) (#34025) Co-authored-by: Spencer Clark <spencerkclark@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix to _get_nearest_indexer for pydata/xarray#3751 #32905

Fix to _get_nearest_indexer for pydata/xarray#3751 #32905

spencerkclark commented Mar 22, 2020 •

edited

Loading

spencerkclark commented Mar 22, 2020

jbrockmendel commented Mar 22, 2020

spencerkclark commented Mar 22, 2020

jbrockmendel commented Mar 23, 2020

spencerkclark commented Mar 23, 2020

jreback Mar 23, 2020

jreback Mar 23, 2020

spencerkclark Mar 23, 2020

jreback Mar 23, 2020

spencerkclark commented Mar 25, 2020

jreback left a comment

spencerkclark commented Mar 26, 2020

jreback Mar 26, 2020

jreback commented Mar 26, 2020

spencerkclark commented Mar 26, 2020 •

edited

Loading

jreback commented Mar 26, 2020

Fix to _get_nearest_indexer for pydata/xarray#3751 #32905

Fix to _get_nearest_indexer for pydata/xarray#3751 #32905

Conversation

spencerkclark commented Mar 22, 2020 • edited Loading

spencerkclark commented Mar 22, 2020

jbrockmendel commented Mar 22, 2020

spencerkclark commented Mar 22, 2020

jbrockmendel commented Mar 23, 2020

spencerkclark commented Mar 23, 2020

jreback Mar 23, 2020

Choose a reason for hiding this comment

jreback Mar 23, 2020

Choose a reason for hiding this comment

spencerkclark Mar 23, 2020

Choose a reason for hiding this comment

jreback Mar 23, 2020

Choose a reason for hiding this comment

spencerkclark commented Mar 25, 2020

jreback left a comment

Choose a reason for hiding this comment

spencerkclark commented Mar 26, 2020

jreback Mar 26, 2020

Choose a reason for hiding this comment

jreback commented Mar 26, 2020

spencerkclark commented Mar 26, 2020 • edited Loading

jreback commented Mar 26, 2020

spencerkclark commented Mar 22, 2020 •

edited

Loading

spencerkclark commented Mar 26, 2020 •

edited

Loading