PERF: vectorize _interp_limit #16592
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Also, the master and 0.20.1 versions scale poorly with
So we're slightly slower than 0.20.1 for the default, but much faster with any limit (notice the units). I'll add a benchmark for |
|
Just finished vs. 0.20.1, and they're essentially the same (I think the first one is my branch, 0.20.1 is second). |
jreback
reviewed
Jun 3, 2017
pandas/core/missing.py
Outdated
| def _interp_limit0(invalid, fw_limit, bw_limit): | ||
| "Get idx of values that won't be filled b/c they exceed the limits." | ||
| for x in np.where(invalid)[0]: | ||
| if invalid[max(0, x - fw_limit):x + bw_limit + 1].all(): |
There was a problem hiding this comment.
should rewrite this as well
|
Oh sorry that's not used anymore. Just left it around for testing that my new implementation was the same. I'll remove it.
… On Jun 3, 2017, at 13:17, Jeff Reback ***@***.***> wrote:
@jreback commented on this pull request.
In pandas/core/missing.py:
> @@ -630,3 +632,65 @@ def fill_zeros(result, x, y, name, fill):
result = result.reshape(shape)
return result
+
+
+def _interp_limit0(invalid, fw_limit, bw_limit):
+ "Get idx of values that won't be filled b/c they exceed the limits."
+ for x in np.where(invalid)[0]:
+ if invalid[max(0, x - fw_limit):x + bw_limit + 1].all():
should rewrite this as well
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Codecov Report
@@ Coverage Diff @@
## master #16592 +/- ##
==========================================
+ Coverage 90.75% 90.92% +0.17%
==========================================
Files 161 161
Lines 51097 49265 -1832
==========================================
- Hits 46372 44795 -1577
+ Misses 4725 4470 -255
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #16592 +/- ##
==========================================
+ Coverage 90.75% 90.93% +0.17%
==========================================
Files 161 161
Lines 51097 49261 -1836
==========================================
- Hits 46372 44794 -1578
+ Misses 4725 4467 -258
Continue to review full report at Codecov.
|
TomAugspurger
added
Missing-data
jreback
reviewed
Jun 3, 2017
pandas/core/missing.py
Outdated
| return f_idx & b_idx | ||
|
|
||
|
|
||
| def rolling_window(a, window): |
There was a problem hiding this comment.
maybe move this inside _interp_limit to avoid confusion that this is public and/or used externally
jreback
approved these changes
Jun 4, 2017
Yeah, it's kind of unreadable, and I have no idea what that striding stuff is doing. I'll leave this open if you want to take a look. I'll tag 0.20.2 early tomorrow morning. |
|
no this is all fine maybe make an issue for 0.21 to have a look |
TomAugspurger
added a commit
to TomAugspurger/pandas
that referenced
this pull request
Jun 4, 2017
* PERF: vectorize _interp_limit * CLN: remove old implementation * fixup! CLN: remove old implementation (cherry picked from commit 473615e)
TomAugspurger
added a commit
that referenced
this pull request
Jun 4, 2017
* PERF: vectorize _interp_limit * CLN: remove old implementation * fixup! CLN: remove old implementation (cherry picked from commit 473615e)
Kiv
pushed a commit
to Kiv/pandas
that referenced
this pull request
Jun 11, 2017
* PERF: vectorize _interp_limit * CLN: remove old implementation * fixup! CLN: remove old implementation
stangirala
pushed a commit
to stangirala/pandas
that referenced
this pull request
Jun 11, 2017
* PERF: vectorize _interp_limit * CLN: remove old implementation * fixup! CLN: remove old implementation
yarikoptic
added a commit
to neurodebian/pandas
that referenced
this pull request
Jul 12, 2017
Version 0.20.2 * tag 'v0.20.2': (68 commits) RLS: v0.20.2 DOC: Update release.rst DOC: Whatsnew fixups (pandas-dev#16596) ERRR: Raise error in usecols when column doesn't exist but length matches (pandas-dev#16460) BUG: convert numpy strings in index names in HDF pandas-dev#13492 (pandas-dev#16444) PERF: vectorize _interp_limit (pandas-dev#16592) DOC: whatsnew 0.20.2 edits (pandas-dev#16587) API: Make is_strictly_monotonic_* private (pandas-dev#16576) BUG: reimplement MultiIndex.remove_unused_levels (pandas-dev#16565) Strictly monotonic (pandas-dev#16555) ENH: add .ngroup() method to groupby objects (pandas-dev#14026) (pandas-dev#14026) fix linting BUG: Incorrect handling of rolling.cov with offset window (pandas-dev#16244) BUG: select_as_multiple doesn't respect start/stop kwargs GH16209 (pandas-dev#16317) return empty MultiIndex for symmetrical difference on equal MultiIndexes (pandas-dev#16486) BUG: Bug in .resample() and .groupby() when aggregating on integers (pandas-dev#16549) BUG: Fixed tput output on windows (pandas-dev#16496) Strictly monotonic (pandas-dev#16555) BUG: fixed wrong order of ordered labels in pd.cut() BUG: Fixed to_html ignoring index_names parameter ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
xref #16584
Here are the timings vs. master
I still need to time vs 0.20.1 (will post those timings later, have to run for a bit now). I may have one more spot to optimize.