Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dispatch_to_series is very slow #26061

Closed
RunningToTheEdgeOfTheWorld opened this issue Apr 12, 2019 · 3 comments
Closed

dispatch_to_series is very slow #26061

RunningToTheEdgeOfTheWorld opened this issue Apr 12, 2019 · 3 comments
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Performance Memory or execution speed performance

Comments

@RunningToTheEdgeOfTheWorld
Copy link

RunningToTheEdgeOfTheWorld commented Apr 12, 2019

Since pandas 0.24.0 I found there is a function in ops.py named dispatch_to_series.
But dispatch_to_series is very slow, that makes my code slower.
when I use pandas==0.23.0, it cost about 100 seconds, but when I use pandas==0.24.* It cost 583 seconds.

This is my profile result.
factor_test_1555059433
A and I found pandas become more slower because you call isinstance function too many times
link to history merge : [PERF] use numexpr in dispatch_to_series #22284

@jreback
Copy link
Contributor

jreback commented Apr 12, 2019

show a minimal example with code

@jbrockmendel
Copy link
Member

Duplicate of #25883, #24990.

@RunningToTheEdgeOfTheWorld dispatch_to_series was implemented because of a correctness/performance tradeoff, and correctness wins every time. There has been some discussion about how to ameliorate the performance hit, but I'm not aware of any concrete steps. If you have any ideas, PRs are welcome.

@jbrockmendel jbrockmendel added Numeric Operations Arithmetic, Comparison, and Logical operations Performance Memory or execution speed performance labels Jul 23, 2019
@TomAugspurger
Copy link
Contributor

Closing in favor of #24990

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Performance Memory or execution speed performance
Projects
None yet
Development

No branches or pull requests

4 participants