PERF: Fix performance regression in infer_dtype #30202

groutr · 2019-12-11T06:34:54Z

Fixes major performance regression in infer_dtype introduced in
aaaac86

closes infer_dtype() function slower in latest version #28814
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Fixes performance regression introduced in aaaac86

groutr · 2019-12-11T06:35:57Z

Profiling the following function (before/after):

def slow_infer_dtype():
    df = pd.DataFrame(np.ones((100000, 1000)))
    for col in df.columns:
        infer_dtype(df[col], skipna=True)

master:
300127297 function calls (300127289 primitive calls) in 67.103 seconds
infer_dtype_perf:
126297 function calls (126289 primitive calls) in 1.050 seconds

jorisvandenbossche

Thanks for the PR!

We should maybe add a asv benchmark to ensure we catch such regression in the future.

doc/source/whatsnew/v1.0.0.rst

jreback

can you add an asv benchmark (put in dtypes.py), ideally with a fair number of types that we can test

Co-Authored-By: Joris Van den Bossche <jorisvandenbossche@gmail.com>

groutr · 2019-12-11T17:27:32Z

I seem to have some issues getting asv running. ASV consistently fails to compile the cython extensions.

jbrockmendel · 2019-12-11T17:30:53Z

nice speedup!

…r_dtype_perf

pep8speaks · 2019-12-11T20:27:37Z

Hello @groutr! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-12-17 00:16:18 UTC

TomAugspurger · 2019-12-12T12:51:26Z

@groutr one more linting issue:

black --version
black, version 19.10b0
Checking black formatting
would reformat /home/runner/work/pandas/pandas/asv_bench/benchmarks/dtypes.py
Oh no! 💥 💔 💥
1 file would be reformatted, 885 files would be left unchanged.

TomAugspurger · 2019-12-12T12:52:57Z

Were you able to get the benchmarks running? Roughly how long do your new ones take?

For testing you can do asv dev -b InferDtypes, which will reuse your same python / pandas install.

groutr · 2019-12-12T15:24:26Z

@TomAugspurger oh, sorry, I didn't know we ran black on asv_bench too.
I was eventually able to get asv running.

[  0.00%] ·· Benchmarking existing-py_home_grout_miniconda3_envs_pandas-dev_bin_python
[ 25.00%] ··· dtypes.InferDtypes.time_infer                                                                                                                        ok
[ 25.00%] ··· ============= =========
                  dtype
              ------------- ---------
                np-object    318±0μs
                py-object    335±0μs
                 np-null     342±0μs
                 py-null     329±0μs
                  np-int     323±0μs
               np-floating   342±0μs
                  empty      329±0μs
                  bytes      321±0μs
              ============= =========

[ 50.00%] ··· dtypes.InferDtypes.time_infer_skipna                                                                                                                 ok
[ 50.00%] ··· ============= =========
                  dtype
              ------------- ---------
                np-object    410±0μs
                py-object    431±0μs
                 np-null     418±0μs
                 py-null     418±0μs
                  np-int     413±0μs
               np-floating   414±0μs
                  empty      415±0μs
                  bytes      415±0μs
              ============= =========

TomAugspurger · 2019-12-12T15:47:40Z

Ah one more error linting error @groutr, sorry :) You might want to setup pre-commit: https://dev.pandas.io/docs/development/contributing.html#python-pep8-black

@datapythonista I don't recall, but before moving CI checks to GitHub actions, did we run all the checks and only report on failures at the end? So if there are both black and isort issues, you'd get both reported, instead of just the first failure? Sorry if that's a known issue.

datapythonista · 2019-12-12T16:44:51Z

I'm quite sure that when we moved to github actions all checks were running even if they were failures, but that stopped working. I reported it to github one or two weeks ago, but haven't heard from them yet.

jreback · 2019-12-15T21:58:18Z

@groutr can you show the asv for master vs this branch.

groutr · 2019-12-16T21:41:02Z

~~@jreback~~
~~Running:~~

git checkout infer_dtype_perf
conda activate pandas-dev
cd asv_bench
asv continuous -f 1.1 master infer_dtype_perf -b dtypes.InferDtypes

~~Produces identical timings for both master and infer_dtype_perf.~~
~~I don't understand why, since my own timings show a major performance improvement.~~

EDIT: Please disregard this comment. I had written the benchmarking code incorrectly.

groutr · 2019-12-17T00:13:54Z

(pandas-dev) ❯ asv continuous upstream/master infer_dtype_perf -b dtypes.InferDtypes
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
·· Installing 903abc2f <infer_dtype_perf> into conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..
· Running 4 total benchmarks (2 commits * 1 environments * 2 benchmarks)
[  0.00%] · For pandas commit 37dfcc1a <master> (round 1/2):
[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 12.50%] ··· Running (dtypes.InferDtypes.time_infer--)..
[ 25.00%] · For pandas commit 903abc2f <infer_dtype_perf> (round 1/2):
[ 25.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 25.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 37.50%] ··· Running (dtypes.InferDtypes.time_infer--)..
[ 50.00%] · For pandas commit 903abc2f <infer_dtype_perf> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 62.50%] ··· dtypes.InferDtypes.time_infer                                                                                                                        ok
[ 62.50%] ··· ============= =============
                  dtype                  
              ------------- -------------
                np-object     1.95±0.2ms 
                py-object    4.51±0.04ms 
                 np-null      4.04±0.1μs 
                 py-null     4.49±0.04ms 
                  np-int      3.97±0.2μs 
               np-floating    3.96±0.4μs 
                  empty      9.32±0.06μs 
                  bytes      4.16±0.03ms 
              ============= =============

[ 75.00%] ··· dtypes.InferDtypes.time_infer_skipna                                                                                                                 ok
[ 75.00%] ··· ============= =============
                  dtype                  
              ------------- -------------
                np-object    7.09±0.06ms 
                py-object     9.66±0.1ms 
                 np-null      3.95±0.1μs 
                 py-null     6.92±0.06ms 
                  np-int     3.87±0.04μs 
               np-floating   3.94±0.03μs 
                  empty       14.3±0.2μs
                  bytes      9.53±0.07ms
              ============= =============

[ 75.00%] · For pandas commit 37dfcc1a <master> (round 2/2):
[ 75.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
[ 75.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 87.50%] ··· dtypes.InferDtypes.time_infer                                                                                                                        ok
[ 87.50%] ··· ============= =============
                  dtype
              ------------- -------------
                np-object    1.92±0.05ms
                py-object    4.47±0.08ms
                 np-null      4.09±0.7μs
                 py-null      4.56±0.2ms
                  np-int      4.06±0.1μs
               np-floating    3.94±0.2μs
                  empty       9.34±0.3μs
                  bytes      4.22±0.06ms
              ============= =============

[100.00%] ··· dtypes.InferDtypes.time_infer_skipna                                                                                                                 ok
[100.00%] ··· ============= =============
                  dtype
              ------------- -------------
                np-object     7.24±0.3ms
                py-object     9.84±0.3ms
                 np-null      11.7±0.2ms
                 py-null     7.04±0.08ms
                  np-int      11.5±0.4ms
               np-floating    11.9±0.5ms
                  empty       14.3±0.2μs
                  bytes       9.62±0.1ms
              ============= =============

       before           after         ratio
     [37dfcc1a]       [903abc2f]
     <master>         <infer_dtype_perf>
-      11.7±0.2ms       3.95±0.1μs     0.00  dtypes.InferDtypes.time_infer_skipna('np-null')
-      11.5±0.4ms      3.87±0.04μs     0.00  dtypes.InferDtypes.time_infer_skipna('np-int')
-      11.9±0.5ms      3.94±0.03μs     0.00  dtypes.InferDtypes.time_infer_skipna('np-floating')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

WillAyd

Nice speedup! Can you add a whatsnew note to 1.0 performance improvements?

groutr · 2019-12-17T17:34:06Z

@WillAyd, I already did in ea579f7

WillAyd

lgtm

jreback · 2019-12-23T17:02:37Z

thanks

Move skipna check to after type casting of values.

9edf9b3

Fixes performance regression introduced in aaaac86

groutr changed the title ~~Move skipna check to after type casting of values.~~ PERF: Fix performance regression in infer_dtype Dec 11, 2019

Add entry to whatsnew.

70c8bed

jorisvandenbossche reviewed Dec 11, 2019

View reviewed changes

doc/source/whatsnew/v1.0.0.rst Outdated Show resolved Hide resolved

jorisvandenbossche added the Performance Memory or execution speed performance label Dec 11, 2019

jorisvandenbossche added this to the 1.0 milestone Dec 11, 2019

jreback requested changes Dec 11, 2019

View reviewed changes

Update doc/source/whatsnew/v1.0.0.rst

ea579f7

Co-Authored-By: Joris Van den Bossche <jorisvandenbossche@gmail.com>

groutr added 2 commits December 11, 2019 14:26

Add asv tests for skipna.

df55864

Merge branch 'infer_dtype_perf' of github.com:groutr/pandas into infe…

28c4969

…r_dtype_perf

make pep8 happy.

94a968c

make black happy.

3ba2229

make isort happy.

f97fb1f

Make dtype inference test cases larger.

a5efb8f

Merge branch 'master' into infer_dtype_perf

903abc2

Fix benchmark params.

8d7ec5d

WillAyd reviewed Dec 17, 2019

View reviewed changes

WillAyd approved these changes Dec 17, 2019

View reviewed changes

jorisvandenbossche approved these changes Dec 18, 2019

View reviewed changes

jbrockmendel mentioned this pull request Dec 21, 2019

Possible performance regression in 9a42cbe85461c28417a5130bc80b035044c5575a #26639

Closed

groutr requested a review from jreback December 23, 2019 16:27

jreback approved these changes Dec 23, 2019

View reviewed changes

jreback merged commit db022e2 into pandas-dev:master Dec 23, 2019

AlexKirko pushed a commit to AlexKirko/pandas that referenced this pull request Dec 29, 2019

PERF: Fix performance regression in infer_dtype (pandas-dev#30202)

2e3ddfa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: Fix performance regression in infer_dtype #30202

PERF: Fix performance regression in infer_dtype #30202

groutr commented Dec 11, 2019 •

edited

Loading

groutr commented Dec 11, 2019 •

edited

Loading

jorisvandenbossche left a comment

jreback left a comment

groutr commented Dec 11, 2019

jbrockmendel commented Dec 11, 2019

pep8speaks commented Dec 11, 2019 •

edited

Loading

TomAugspurger commented Dec 12, 2019

TomAugspurger commented Dec 12, 2019

groutr commented Dec 12, 2019

TomAugspurger commented Dec 12, 2019

datapythonista commented Dec 12, 2019

jreback commented Dec 15, 2019

groutr commented Dec 16, 2019 •

edited

Loading

groutr commented Dec 17, 2019

WillAyd left a comment

groutr commented Dec 17, 2019

WillAyd left a comment

jreback commented Dec 23, 2019

PERF: Fix performance regression in infer_dtype #30202

PERF: Fix performance regression in infer_dtype #30202

Conversation

groutr commented Dec 11, 2019 • edited Loading

groutr commented Dec 11, 2019 • edited Loading

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

groutr commented Dec 11, 2019

jbrockmendel commented Dec 11, 2019

pep8speaks commented Dec 11, 2019 • edited Loading

Comment last updated at 2019-12-17 00:16:18 UTC

TomAugspurger commented Dec 12, 2019

TomAugspurger commented Dec 12, 2019

groutr commented Dec 12, 2019

TomAugspurger commented Dec 12, 2019

datapythonista commented Dec 12, 2019

jreback commented Dec 15, 2019

groutr commented Dec 16, 2019 • edited Loading

groutr commented Dec 17, 2019

WillAyd left a comment

Choose a reason for hiding this comment

groutr commented Dec 17, 2019

WillAyd left a comment

Choose a reason for hiding this comment

jreback commented Dec 23, 2019

groutr commented Dec 11, 2019 •

edited

Loading

groutr commented Dec 11, 2019 •

edited

Loading

pep8speaks commented Dec 11, 2019 •

edited

Loading

groutr commented Dec 16, 2019 •

edited

Loading