PERF-#5268: Call `get` on all partitions at once in `to_pandas` #4776

anmyachev · 2022-08-04T20:28:41Z

Signed-off-by: Myachev anatoly.myachev@intel.com

What do these changes do?

commit message follows format outlined here
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves Improve scaling of to_pandas; getting all objects from partitions at once #5268
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date
added (Issue Number: PR title (PR Number)) and github username to release notes for next major release

codecov · 2022-08-04T20:39:02Z

Codecov Report

Merging #4776 (be44af7) into master (19ac8ff) will decrease coverage by 18.90%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           master    #4776       +/-   ##
===========================================
- Coverage   84.82%   65.92%   -18.91%     
===========================================
  Files         268      269        +1     
  Lines       19701    19986      +285     
===========================================
- Hits        16712    13176     -3536     
- Misses       2989     6810     +3821

Impacted Files	Coverage Δ
...dataframe/pandas/partitioning/partition_manager.py	`89.31% <100.00%> (+3.94%)`	⬆️
modin/core/execution/dask/__init__.py	`0.00% <0.00%> (-100.00%)`	⬇️
modin/experimental/sklearn/__init__.py	`0.00% <0.00%> (-100.00%)`	⬇️
modin/core/execution/dask/common/__init__.py	`0.00% <0.00%> (-100.00%)`	⬇️
modin/experimental/xgboost/test/test_dmatrix.py	`0.00% <0.00%> (-100.00%)`	⬇️
modin/experimental/xgboost/test/test_xgboost.py	`0.00% <0.00%> (-100.00%)`	⬇️
modin/experimental/core/execution/ray/__init__.py	`0.00% <0.00%> (-100.00%)`	⬇️
modin/core/execution/dask/common/engine_wrapper.py	`0.00% <0.00%> (-100.00%)`	⬇️
...odin/experimental/core/storage_formats/__init__.py	`0.00% <0.00%> (-100.00%)`	⬇️
...in/core/execution/dask/implementations/__init__.py	`0.00% <0.00%> (-100.00%)`	⬇️
... and 112 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

anmyachev · 2022-11-22T19:03:26Z

@vnlitvinov could you take a look?

vnlitvinov

Got benchmarks for this change?

modin/core/dataframe/pandas/partitioning/partition_manager.py

modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.py

modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py

anmyachev · 2022-11-23T14:51:13Z

Got benchmarks for this change?

@vnlitvinov results by using asv:

the PR

          --                             cpus
          --------------- ---------------------------------
               shape          4          16          32    
          =============== ========== ========== ===========
            [5000, 5000]   164±10ms   283±10ms    449±30ms
           [1000000, 10]   60.2±4ms   74.6±3ms   84.9±10ms
          =============== ========== ========== ===========

Master branch

          --                             cpus
          --------------- ---------------------------------
               shape          4          16          32
          =============== ========== ========== ===========
            [5000, 5000]   167±4ms    368±10ms   1.01±0.1s
           [1000000, 10]   63.3±6ms   77.3±4ms    96.5±6ms
          =============== ========== ========== ===========

…pandas' Signed-off-by: Myachev <anatoly.myachev@intel.com>

Signed-off-by: Myachev <anatoly.myachev@intel.com>

Co-authored-by: Vasily Litvinov <fam1ly.n4me@yandex.ru>

Signed-off-by: Myachev <anatoly.myachev@intel.com>

anmyachev · 2022-11-24T15:55:04Z

@vnlitvinov ready for review

modin/core/dataframe/pandas/partitioning/partition_manager.py

Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.py

Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

Signed-off-by: Myachev <anatoly.myachev@intel.com>

modin/core/dataframe/pandas/partitioning/partition_manager.py

Co-authored-by: Vasily Litvinov <fam1ly.n4me@yandex.ru>

Signed-off-by: Myachev <anatoly.myachev@intel.com>

vnlitvinov

LGTM, minor questions left only

modin/core/dataframe/pandas/partitioning/partition_manager.py

Signed-off-by: Myachev <anatoly.myachev@intel.com>

vnlitvinov

LGTM!

vnlitvinov · 2022-11-25T14:46:28Z

@anmyachev one note - this PR is solving only half of problem stated in #2814 (getting things serially), but not the other (using N*N pandas.concat operations), maybe we should open a new issue for the N-square concat-s?

anmyachev · 2022-11-25T16:10:26Z

@anmyachev one note - this PR is solving only half of problem stated in #2814 (getting things serially), but not the other (using N*N pandas.concat operations), maybe we should open a new issue for the N-square concat-s?

I created a separate issue: #5268

anmyachev force-pushed the issue2814 branch from afdd905 to f73c3f5 Compare August 4, 2022 22:01

anmyachev force-pushed the issue2814 branch 2 times, most recently from 23511c1 to cca6978 Compare November 22, 2022 18:36

anmyachev changed the title ~~PERF-#2814: Call 'get' on all partitions at once in 'to_pandas'~~ PERF-#2814: Call get on all partitions at once in to_pandas Nov 22, 2022

anmyachev marked this pull request as ready for review November 22, 2022 18:55

anmyachev requested a review from a team as a code owner November 22, 2022 18:55

anmyachev added the Ready for review label Nov 22, 2022

vnlitvinov reviewed Nov 22, 2022

View reviewed changes

anmyachev and others added 7 commits November 24, 2022 14:55

PERF-modin-project#2814: Call 'get' on all partitions at once in 'to_…

bf4c0aa

…pandas' Signed-off-by: Myachev <anatoly.myachev@intel.com>

remove old release note

e0fce8d

Signed-off-by: Myachev <anatoly.myachev@intel.com>

fixes

74dd77c

Signed-off-by: Myachev <anatoly.myachev@intel.com>

fix

2efe8b5

Signed-off-by: Myachev <anatoly.myachev@intel.com>

Apply suggestions from code review

75ea6c5

Co-authored-by: Vasily Litvinov <fam1ly.n4me@yandex.ru>

addres review comments

85aa805

Signed-off-by: Myachev <anatoly.myachev@intel.com>

add fix for empty frames

3b54eca

Signed-off-by: Myachev <anatoly.myachev@intel.com>

anmyachev force-pushed the issue2814 branch from 0b23477 to 3b54eca Compare November 24, 2022 13:55

dchigarev reviewed Nov 24, 2022

View reviewed changes

modin/core/dataframe/pandas/partitioning/partition_manager.py Outdated Show resolved Hide resolved

modin/core/dataframe/pandas/partitioning/partition_manager.py Outdated Show resolved Hide resolved

Update modin/core/dataframe/pandas/partitioning/partition_manager.py

811c663

Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

dchigarev reviewed Nov 24, 2022

View reviewed changes

modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.py Show resolved Hide resolved

anmyachev and others added 4 commits November 24, 2022 17:16

Update modin/core/dataframe/pandas/partitioning/partition_manager.py

81b0908

Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>

try to address review comments

8b64968

Signed-off-by: Myachev <anatoly.myachev@intel.com>

fix

425edd7

Signed-off-by: Myachev <anatoly.myachev@intel.com>

return changes

85c2caf

Signed-off-by: Myachev <anatoly.myachev@intel.com>

vnlitvinov reviewed Nov 24, 2022

View reviewed changes

modin/core/dataframe/pandas/partitioning/partition_manager.py Outdated Show resolved Hide resolved

anmyachev and others added 2 commits November 24, 2022 19:39

Update modin/core/dataframe/pandas/partitioning/partition_manager.py

9c6f11e

Co-authored-by: Vasily Litvinov <fam1ly.n4me@yandex.ru>

add comment

89453a7

Signed-off-by: Myachev <anatoly.myachev@intel.com>

vnlitvinov previously approved these changes Nov 25, 2022

View reviewed changes

modin/core/dataframe/pandas/partitioning/partition_manager.py Show resolved Hide resolved

modin/core/dataframe/pandas/partitioning/partition_manager.py Outdated Show resolved Hide resolved

change restore procedure

40f9566

Signed-off-by: Myachev <anatoly.myachev@intel.com>

anmyachev dismissed vnlitvinov’s stale review via 40f9566 November 25, 2022 09:44

vnlitvinov approved these changes Nov 25, 2022

View reviewed changes

anmyachev changed the title ~~PERF-#2814: Call get on all partitions at once in to_pandas~~ PERF-#5268: Call get on all partitions at once in to_pandas Nov 25, 2022

dchigarev approved these changes Nov 25, 2022

View reviewed changes

anmyachev mentioned this pull request Nov 25, 2022

Consider to avoid force_materialization call in to_pandas function #5269

Open

YarShev merged commit 9534478 into modin-project:master Nov 27, 2022

anmyachev deleted the issue2814 branch November 28, 2022 10:35

arunjose696 mentioned this pull request Feb 12, 2024

FIX-#6879: Convert the right DF to single partition before broadcasting in query_compiler.merge #6880

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF-#5268: Call `get` on all partitions at once in `to_pandas` #4776

PERF-#5268: Call `get` on all partitions at once in `to_pandas` #4776

anmyachev commented Aug 4, 2022 •

edited

Loading

codecov bot commented Aug 4, 2022 •

edited

Loading

anmyachev commented Nov 22, 2022

vnlitvinov left a comment

anmyachev commented Nov 23, 2022

anmyachev commented Nov 24, 2022

vnlitvinov left a comment

vnlitvinov left a comment

vnlitvinov commented Nov 25, 2022

anmyachev commented Nov 25, 2022 •

edited

Loading

PERF-#5268: Call get on all partitions at once in to_pandas #4776

PERF-#5268: Call get on all partitions at once in to_pandas #4776

Conversation

anmyachev commented Aug 4, 2022 • edited Loading

What do these changes do?

codecov bot commented Aug 4, 2022 • edited Loading

Codecov Report

anmyachev commented Nov 22, 2022

vnlitvinov left a comment

Choose a reason for hiding this comment

anmyachev commented Nov 23, 2022

the PR

Master branch

anmyachev commented Nov 24, 2022

vnlitvinov left a comment

Choose a reason for hiding this comment

vnlitvinov left a comment

Choose a reason for hiding this comment

vnlitvinov commented Nov 25, 2022

anmyachev commented Nov 25, 2022 • edited Loading

PERF-#5268: Call `get` on all partitions at once in `to_pandas` #4776

PERF-#5268: Call `get` on all partitions at once in `to_pandas` #4776

anmyachev commented Aug 4, 2022 •

edited

Loading

codecov bot commented Aug 4, 2022 •

edited

Loading

anmyachev commented Nov 25, 2022 •

edited

Loading