Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: benchmark mode should wait for all partitions in parallel #4491

Closed
mvashishtha opened this issue May 24, 2022 · 0 comments · Fixed by #4656
Closed

BUG: benchmark mode should wait for all partitions in parallel #4491

mvashishtha opened this issue May 24, 2022 · 0 comments · Fixed by #4656
Assignees
@noloerino
Labels
Benchmarking 🏁 Issues and pull requests for evaluating runtime bug 🦗 Something isn't working

Comments

@mvashishtha
Copy link
Collaborator

We are waiting for all partitions in sequence, but it's more accurate to wait for them all at once, e.g with ray.wait(partitions, num_returns=len(partitions)) for Ray. The performance difference is probably negligible because iterating across all the partitions should be really fast.

modin/modin/core/dataframe/pandas/partitioning/partition_manager.py

Line 66 in c17dde7

all(map(lambda partition: partition.wait() or True, partitions.flatten()))

@mvashishtha mvashishtha mentioned this issue May 24, 2022
Merged
8 tasks
@mvashishtha mvashishtha added bug 🦗 Something isn't working Benchmarking 🏁 Issues and pull requests for evaluating runtime labels May 24, 2022
@mvashishtha mvashishtha mentioned this issue Jun 9, 2022
Merged
8 tasks
@noloerino noloerino self-assigned this Jul 6, 2022
@noloerino noloerino mentioned this issue Jul 7, 2022
Merged
8 tasks
noloerino added a commit to noloerino/modin that referenced this issue Jul 12, 2022
@noloerino
FIX-modin-project#4491: Wait for all partitions in parallel in benchm…
7db9a58 
…ark mode

Signed-off-by: Jonathan Shi <jhshi@ponder.io>
mvashishtha pushed a commit that referenced this issue Jul 15, 2022
@noloerino
FIX-#4491: Wait for all partitions in parallel in benchmark mode (#4656)
7a36071 
* FIX-#4491: Wait for all partitions in parallel in benchmark mode

Signed-off-by: Jonathan Shi <jhshi@ponder.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@noloerino noloerino

Labels
Benchmarking 🏁 Issues and pull requests for evaluating runtime bug 🦗 Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

FIX-#4491: Wait for all partitions in parallel in benchmark mode noloerino/modin
2 participants
@mvashishtha @noloerino