Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: go direct to _map_partitions in functions that receive already flat deps #558

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

lgray
Copy link
Collaborator

@lgray lgray commented Dec 9, 2024

Justification: if the dependencies are already flat there's no need to unpack them.

@martindurant
Copy link
Collaborator

I suppose we need a dask version pin or wait for the Task stuff?

@lgray
Copy link
Collaborator Author

lgray commented Dec 16, 2024

I thought of something even better for this.

I think I just want to change map_partitions to check for nested collections (compare traverse=True to traverse=False deps) and if they're the same just go directly to _map_partitions. It'll save a lot of wrapping.

@lgray lgray changed the title perf: go direct to _map_partitions in behavior functions perf: go direct to _map_partitions in functions that receive already flat deps Dec 16, 2024
@lgray
Copy link
Collaborator Author

lgray commented Dec 16, 2024

cool old tests pass. I'll devise a test to make sure it is doing what I expect.

@lgray lgray closed this Dec 16, 2024
@lgray lgray reopened this Dec 16, 2024
@lgray lgray force-pushed the behavior-raw-map-partitions branch from d0f41d1 to be4e7ba Compare December 17, 2024 02:00
@lgray
Copy link
Collaborator Author

lgray commented Dec 17, 2024

so this is ready to go except for the fact it uncovered a bug in fuse linear chains. it does not respect kwargs for the older dask versions.

@martindurant
Copy link
Collaborator

except for the fact it uncovered a bug in fuse linear chains

Make that a separate issue and merge this?

@lgray
Copy link
Collaborator Author

lgray commented Dec 17, 2024

I'd have to back out the change that allows non-collection kwargs to pass through if I didn't fix that issue.

@lgray
Copy link
Collaborator Author

lgray commented Dec 17, 2024

But I do agree it's probably better to factor.

pfackeldey added a commit to pfackeldey/dask-awkward that referenced this pull request Dec 17, 2024
@lgray lgray closed this Dec 17, 2024
@lgray lgray reopened this Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants