Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Dask/Distributed version #3978

Merged
merged 3 commits into from
Jun 14, 2021
Merged

Update Dask/Distributed version #3978

merged 3 commits into from
Jun 14, 2021

Conversation

pentschev
Copy link
Member

No description provided.

@pentschev pentschev requested a review from a team as a code owner June 10, 2021 20:02
@github-actions github-actions bot added conda conda issue gpuCI gpuCI issue labels Jun 10, 2021
@ajschmidt8 ajschmidt8 added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jun 10, 2021
@dantegd
Copy link
Member

dantegd commented Jun 10, 2021

@gpucibot merge

@dantegd
Copy link
Member

dantegd commented Jun 10, 2021

@ajschmidt8 @pentschev CI seems to be installing an older nightly of dask-cudf:

dask-cudf                 21.08.00a210609 py38_g83bc0320b7_125    rapidsai-nightly

which probably is the reason we're getting these fails?

@jakirkham
Copy link
Member

Guessing this is relevant ( rapidsai/cugraph#1637 (comment) )

@dantegd
Copy link
Member

dantegd commented Jun 11, 2021

rerun tests

1 similar comment
@jakirkham
Copy link
Member

rerun tests

dantegd
dantegd requested changes Jun 11, 2021
View reviewed changes
Copy link
Member

@dantegd dantegd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the change

conda/environments/cuml_dev_cuda11.0.yml Outdated
Comment on lines 17 to 18
- dask>=2021.6.0
- distributed>=2021.6.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change from installing from main? Since CI is installing from main, we should keep this so dev environments behave like CI (and mirror cudf https://github.com/rapidsai/cudf/blob/0a4e8a1c860518c9e30f5289dea07bd0879fb642/conda/environments/cudf_dev_cuda11.2.yml#L64)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's still done above. Peter is just setting a minimum version

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think what Dante was asking for is that the conda dev environment files install Dask/Distributed main to match the same as CI is running on. I did that in 80c340d now.

@pentschev
Copy link
Member Author

pentschev commented Jun 11, 2021 •
edited
Loading

Failures seem legit to the changes that were introduced in Distributed 2021.6.0, for example:

ValueError: Metadata inference failed in `len`.

You have supplied a custom function and Dask is unable to 
determine the type of output that that function returns. 

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.

I'm guessing cuML will require an update to adapt to the changes above, similar to what has been done in rapidsai/cudf#8426 . Also ccing @galipremsagar in case he has thoughts or could work on addressing those.

Also FYI, I'll be on PTO for the next couple weeks, so maybe someone will need to take this over or ensure those issues are addressed on a different PR before merging this.

@pentschev pentschev mentioned this pull request Jun 11, 2021
Merged
@galipremsagar
Copy link
Contributor

Failures seem legit to the changes that were introduced in Distributed 2021.6.0, for example:

ValueError: Metadata inference failed in `len`.

You have supplied a custom function and Dask is unable to 
determine the type of output that that function returns. 

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.

I'm guessing cuML will require an update to adapt to the changes above, similar to what has been done in rapidsai/cudf#8426 . Also ccing @galipremsagar in case he has thoughts or could work on addressing those.

Just looked at the errors, this bug is in dask-cudf which is introduced by : rapidsai/cudf#8309, I can throw in a fix for this real quick.

@pentschev
Copy link
Member Author

Just looked at the errors, this bug is in dask-cudf which is introduced by : rapidsai/cudf#8309, I can throw in a fix for this real quick.

That would be great, thanks @galipremsagar !

@galipremsagar
Copy link
Contributor

galipremsagar commented Jun 12, 2021 •
edited
Loading

I've checked out cugraph(cugraph CI is also having a similar issue: rapidsai/cugraph#1637 (comment)) locally and built from source with latest dask and don't find the errors that I see in CI. This looks like somehow the older version of dask is not being uninstalled correctly in those images. I see that tests in the jobs that are currently running are passing, so will keep an eye out on this one.

@dantegd
Copy link
Member

dantegd commented Jun 12, 2021

rerun tests

@dantegd
Copy link
Member

dantegd commented Jun 12, 2021

@gpucibot merge

@galipremsagar
Copy link
Contributor

rerun tests

4 similar comments
@jakirkham
Copy link
Member

rerun tests

@jakirkham
Copy link
Member

rerun tests

@jakirkham
Copy link
Member

rerun tests

@jakirkham
Copy link
Member

rerun tests

@codecov-commenter
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.08@8fe1b05). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@               Coverage Diff               @@
##             branch-21.08    #3978   +/-   ##
===============================================
  Coverage                ?   85.32%           
===============================================
  Files                   ?      230           
  Lines                   ?    18093           
  Branches                ?        0           
===============================================
  Hits                    ?    15437           
  Misses                  ?     2656           
  Partials                ?        0
Flag Coverage Δ
dask 47.90% <0.00%> (?)
non-dask 77.66% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8fe1b05...80c340d. Read the comment docs.

@galipremsagar
Copy link
Contributor

rerun tests

@rapids-bot rapids-bot bot merged commit 4f13873 into rapidsai:branch-21.08 Jun 14, 2021
@pentschev pentschev deleted the update-dask-versions branch June 28, 2021 10:59
vimarsh6739 pushed a commit to vimarsh6739/cuml that referenced this pull request Oct 9, 2023
@pentschev
Update Dask/Distributed version (rapidsai#3978)
99cfdd2 
Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - https://github.com/jakirkham
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: rapidsai#3978
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jakirkham jakirkham jakirkham approved these changes

@dantegd dantegd dantegd approved these changes

@ajschmidt8 ajschmidt8 ajschmidt8 approved these changes

Assignees
No one assigned
Labels
conda conda issue gpuCI gpuCI issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@pentschev @dantegd @jakirkham @galipremsagar @codecov-commenter @ajschmidt8