[BUG] Kmeans MNMG Notebook: NCCL Error #2458
Comments
Quentin-Anthony
added
? - Needs Triage
|
Hello, I am also having the same problem using MNMG K-means implementation in an HPC environment |
|
Can you provide the output of |
|
Sure. |
|
In general, we expect that the cuml Dask algorithms be executed on a cluster that is running a set of I’m not sure if it’s possible to launch Dask-CUDA workers with Dask-MPI. Cuml’s distributed algorithms will create a NCCL clique behind the scenes across the Dask workers so you shouldn’t need to use Dask-MPI at all if you are just working with cuml estimators in Python. |
|
We previously tried using the Dask-CUDA project via the instructions in the kmeans mnmg notebook, but came across the following error: |
|
I just encountered the (NCCL) error in this issue's description in the nightly but after downgrading the NCCL conda package to 2.5, I was not able to reproduce it again. On a fresh install of 0.13, I was able to reproduce your latest error and the NCCL version didn't seem to matter. It's possible this could be some combination of updated Dask/Distributed versions and binary compatiblity of recent NCCL versions. I was able to successfully execute your reproducible example in 0.14, though the conda create --name cuml_014_070220
conda activate cuml_014_070220
conda install -c rapidsai -c nvidia -c conda-forge cuml=0.14 dask-cuda=0.14 cudf=0.14 dask-cudf=0.14 dask distributed cudatoolkit=10.2 scipy scikit-learnAnd here's the updated distributed k-means example: from cuml.dask.cluster.kmeans import KMeans as cuKMeans
from cuml.dask.datasets import make_blobs
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster()
c = Client(cluster)
n_samples = 1000000
n_features = 2
n_total_partitions = len(list(c.has_what().keys()))
X, Y = make_blobs(n_samples,
n_features,
centers = 5,
n_parts = n_total_partitions,
cluster_std=0.1,
verbose=True)
kmeans_cuml = cuKMeans(init="k-means||",
n_clusters=5,
random_state=100)
kmeans_cuml.fit(X)
print(str(kmeans_cuml.predict(X).compute()))Can you give this a try and let us know if it runs successfully for you? |
|
Also created #2504 to update the k-means notebook. |
|
Hello @cjnolet Unfortunately I no longer have access to the machines used.
I was using a fresh n1-highmem google cloud instance with 4xT4 Nvidia gpus in it, using CUDA 10.2.
I will try to request access again. Att, |
|
This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. |
|
This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d. |
|
Unable to reproduce this error with the update code from Corey. The seems to have resolved. |
Describe the bug
I'm running the kmeans mnmg notebook example on an HPC system, and running 'kmeans_cuml.fit(X_cudf)' gives the error:
Full trace on Client:
Full trace on Worker:
Steps/Code to reproduce bug
Environment details (please complete the following information):
Additional context
I have no issues with running mnmg random forests or mnmg nearest neighbors with the same methods. This issue seems to be specific to kmeans
The text was updated successfully, but these errors were encountered: