Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] n_neighbors should be smaller than the graph degree computed by nn descent #6091

Open
thorstenwagner opened this issue Oct 1, 2024 · 5 comments
Labels
? - Needs Triage Need team to review and classify bug Something isn't working cuml-cpu

Comments

@thorstenwagner
Copy link

thorstenwagner commented Oct 1, 2024

I updated to the latest cuml (from 23.12). I'm fitting a umap to dataset with 32 features and 400k samples.

With 23.12 I did that with n_neighbors=200 and n_components=2 and it worked. With the latest version (24.08) I get:

Traceback (most recent call last):
  File "/mnt/data/twagner/Projects/TomoTwin/results/test_runs/test.py", line 11, in <module>
    reducer.fit(np.random.randn(52000,32))
  File "/opt/user_software/miniconda3_envs/tomotwin2/lib/python3.11/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper
    ret = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/user_software/miniconda3_envs/tomotwin2/lib/python3.11/site-packages/cuml/internals/api_decorators.py", line 393, in dispatch
    return self.dispatch_func(func_name, gpu_func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/user_software/miniconda3_envs/tomotwin2/lib/python3.11/site-packages/cuml/internals/api_decorators.py", line 190, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "base.pyx", line 687, in cuml.internals.base.UniversalBase.dispatch_func
  File "umap.pyx", line 668, in cuml.manifold.umap.UMAP.fit
RuntimeError: RAFT failure at file=/opt/conda/conda-bld/work/cpp/src/umap/knn_graph/algo.cuh line=115: n_neighbors should be smaller than the graph degree computed by nn descent
Obtained 25 stack frames

The magic n_neighbors number when it starts working is 64, which seems to be the default according this documentation: https://docs.rapids.ai/api/cuvs/stable/cpp_api/neighbors_nn_descent/

Here is a script to reproduce the issue:

import cuml
import numpy as np
reducer = cuml.UMAP(
    n_neighbors=200,
    n_components=2,
    n_epochs=None,  # means automatic selection
    min_dist=0.0,
    random_state=19,
    metric="euclidean"
)
reducer.fit(np.random.randn(400000,32))
print("Done")

Interestingly, when I reduce the number of samples from 400k to 50k it also works.

Any ideas what I'm doing wrong?

@daniyalaliev
Copy link

+, similar problem

@beckernick
Copy link
Member

Thanks for reporting. Would you be able to give this a try with the stable v24.10 release that just came out? I'm not able to reproduce it in the current release (though I saw it in 24.08).

import cuml
import numpy as np
reducer = cuml.UMAP(
    n_neighbors=200,
    n_components=2,
    n_epochs=None,  # means automatic selection
    min_dist=0.0,
    random_state=19,
    metric="euclidean"
)
reducer.fit(np.random.randn(400000,32))
print("Done")
Done

@beckernick
Copy link
Member

Closing, but please feel free to re-open if this reproduces in v24.10+

@aimgo
Copy link

aimgo commented Nov 21, 2024

This appears to remain an issue in not only v24.10, but also v24.12.

    # ~ Create and fit CUML UMAP
    cuml_umap = UMAP(
        n_components=2,
        n_neighbors = 100, 
        init="spectral", 
        target_weight=0.25, 
        min_dist=0.00001,
        n_epochs=1000
    ) 
    
    # ~ Fit 
    # ~ Transform
    reduced_embeddings = cuml_umap.fit_transform(X_gpu, Y_gpu) 

Used with ~ 1 million samples of dimension 784 resulted in

File ~/.local/lib/python3.12/site-packages/cuml/internals/api_decorators.py:393, in enable_device_interop.<locals>.dispatch(self, *args, **kwargs)
    391 if hasattr(self, "dispatch_func"):
    392     func_name = gpu_func.__name__
--> 393     return self.dispatch_func(func_name, gpu_func, *args, **kwargs)
    394 else:
    395     return gpu_func(self, *args, **kwargs)

File ~/.local/lib/python3.12/site-packages/cuml/internals/api_decorators.py:190, in _make_decorator_function.<locals>.decorator_function.<locals>.decorator_closure.<locals>.wrapper(*args, **kwargs)
    188         ret = func(*args, **kwargs)
    189     else:
--> 190         return func(*args, **kwargs)
    192 return cm.process_return(ret)

File base.pyx:689, in cuml.internals.base.UniversalBase.dispatch_func()

File umap.pyx:678, in cuml.manifold.umap.UMAP.fit()

RuntimeError: RAFT failure at file=/__w/cuml/cuml/cpp/src/umap/knn_graph/algo.cuh line=111: n_neighbors should be smaller than the graph degree computed by nn descent

for both versions.

@beckernick beckernick reopened this Nov 21, 2024
@beckernick
Copy link
Member

@cjnolet , do you expect this is as straighforward as adding a graph_degree = max(graph_degree, n_neighbors) for this codepath?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
? - Needs Triage Need team to review and classify bug Something isn't working cuml-cpu
Projects
None yet
Development

No branches or pull requests

4 participants