Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Semi-Supervised UMAP, with euclidean target_metric, reduction errors when input passes a certain size. #2333

Open
Tracked by #4139
DavidEverlaw opened this issue May 27, 2020 · 5 comments
Open
Tracked by #4139

[BUG] Semi-Supervised UMAP, with euclidean target_metric, reduction errors when input passes a certain size. #2333

DavidEverlaw opened this issue May 27, 2020 · 5 comments
Assignees
@cjnolet
Labels
bug Something isn't working

Comments

@DavidEverlaw
Copy link

Describe the bug
UMAP errors for supervised reduction with different sized input when using target labels and a target_metric of euclidean

Steps/Code to reproduce bug

Categorical Metric Working Input

from sklearn.datasets import make_classification 
from cuml import UMAP 
def run(n): 
    X, Y = make_classification(n_samples = n, n_features = 10, n_redundant = 0, n_informative = 10, n_clusters_per_class = 1, n_classes = 1000) 
    reducer = UMAP(
      n_neighbors = 20, 
      init="spectral", 
      target_weights=0.25, 
      target_metric="categorical") 
    X2 = reducer.fit_transform(X, Y) 
    print(X2.shape) 
    return X2 

for i in [50_000, 75_000, 76_000, 77_000, 78_000, 79_000, 80_000, 100_000, 150_000, 200_000]:
   run(i)

Categorical Output

(50000, 2)
(75000, 2)
(76000, 2)
(77000, 2)
(78000, 2)
(79000, 2)
(80000, 2)
(100000, 2)
(150000, 2)
(200000, 2)

Euclidean Metric Erroring Input

from sklearn.datasets import make_classification 
from cuml import UMAP 
def run(n): 
    X, Y = make_classification(n_samples = n, n_features = 10, n_redundant = 0, n_informative = 10, n_clusters_per_class = 1, n_classes = 1000) 
    reducer = UMAP(
      n_neighbors = 20, 
      init="spectral", 
      target_weights=0.25, 
      target_metric="euclidean") 
    X2 = reducer.fit_transform(X, Y) 
    print(X2.shape) 
    return X2 

for i in [50_000, 75_000, 76_000, 77_000, 78_000, 79_000, 80_000, 100_000, 150_000, 200_000]:
   run(i)

Output with error

(50000, 2)
(75000, 2)
(76000, 2)
(77000, 2)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-f02ae8190cb2> in <module>
     13 
     14 for i in [50_000, 75_000, 76_000, 77_000, 78_000, 79_000, 80_000, 100_000, 150_000, 200_000]:
---> 15    run(i)
     16 

<ipython-input-1-f02ae8190cb2> in run(n)
      8       target_weights=0.25,
      9       target_metric="euclidean") 
---> 10     X2 = reducer.fit_transform(X, Y)
     11     print(X2.shape)
     12     return X2

cuml/manifold/umap.pyx in cuml.manifold.umap.UMAP.fit_transform()

~/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/memory_utils.py in cupy_rmm_wrapper(*args, **kwargs)
     54     def cupy_rmm_wrapper(*args, **kwargs):
     55         with cupy_using_allocator(rmm.rmm_cupy_allocator):
---> 56             return func(*args, **kwargs)
     57 
     58     return cupy_rmm_wrapper

cuml/manifold/umap.pyx in cuml.manifold.umap.UMAP.fit()

RuntimeError: Exception occured! file=/conda/conda-bld/libcuml_1589443760507/work/cpp/src_prims/common/cudart_utils.h line=56: FAIL: call='cudaMemcpyAsync(dst, src, len * sizeof(Type), cudaMemcpyDefault, stream)'. Reason:an illegal memory access was encountered
Obtained 64 stack frames
#0 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7fa3b08063ae]
#1 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7fa3b0806ec0]
#2 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon4copyIiEEvPT_PKS1_mP11CUstream_st+0x105) [0x7fa391256195]
#3 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon6Sparse17csr_add_calc_indsIfLi32EEEmPKiS3_PKT_iS3_S3_S6_iiPiSt10shared_ptrINS_15deviceAllocatorEEP11CUstream_st+0x1a9) [0x7fa3915f40d9]
#4 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8UMAPAlgo10Supervised35general_simplicial_set_intersectionIfLi256EEEvPiPN8MLCommon6Sparse3COOIT_iEES2_S8_S8_fSt10shared_ptrINS3_15deviceAllocatorEEP11CUstream_st+0x164) [0x7fa3915f9ab4]
#5 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8UMAPAlgo10Supervised28perform_general_intersectionILi256EfEEvRKN2ML10cumlHandleEPT0_PN8MLCommon6Sparse3COOIS6_iEESC_PNS2_10UMAPParamsEP11CUstream_st+0x9f4) [0x7fa391600f74]
#6 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8UMAPAlgo4_fitIfLi256EEEvRKN2ML10cumlHandleEPT_S6_iiPlS6_PNS1_10UMAPParamsES6_+0x50d) [0x7fa39160224d]
#7 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/manifold/umap.cpython-37m-x86_64-linux-gnu.so(+0x17a63) [0x7fa388aeda63]
#8 in /var/lib/miniconda/envs/test-env/bin/python(PyObject_Call+0x6e) [0x5563af51bfde]
#9 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x1e9d) [0x5563af5c154d]
#10 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x5da) [0x5563af50966a]
#11 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallDict+0x3ff) [0x5563af50a6ef]
#12 in /var/lib/miniconda/envs/test-env/bin/python(_PyObject_Call_Prepend+0x63) [0x5563af529a73]
#13 in /var/lib/miniconda/envs/test-env/lib/python3.7/site-packages/cuml/manifold/umap.cpython-37m-x86_64-linux-gnu.so(+0xf846) [0x7fa388ae5846]
#14 in /var/lib/miniconda/envs/test-env/bin/python(_PyObject_FastCallKeywords+0x48b) [0x5563af5722db]
#15 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x5389) [0x5563af5c4a39]
#16 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x5563af55d02b]
#17 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x416) [0x5563af5bfac6]
#18 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x5563af509389]
#19 in /var/lib/miniconda/envs/test-env/bin/python(PyEval_EvalCodeEx+0x44) [0x5563af50a2b4]
#20 in /var/lib/miniconda/envs/test-env/bin/python(PyEval_EvalCode+0x1c) [0x5563af50a2dc]
#21 in /var/lib/miniconda/envs/test-env/bin/python(+0x1db30d) [0x5563af5cf30d]
#22 in /var/lib/miniconda/envs/test-env/bin/python(_PyMethodDef_RawFastCallKeywords+0xe9) [0x5563af55d939]
#23 in /var/lib/miniconda/envs/test-env/bin/python(_PyCFunction_FastCallKeywords+0x21) [0x5563af55dbd1]
#24 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x47a4) [0x5563af5c3e54]
#25 in /var/lib/miniconda/envs/test-env/bin/python(_PyGen_Send+0x2a2) [0x5563af572f82]
#26 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x1a76) [0x5563af5c1126]
#27 in /var/lib/miniconda/envs/test-env/bin/python(_PyGen_Send+0x2a2) [0x5563af572f82]
#28 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x1a76) [0x5563af5c1126]
#29 in /var/lib/miniconda/envs/test-env/bin/python(_PyGen_Send+0x2a2) [0x5563af572f82]
#30 in /var/lib/miniconda/envs/test-env/bin/python(_PyMethodDef_RawFastCallKeywords+0x8d) [0x5563af55d8dd]
#31 in /var/lib/miniconda/envs/test-env/bin/python(_PyMethodDescr_FastCallKeywords+0x4f) [0x5563af571dbf]
#32 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x4c9d) [0x5563af5c434d]
#33 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x5563af55d02b]
#34 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x416) [0x5563af5bfac6]
#35 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x5563af55d02b]
#36 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x690) [0x5563af5bfd40]
#37 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x5563af509389]
#38 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0x387) [0x5563af55d2b7]
#39 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x14d4) [0x5563af5c0b84]
#40 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x5563af509389]
#41 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0x325) [0x5563af55d255]
#42 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x690) [0x5563af5bfd40]
#43 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x5563af509389]
#44 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0x325) [0x5563af55d255]
#45 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x690) [0x5563af5bfd40]
#46 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0xfb) [0x5563af55d02b]
#47 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x690) [0x5563af5bfd40]
#48 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x5563af509389]
#49 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallDict+0x3ff) [0x5563af50a6ef]
#50 in /var/lib/miniconda/envs/test-env/bin/python(_PyObject_Call_Prepend+0x63) [0x5563af529a73]
#51 in /var/lib/miniconda/envs/test-env/bin/python(PyObject_Call+0x6e) [0x5563af51bfde]
#52 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x1e9d) [0x5563af5c154d]
#53 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x5563af509389]
#54 in /var/lib/miniconda/envs/test-env/bin/python(_PyFunction_FastCallKeywords+0x387) [0x5563af55d2b7]
#55 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalFrameDefault+0x416) [0x5563af5bfac6]
#56 in /var/lib/miniconda/envs/test-env/bin/python(_PyEval_EvalCodeWithName+0x2f9) [0x5563af509389]
#57 in /var/lib/miniconda/envs/test-env/bin/python(PyEval_EvalCodeEx+0x44) [0x5563af50a2b4]
#58 in /var/lib/miniconda/envs/test-env/bin/python(PyEval_EvalCode+0x1c) [0x5563af50a2dc]
#59 in /var/lib/miniconda/envs/test-env/bin/python(+0x22c664) [0x5563af620664]
#60 in /var/lib/miniconda/envs/test-env/bin/python(PyRun_FileExFlags+0xa1) [0x5563af62aa91]
#61 in /var/lib/miniconda/envs/test-env/bin/python(PyRun_SimpleFileExFlags+0x1c3) [0x5563af62ac83]
#62 in /var/lib/miniconda/envs/test-env/bin/python(+0x237db5) [0x5563af62bdb5]
#63 in /var/lib/miniconda/envs/test-env/bin/python(_Py_UnixMain+0x3c) [0x5563af62bedc]

Expected behavior
fit_transform should not error with euclidean target_metric

Environment details (please complete the following information):

NVIDIA-SMI Driver: 440.33.01, CUDA: 10.2 Tesla V100 
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   36C    P0    37W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
  • Environment location: AWS EC2 p3.2xlarge
  • Linux Distro/Architecture: Ubuntu 18.04
  • CUDA: 10.2
  • Method of cuDF & cuML install: conda install -c rapidsai-nightly -c nvidia -c conda-forge
    -c defaults rapids=0.14 python=3.7 cudatoolkit=10.2

Additional context
@DavidEverlaw DavidEverlaw added ? - Needs Triage Need team to review and classify bug Something isn't working labels May 27, 2020
@DavidEverlaw
Copy link
Author

DavidEverlaw commented May 27, 2020 •
edited
Loading

Possibly related to #1604

@cjnolet
Copy link
Member

cjnolet commented May 27, 2020

Linking to #2241. Looking into this.

@cjnolet cjnolet self-assigned this May 27, 2020
@cjnolet cjnolet removed the ? - Needs Triage Need team to review and classify label May 27, 2020
@hershkoy
Copy link

hershkoy commented Jul 15, 2020 •
edited
Loading

I am having the same issue. I tried with 0.14 and then with 0.15a nightly.
Initial attempt was with 280K samples (121 dimensions). When it crashed, I to downsample by 50 and it worked. But downsample by 30 already crashed.

Unfortunately I don't have a way for you to reproduce it on a sample dataset. I just wanted to let you know that it is still happening

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-4319d52b364b> in <module>
     25 y_tmp = y_data[:,0]
     26 
---> 27 S3 = umap_reducer.fit_transform(x_tmp[::30],y_tmp[::30])
     28 S4 = db.fit(S3)

cuml/manifold/umap.pyx in cuml.manifold.umap.UMAP.fit_transform()

/opt/conda/envs/rapids/lib/python3.7/site-packages/cuml/common/memory_utils.py in cupy_rmm_wrapper(*args, **kwargs)
     54     def cupy_rmm_wrapper(*args, **kwargs):
     55         with cupy_using_allocator(rmm.rmm_cupy_allocator):
---> 56             return func(*args, **kwargs)
     57 
     58     return cupy_rmm_wrapper

cuml/manifold/umap.pyx in cuml.manifold.umap.UMAP.fit()

RuntimeError: Exception occured! file=/rapids/cuml/cpp/src_prims/common/cudart_utils.h line=56: FAIL: call='cudaMemcpyAsync(dst, src, len * sizeof(Type), cudaMemcpyDefault, stream)'. Reason:an illegal memory access was encountered
Obtained 64 stack frames
#0 in /opt/conda/envs/rapids/lib/libcuml++.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7f871439960e]
#1 in /opt/conda/envs/rapids/lib/libcuml++.so(_ZN8MLCommon9ExceptionC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x71) [0x7f871439a181]
#2 in /opt/conda/envs/rapids/lib/libcuml++.so(_ZN8MLCommon6Sparse17csr_add_calc_indsIfLi32EEEmPKiS3_PKT_iS3_S3_S6_iiPiSt10shared_ptrINS_15deviceAllocatorEEP11CUstream_st+0x541) [0x7f8714759ef1]
#3 in /opt/conda/envs/rapids/lib/libcuml++.so(_ZN8UMAPAlgo10Supervised35general_simplicial_set_intersectionIfLi256EEEvPiPN8MLCommon6Sparse3COOIT_iEES2_S8_S8_fSt10shared_ptrINS3_15deviceAllocatorEEP11CUstream_st+0x163) [0x7f8714754113]
#4 in /opt/conda/envs/rapids/lib/libcuml++.so(_ZN8UMAPAlgo10Supervised28perform_general_intersectionILi256EfEEvRKN2ML10cumlHandleEPT0_PN8MLCommon6Sparse3COOIS6_iEESC_PNS2_10UMAPParamsEP11CUstream_st+0x940) [0x7f871474f760]
#5 in /opt/conda/envs/rapids/lib/libcuml++.so(_ZN8UMAPAlgo4_fitIfLi256EEEvRKN2ML10cumlHandleEPT_S6_iiPlS6_PNS1_10UMAPParamsES6_+0x4e2) [0x7f871473df42]
#6 in /opt/conda/envs/rapids/lib/python3.7/site-packages/cuml/manifold/umap.cpython-37m-x86_64-linux-gnu.so(+0x1355a) [0x7f8706f5e55a]
#7 in /opt/conda/envs/rapids/bin/python(PyObject_Call+0xb4) [0x5641066fb214]
#8 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x1ccd) [0x5641067a4b8d]
#9 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x7cd) [0x5641066f9a7d]
#10 in /opt/conda/envs/rapids/bin/python(_PyObject_FastCallDict+0x5be) [0x5641066faebe]
#11 in /opt/conda/envs/rapids/bin/python(+0x12f071) [0x56410670f071]
#12 in /opt/conda/envs/rapids/lib/python3.7/site-packages/cuml/manifold/umap.cpython-37m-x86_64-linux-gnu.so(+0x10a8f) [0x7f8706f5ba8f]
#13 in /opt/conda/envs/rapids/bin/python(_PyObject_FastCallKeywords+0x15c) [0x564106760bec]
#14 in /opt/conda/envs/rapids/bin/python(+0x181661) [0x564106761661]
#15 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x48a2) [0x5641067a7762]
#16 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x255) [0x5641066f9505]
#17 in /opt/conda/envs/rapids/bin/python(+0x1d7525) [0x5641067b7525]
#18 in /opt/conda/envs/rapids/bin/python(_PyMethodDef_RawFastCallKeywords+0xe9) [0x564106729789]
#19 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x4568) [0x5641067a7428]
#20 in /opt/conda/envs/rapids/bin/python(+0x180434) [0x564106760434]
#21 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x19aa) [0x5641067a486a]
#22 in /opt/conda/envs/rapids/bin/python(+0x180434) [0x564106760434]
#23 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x19aa) [0x5641067a486a]
#24 in /opt/conda/envs/rapids/bin/python(+0x180434) [0x564106760434]
#25 in /opt/conda/envs/rapids/bin/python(_PyMethodDescr_FastCallKeywords+0xdb) [0x5641067607cb]
#26 in /opt/conda/envs/rapids/bin/python(+0x18153e) [0x56410676153e]
#27 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x611) [0x5641067a34d1]
#28 in /opt/conda/envs/rapids/bin/python(_PyFunction_FastCallKeywords+0x187) [0x564106718d37]
#29 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x3f5) [0x5641067a32b5]
#30 in /opt/conda/envs/rapids/bin/python(_PyFunction_FastCallKeywords+0x187) [0x564106718d37]
#31 in /opt/conda/envs/rapids/bin/python(+0x181455) [0x564106761455]
#32 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x611) [0x5641067a34d1]
#33 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x255) [0x5641066f9505]
#34 in /opt/conda/envs/rapids/bin/python(_PyObject_FastCallDict+0x5be) [0x5641066faebe]
#35 in /opt/conda/envs/rapids/bin/python(+0x12f071) [0x56410670f071]
#36 in /opt/conda/envs/rapids/bin/python(PyObject_Call+0xb4) [0x5641066fb214]
#37 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x1ccd) [0x5641067a4b8d]
#38 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x7cd) [0x5641066f9a7d]
#39 in /opt/conda/envs/rapids/bin/python(_PyFunction_FastCallKeywords+0x583) [0x564106719133]
#40 in /opt/conda/envs/rapids/bin/python(+0x181455) [0x564106761455]
#41 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x1411) [0x5641067a42d1]
#42 in /opt/conda/envs/rapids/bin/python(+0x192945) [0x564106772945]
#43 in /opt/conda/envs/rapids/bin/python(_PyMethodDef_RawFastCallKeywords+0xe9) [0x564106729789]
#44 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x4568) [0x5641067a7428]
#45 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x7cd) [0x5641066f9a7d]
#46 in /opt/conda/envs/rapids/bin/python(_PyFunction_FastCallKeywords+0x583) [0x564106719133]
#47 in /opt/conda/envs/rapids/bin/python(+0x181455) [0x564106761455]
#48 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x611) [0x5641067a34d1]
#49 in /opt/conda/envs/rapids/bin/python(+0x192945) [0x564106772945]
#50 in /opt/conda/envs/rapids/bin/python(_PyMethodDef_RawFastCallKeywords+0xe9) [0x564106729789]
#51 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x4568) [0x5641067a7428]
#52 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x7cd) [0x5641066f9a7d]
#53 in /opt/conda/envs/rapids/bin/python(_PyFunction_FastCallKeywords+0x583) [0x564106719133]
#54 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x3f5) [0x5641067a32b5]
#55 in /opt/conda/envs/rapids/bin/python(+0x192945) [0x564106772945]
#56 in /opt/conda/envs/rapids/bin/python(_PyMethodDef_RawFastCallKeywords+0xe9) [0x564106729789]
#57 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x4568) [0x5641067a7428]
#58 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x7cd) [0x5641066f9a7d]
#59 in /opt/conda/envs/rapids/bin/python(_PyObject_FastCallDict+0x316) [0x5641066fac16]
#60 in /opt/conda/envs/rapids/bin/python(+0x12f071) [0x56410670f071]
#61 in /opt/conda/envs/rapids/bin/python(PyObject_Call+0xb4) [0x5641066fb214]
#62 in /opt/conda/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x1ccd) [0x5641067a4b8d]
#63 in /opt/conda/envs/rapids/bin/python(+0x180434) [0x564106760434]

image

Environment location: AWS EC2 g4dn.8xlarge
Linux Distro/Architecture: Ubuntu 18.04
CUDA: 10.2
Method of cuDF & cuML install: docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 rapidsai/rapidsai-dev-nightly:0.15-cuda10.1-devel-ubuntu18.04-py3.7

@github-actions
Copy link

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

@github-actions github-actions bot added the stale label Feb 17, 2021
@github-actions
Copy link

This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

@dantegd dantegd mentioned this issue Aug 2, 2021
Open
43 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cjnolet cjnolet

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cjnolet @mike-wendt @hershkoy @DavidEverlaw