[BUG] dask scheduler not working: dask_cuda/cli.py AttributeError: 'function' object has no attribute 'command'

[pyotr777]

Describe the bug
After installing cudf cannot start dask scheduler.

Steps/Code to reproduce bug

> dask scheduler
Traceback (most recent call last):
  File "/home/user/.local/bin/dask", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/lib/python3.8/site-packages/dask/__main__.py", line 5, in main
    run_cli()
  File "/home/user/.local/lib/python3.8/site-packages/dask/cli.py", line 79, in run_cli
    _register_command_ep(cli, ep)
  File "/home/user/.local/lib/python3.8/site-packages/dask/cli.py", line 56, in _register_command_ep
    command = entry_point.load()
  File "/home/user/.local/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 210, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/user/.local/lib/python3.8/site-packages/dask_cuda/cli.py", line 61, in <module>
    @cuda.command(name="worker", context_settings=dict(ignore_unknown_options=True))
AttributeError: 'function' object has no attribute 'command'

Expected behavior
Dask scheduler works as usual.

Environment overview (please complete the following information)
Environment location: [Bare-metal, Docker, Cloud(specify cloud provider)]

Bare metal, Ubuntu 20.04.2 LTS, Python 3.8.10

Installed with
pip3 install cudf-cu11 --extra-index-url=https://pypi.nvidia.com

Environment details

> ./print_env.sh

<details><summary>Click here to see environment details</summary><pre>

     **git***
     Not inside a git repository

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=20.04
     DISTRIB_CODENAME=focal
     DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
     NAME="Ubuntu"
     VERSION="20.04.2 LTS (Focal Fossa)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 20.04.2 LTS"
     VERSION_ID="20.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=focal
     UBUNTU_CODENAME=focal
     Linux narita 5.4.0-77-generic rapidsai/cudf#86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Thu May 25 17:51:55 2023
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 470.42.01    Driver Version: 470.42.01    CUDA Version: 11.4     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  NVIDIA RTX A6000    On   | 00000000:4B:00.0 Off |                  Off |
     | 30%   32C    P8    22W / 300W |      5MiB / 48685MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |    0   N/A  N/A      1912      G   /usr/lib/xorg/Xorg                  4MiB |
     +-----------------------------------------------------------------------------+

     ***CPU***
     Architecture:                    x86_64
     CPU op-mode(s):                  32-bit, 64-bit
     Byte Order:                      Little Endian
     Address sizes:                   46 bits physical, 57 bits virtual
     CPU(s):                          32
     On-line CPU(s) list:             0-31
     Thread(s) per core:              1
     Core(s) per socket:              16
     Socket(s):                       2
     NUMA node(s):                    2
     Vendor ID:                       GenuineIntel
     CPU family:                      6
     Model:                           106
     Model name:                      Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
     Stepping:                        6
     Frequency boost:                 enabled
     CPU MHz:                         1229.059
     CPU max MHz:                     3500.0000
     CPU min MHz:                     800.0000
     BogoMIPS:                        5800.00
     Virtualization:                  VT-x
     L1d cache:                       1.5 MiB
     L1i cache:                       1 MiB
     L2 cache:                        40 MiB
     L3 cache:                        48 MiB
     NUMA node0 CPU(s):               0-15
     NUMA node1 CPU(s):               16-31
     Vulnerability Itlb multihit:     Not affected
     Vulnerability L1tf:              Not affected
     Vulnerability Mds:               Not affected
     Vulnerability Meltdown:          Not affected
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Not affected
     Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities

     ***CMake***
     /home/peter/.local/bin/cmake
     cmake version 3.26.3

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
     Copyright (C) 2019 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


     ***nvcc***

     ***Python***

     ***Environment Variables***
     PATH                            : /home/peter/.local/bin:/home/peter/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
     LD_LIBRARY_PATH                 :
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    :
     PYTHON_PATH                     :

     conda not found
     ***pip packages***
     /home/peter/.local/bin/pip
     Package                           Version
     --------------------------------- --------------------
     absl-py                           1.0.0
     aiohttp                           3.8.4
     aiosignal                         1.3.1
     alembic                           1.7.7
     anyio                             3.6.2
     apturl                            0.5.2
     argon2-cffi                       21.3.0
     argon2-cffi-bindings              21.2.0
     arrow                             1.2.3
     asttokens                         2.0.5
     async-generator                   1.10
     async-timeout                     4.0.2
     attrs                             19.3.0
     Automat                           0.8.0
     autopage                          0.5.0
     backcall                          0.2.0
     bcrypt                            3.1.7
     beautifulsoup4                    4.11.1
     black                             22.1.0
     bleach                            4.1.0
     blinker                           1.4
     bokeh                             2.4.3
     botorch                           0.8.4
     Brlapi                            0.7.0
     Brotli                            1.0.9
     cachetools                        5.0.0
     certifi                           2019.11.28
     cffi                              1.15.0
     chardet                           3.0.4
     charset-normalizer                2.1.0
     chromedriver-binary               104.0.5112.29.0
     click                             8.0.3
     cliff                             3.10.1
     cloud-init                        21.2
     cloudpickle                       2.2.1
     cmaes                             0.9.1
     cmake                             3.26.3
     cmd2                              2.4.1
     colorama                          0.4.3
     colorlog                          6.6.0
     comm                              0.1.3
     command-not-found                 0.3
     configobj                         5.0.6
     constantly                        15.1.0
     contourpy                         1.0.7
     cryptography                      2.8
     cubinlinker-cu11                  0.3.0.post1
     cuda-python                       11.8.1
     cudf-cu11                         23.4.1
     cuml-cu11                         23.4.1
     cupshelpers                       1.0
     cupy-cuda11x                      11.6.0
     cycler                            0.11.0
     Cython                            0.29.34
     dask                              2023.3.2
     dask-cuda                         23.4.0
     dask-cudf-cu11                    23.4.1
     dbus-python                       1.2.16
     debugpy                           1.6.7
     decorator                         5.1.1
     defer                             1.0.6
     defusedxml                        0.7.1
     distlib                           0.3.6
     distributed                       2023.3.2.1
     distro                            1.4.0
     distro-info                       0.23ubuntu1
     dnspython                         1.16.0
     duplicity                         0.8.12.0
     entrypoints                       0.3
     executing                         0.8.2
     fasteners                         0.14.1
     fastjsonschema                    2.16.3
     fastrlock                         0.8.1
     filelock                          3.12.0
     Flask                             2.1.2
     Flask-Compress                    1.12
     fonttools                         4.29.1
     fqdn                              1.5.1
     frozenlist                        1.3.3
     fsspec                            2023.4.0
     future                            0.18.2
     google-auth                       2.6.3
     google-auth-oauthlib              0.4.6
     gpg                               1.13.1-unknown
     gpytorch                          1.10
     grandalf                          0.8
     greenlet                          1.1.2
     grpcio                            1.44.0
     h11                               0.13.0
     hiplot                            0.1.32
     httplib2                          0.14.0
     hyperlink                         19.0.0
     idna                              2.8
     imageio                           2.27.0
     importlib-metadata                6.5.1
     importlib-resources               5.7.1
     incremental                       16.10.1
     iniconfig                         1.1.1
     ipykernel                         6.23.0
     ipython                           8.12.2
     ipython-genutils                  0.2.0
     ipywidgets                        7.6.5
     isoduration                       20.11.0
     itsdangerous                      2.1.2
     jedi                              0.18.1
     Jinja2                            3.1.2
     joblib                            1.1.0
     jsonpatch                         1.22
     jsonpickle                        3.0.1
     jsonpointer                       2.0
     jsonschema                        4.17.3
     jupyter_client                    8.2.0
     jupyter-console                   6.4.0
     jupyter-contrib-core              0.3.3
     jupyter-contrib-nbextensions      0.7.0
     jupyter_core                      5.3.0
     jupyter-events                    0.6.3
     jupyter-highlight-selected-word   0.2.0
     jupyter-latex-envs                1.4.6
     jupyter-nbextensions-configurator 0.4.1
     jupyter_server                    2.5.0
     jupyter_server_proxy              4.0.0
     jupyter_server_terminals          0.4.4
     jupyterlab-pygments               0.1.2
     jupyterlab-widgets                1.0.2
     keyring                           18.0.1
     kiwisolver                        1.3.2
     language-selector                 0.1
     launchpadlib                      1.10.13
     lazr.restfulclient                0.14.2
     lazr.uri                          1.0.3
     line-profiler                     3.5.1
     linear-operator                   0.4.0
     lit                               16.0.3
     llvmlite                          0.39.1
     locket                            1.0.0
     lockfile                          0.12.2
     louis                             3.12.0
     lxml                              4.7.1
     lz4                               4.3.2
     macaroonbakery                    1.3.1
     Mako                              1.1.0
     Markdown                          3.1.1
     MarkupSafe                        2.1.1
     matplotlib                        3.7.1
     matplotlib-inline                 0.1.3
     mistune                           2.0.5
     monotonic                         1.5
     more-itertools                    4.2.0
     mpmath                            1.3.0
     msgpack                           1.0.5
     multidict                         6.0.4
     multipledispatch                  0.6.0
     mypy-extensions                   0.4.3
     nbclassic                         1.0.0
     nbclient                          0.5.10
     nbconvert                         7.4.0
     nbformat                          5.8.0
     nest-asyncio                      1.5.4
     netgraph                          4.12.8
     netifaces                         0.10.4
     networkx                          3.1
     notebook                          6.5.4
     notebook_shim                     0.2.3
     numba                             0.56.4
     numpy                             1.22.4
     nvidia-cublas-cu11                11.10.3.66
     nvidia-cuda-cupti-cu11            11.7.101
     nvidia-cuda-nvrtc-cu11            11.7.99
     nvidia-cuda-runtime-cu11          11.7.99
     nvidia-cudnn-cu11                 8.5.0.96
     nvidia-cufft-cu11                 10.9.0.58
     nvidia-curand-cu11                10.2.10.91
     nvidia-cusolver-cu11              11.4.0.1
     nvidia-cusparse-cu11              11.7.4.91
     nvidia-nccl-cu11                  2.14.3
     nvidia-nvtx-cu11                  11.7.91
     nvtx                              0.2.5
     oauthlib                          3.1.0
     objprint                          0.2.0
     olefile                           0.46
     opt-einsum                        3.3.0
     optuna                            3.1.1
     outcome                           1.2.0
     packaging                         21.3
     pandas                            1.5.3
     pandocfilters                     1.5.0
     paramiko                          2.6.0
     parso                             0.8.3
     partd                             1.4.0
     pathspec                          0.9.0
     pbr                               5.9.0
     pexpect                           4.6.0
     pick                              1.2.0
     pickleshare                       0.7.5
     Pillow                            9.2.0
     pip                               23.1.2
     pkgutil_resolve_name              1.3.10
     platformdirs                      3.5.1
     plotly                            5.7.0
     pluggy                            1.0.0
     pooch                             1.7.0
     prettytable                       3.3.0
     prometheus-client                 0.13.1
     prompt-toolkit                    3.0.38
     protobuf                          4.21.12
     psutil                            5.9.0
     ptxcompiler-cu11                  0.7.0.post1
     ptyprocess                        0.7.0
     pure-eval                         0.2.2
     py                                1.11.0
     pyarrow                           10.0.1
     pyasn1                            0.4.2
     pyasn1-modules                    0.2.1
     pycairo                           1.16.2
     pycparser                         2.21
     pycrypto                          2.6.1
     pycups                            1.9.73
     Pygments                          2.11.2
     PyGObject                         3.36.0
     pygraphviz                        1.10
     PyHamcrest                        1.9.0
     PyJWT                             1.7.1
     pylibraft-cu11                    23.4.1
     pymacaroons                       0.13.0
     PyNaCl                            1.3.0
     pynvml                            11.4.1
     pyOpenSSL                         19.0.0
     pyparsing                         2.4.6
     pyperclip                         1.8.2
     pyRFC3339                         1.1
     pyro-api                          0.1.2
     pyro-ppl                          1.8.4
     pyrsistent                        0.15.5
     pyserial                          3.4
     PySocks                           1.7.1
     pytest                            7.1.2
     python-apt                        2.0.0+ubuntu0.20.4.5
     python-dateutil                   2.8.2
     python-debian                     0.1.36ubuntu1
     python-json-logger                2.0.7
     pytz                              2021.3
     pyvis                             0.3.2
     pyvista                           0.38.5
     pyxdg                             0.26
     PyYAML                            5.3.1
     pyzmq                             25.0.2
     qtconsole                         5.2.2
     QtPy                              2.0.1
     raft-dask-cu11                    23.4.1
     readchar                          3.0.5
     rectangle-packer                  2.0.1
     reportlab                         3.5.34
     requests                          2.28.1
     requests-oauthlib                 1.3.1
     requests-unixsocket               0.2.0
     rfc3339-validator                 0.1.4
     rfc3986-validator                 0.1.1
     rmm-cu11                          23.4.1
     rsa                               4.8
     scikit-learn                      1.1.1
     scipy                             1.8.0
     scooby                            0.7.1
     screen-resolution-extra           0.0.0
     seaborn                           0.12.2
     SecretStorage                     2.3.1
     selenium                          4.3.0
     Send2Trash                        1.8.0
     service-identity                  18.1.0
     setuptools                        45.2.0
     shap                              0.41.0
     simpervisor                       0.4
     simplejson                        3.16.0
     six                               1.14.0
     sklearn                           0.0
     slicer                            0.0.7
     sniffio                           1.2.0
     sortedcontainers                  2.4.0
     sos                               4.1
     soupsieve                         2.3.2.post1
     SQLAlchemy                        1.4.36
     ssh-import-id                     5.10
     stack-data                        0.1.4
     stevedore                         3.5.0
     sympy                             1.11.1
     systemd-python                    234
     tblib                             1.7.0
     tenacity                          8.0.1
     tensorboard                       2.8.0
     tensorboard-data-server           0.6.1
     tensorboard-plugin-wit            1.8.1
     terminado                         0.13.1
     testpath                          0.5.0
     threadpoolctl                     3.1.0
     tinycss2                          1.2.1
     tomli                             2.0.1
     toolz                             0.12.0
     torch                             2.0.1
     torchaudio                        2.0.2
     torchsummaryX                     1.3.0
     torchvision                       0.15.2
     tornado                           6.3.2
     tqdm                              4.64.0
     traitlets                         5.9.0
     treelite                          3.2.0
     treelite-runtime                  3.2.0
     trio                              0.21.0
     trio-websocket                    0.9.2
     triton                            2.0.0
     Twisted                           18.9.0
     typing_extensions                 4.0.1
     tzdata                            2023.3
     ubuntu-advantage-tools            27.0
     ubuntu-drivers-common             0.0.0
     ucx-py-cu11                       0.31.1
     ufw                               0.36
     unattended-upgrades               0.1
     uri-template                      1.2.0
     urllib3                           1.26.10
     usb-creator                       0.3.7
     virtualenv                        20.23.0
     viztracer                         0.15.3
     vtk                               9.2.6
     wadllib                           1.3.3
     wcwidth                           0.2.5
     webcolors                         1.13
     webencodings                      0.5.1
     websocket-client                  1.5.1
     Werkzeug                          2.1.1
     wheel                             0.34.2
     widgetsnbextension                3.5.2
     wsproto                           1.1.0
     xgboost                           1.7.5
     xkit                              0.0.0
     yapf                              0.32.0
     yarl                              1.9.2
     zict                              3.0.0
     zipp                              3.8.0
     zope.interface                    4.7.1

</pre></details>

Additional context
Add any other context about the problem here.

[bdice]

This seems like an issue in dask-cuda, not cudf. @pentschev Would you be familiar with this?

[quasiben]

This looks really strange. dask scheduler should not be calling into dask-cuda at all. cc @jacobtomlinson

I'm also going to move to the dask-cuda issue tracker

[pentschev]

This is the first time I'm installing RAPIDS/cuDF with pip, but I created a new environment with Python 3.8 installing the packages as follows:

$ pip3 install cudf-cu11 --extra-index-url=https://pypi.nvidia.com
$ pip3 install dask-cuda=23.04

Then ran the scheduler successfully:

$ dask scheduler
2023-05-25 10:22:31,045 - distributed.scheduler - INFO - -----------------------------------------------
2023-05-25 10:22:31,071 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2023-05-25 10:22:31,076 - distributed.scheduler - INFO - State start
2023-05-25 10:22:31,095 - distributed.scheduler - INFO - -----------------------------------------------
2023-05-25 10:22:31,096 - distributed.scheduler - INFO -   Scheduler at:  tcp://10.33.227.163:8786
2023-05-25 10:22:31,096 - distributed.scheduler - INFO -   dashboard at:  http://10.33.227.163:8787/status

@pyotr777 could you try creating a new minimal environment like the above and see if that works?

[quasiben]

I also installed in a new env and was unable to reproduce

[pyotr777]

In clean virtual environment installed as the following:

> pip3 install cudf-cu11 --extra-index-url=https://pypi.nvidia.com

Install completed, but displayed an error:

ERROR: numba 0.56.4 has requirement numpy<1.24,>=1.18, but you'll have numpy 1.24.3 which is incompatible.

I continued installation:

> pip3 install dask-cuda==23.04

...
Requirement already satisfied: numba>=0.54 in ./daskcuda/lib/python3.8/site-packages (from dask-cuda==23.04) (0.56.4)
...

I confirm that dask scheduler starts without errors.
However, cudf is broken.

import cudf

SystemError                               Traceback (most recent call last)
Cell In[1], line 6
      4 from matplotlib import pyplot as plt
      5 import seaborn as sns
----> 6 import cudf

File ~/daskcuda/lib/python3.8/site-packages/cudf/__init__.py:5
      1 # Copyright (c) 2018-2023, NVIDIA CORPORATION.
      3 from cudf.utils.gpu_utils import validate_setup
----> 5 validate_setup()
      7 import cupy
      8 from numba import config as numba_config, cuda

File ~/daskcuda/lib/python3.8/site-packages/cudf/utils/gpu_utils.py:95, in validate_setup()
     85     minor_version = getDeviceAttribute(
     86         cudaDeviceAttr.cudaDevAttrComputeCapabilityMinor, 0
     87     )
     88     warnings.warn(
     89         "A GPU with NVIDIA Pascal™ (Compute Capability 6.0) "
     90         "or newer architecture is required.\n"
     91         f"Detected GPU 0: {device_name}\n"
     92         f"Detected Compute Capability: {major_version}.{minor_version}"
     93     )
---> 95 cuda_runtime_version = runtimeGetVersion()
     97 if cuda_runtime_version < 11000:
     98     # Require CUDA Runtime version 11.0 or greater.
     99     major_version = cuda_runtime_version // 1000

File ~/daskcuda/lib/python3.8/site-packages/rmm/_cuda/gpu.py:86, in runtimeGetVersion()
     76 """
     77 Returns the version number of the current CUDA Runtime instance.
     78 The version is returned as (1000 major + 10 minor). For example,
   (...)
     82 current limitations in cuda-python.
     83 """
     84 # TODO: Replace this with `cuda.cudart.cudaRuntimeGetVersion()` when the
     85 # limitation is fixed.
---> 86 import numba.cuda
     88 major, minor = numba.cuda.runtime.get_version()
     89 return major * 1000 + minor * 10

File ~/daskcuda/lib/python3.8/site-packages/numba/__init__.py:42
     38 from numba.core.decorators import (cfunc, generated_jit, jit, njit, stencil,
     39                                    jit_module)
     41 # Re-export vectorize decorators and the thread layer querying function
---> 42 from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
     43                             get_num_threads, set_num_threads,
     44                             set_parallel_chunksize, get_parallel_chunksize,
     45                             get_thread_id)
     47 # Re-export Numpy helpers
     48 from numba.np.numpy_support import carray, farray, from_dtype

File ~/daskcuda/lib/python3.8/site-packages/numba/np/ufunc/__init__.py:3
      1 # -*- coding: utf-8 -*-
----> 3 from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
      4 from numba.np.ufunc._internal import PyUFunc_None, PyUFunc_Zero, PyUFunc_One
      5 from numba.np.ufunc import _internal, array_exprs

File ~/daskcuda/lib/python3.8/site-packages/numba/np/ufunc/decorators.py:3
      1 import inspect
----> 3 from numba.np.ufunc import _internal
      4 from numba.np.ufunc.parallel import ParallelUFuncBuilder, ParallelGUFuncBuilder
      6 from numba.core.registry import DelayedRegistry

SystemError: initialization of _internal failed without raising an exception

I tried to update numba.

pip3 install --upgrade --no-cache-dir numba
Collecting numba
  Downloading numba-0.57.0-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB)
     |████████████████████████████████| 3.6 MB 27.3 MB/s
Collecting llvmlite<0.41,>=0.40.0dev0
  Downloading llvmlite-0.40.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.1 MB)
     |████████████████████████████████| 42.1 MB 175.5 MB/s
Requirement already satisfied, skipping upgrade: numpy<1.25,>=1.21 in ./daskcuda/lib/python3.8/site-packages (from numba) (1.24.3)
Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < "3.9" in ./daskcuda/lib/python3.8/site-packages (from numba) (6.6.0)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in ./daskcuda/lib/python3.8/site-packages (from importlib-metadata; python_version < "3.9"->numba) (3.15.0)
ERROR: cudf-cu11 23.4.1 has requirement numba<0.57,>=0.56.4, but you'll have numba 0.57.0 which is incompatible.
Installing collected packages: llvmlite, numba
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Uninstalling llvmlite-0.39.1:
      Successfully uninstalled llvmlite-0.39.1
  Attempting uninstall: numba
    Found existing installation: numba 0.56.4
    Uninstalling numba-0.56.4:
      Successfully uninstalled numba-0.56.4
Successfully installed llvmlite-0.40.0 numba-0.57.0

The last installation finished with errors, however, now import cudf works without error, dask scheduler also seems to be working (haven't tested it).

[pentschev]

I don't know exactly why that error happened to you, but since I tried to create a new environment and got no install errors either. I know the cuDF PR to support NumPy>=1.24 was merged yesterday, but it doesn't pick NumPy>=1.24 for me either. To be honest I don't know exactly where the pip packages stand regarding that, hopefully this can be answered by @bdice or @vyasr , but at this point it doesn't seem like a Dask-related issue anymore, and perhaps should be transferred back to cuDF.

[bdice]

ERROR: numba 0.56.4 has requirement numpy<1.24,>=1.18, but you'll have numpy 1.24.3 which is incompatible.

To resolve this, you'll need to use a compatible numpy version, less than 1.24. The cuDF 23.06 release will support numpy 1.24.

[pentschev]

Ah right, that means NumPy>=1.24 was already installed when attempting to install cuDF, I didn't realize that. So a "clean environment" should have Python and PIP only, nothing else, not even NumPy to ensure cuDF will pick the correct supported versions of packages.

[vyasr]

pip's solver is limited and can't always fix issues such as with preinstalled packages. Manually downgrading the numpy version here should fix the issue, though.

[pentschev]

This is most likely automatically resolved in 23.06 and up, tentatively closing but feel free to reopen should this still be a problem.