Add dependencies.yaml for rapids-dependency-file-generator [skip gpuci]

[beckernick]

This PR:

• Adds a dependencies.yaml file to auto-generate development conda environment files via the rapids-dependency-file-generator
• Adds the generated environment files. I believe these will be helpful in the review, even if we remove them before merging.

[beckernick]

I believe this is now ready for initial review. I've marked several dependencies for which I have some open questions with comments in dependencies.yaml.

The questions relate to: 1) build-time or conda package dependencies that are not explicitly included in the current development environment (such as Cython and others); and 2) dependencies for which I'm unsure if we want to include in a generated pip requirements file

[beckernick]

This is the diff of the (alphabetically sorted) original 11.5 file and the (sorted) auto-generated file. Note that the bulk of the differences are due to the inclusion of build/runtime dependencies not currently specified in the original yamls but noted in BUILD.md and our runtime conda package requirements. The auto-generated files also change the channel ordering for the CTK 11.2 file to match the others (discussed offline).

sort cuml_dev_cuda11.5.yml > sorted_cuml_dev_cuda11.5.yml
sort all_cuda-115_arch-x86_64.yaml > sorted_all_cuda-115_arch-x86_64.yaml
diff sorted_all_cuda-115_arch-x86_64.yaml sorted_cuml_dev_cuda11.5.yml

1,6c1,11
<   - git+https://github.com/dask/dask-glm@main
<   - git+https://github.com/scikit-learn-contrib/hdbscan.git@master
<   - sparse
< # This file is generated by `rapids-dependency-file-generator`.
< # To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
< - cmake>=3.23.1
---
>
>
>     - git+https://github.com/dask/dask-glm@main
>     - git+https://github.com/scikit-learn-contrib/hdbscan.git@master
>     - sparse
> # To install different versions of packages contained in those meta packages,
> # conda remove --force rapids-build-env rapids-notebook-env rapids-doc-env
> # https://docs.rapids.ai/maintainers/depmgmt/
> # it is recommended to remove those meta packages (without removing the actual
> # packages contained in the environment) first with:
> # rapids-build-env, notebook-env and doc-env are defined in
8c13
< - cuda-python>=11.7.1,<12.0
---
> - cuda-python >=11.7.1,<12.0
11,12d15
< - cupy>=7.8.0,<12.0.0a0
< - cython>=0.29,<0.30
21,22d23
< - gcc_linux-64=9.*
< - joblib>=0.11
28d28
< - nccl>=2.9.9
30d29
< - nvcc_linux-64=11.5
35d33
< - python>=3.8,<3.10
48d45
< - sysroot_linux-64==2.17
56c53
< name: all_cuda-115_arch-x86_64
---
> name: cuml_dev
(dependency-generator) nickbecker-mlt:environments nickbecker$

[codecov-commenter]

Codecov Report

Base: 79.38% // Head: 79.44% // Increases project coverage by +0.05% 🎉

Coverage data is based on head (3aa92ea) compared to base (dd197b9).
Patch has no changes to coverable lines.

Additional details and impacted files

@@               Coverage Diff                @@
##           branch-22.12    #5003      +/-   ##
================================================
+ Coverage         79.38%   79.44%   +0.05%     
================================================
  Files               184      184              
  Lines             11698    11698              
================================================
+ Hits               9286     9293       +7     
+ Misses             2412     2405       -7     

Flag	Coverage Δ
dask	45.93% <ø> (ø)
non-dask	68.97% <ø> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
python/cuml/metrics/pairwise_kernels.py	79.83% <0.00%> (+5.64%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[dantegd]

Small thing to avoid the CMake 3.25 bug

[beckernick]

Reminder for us to remove these comments once we decide whether to keep/remove these dependencies

[beckernick]

Running into build slowdowns locally with the generated files vs. the existing conda files. Will do a bit more investigation of the additional dependencies now made explicit.

[beckernick]

Build is not slower in the new dev environment. Original and new conda environments both take about 36 minutes on my workstation. I was thrown by seeing the following when building in the new environment, but it turns out I also see it in the old one:

-- Trying "Ninja" generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /home/nicholasb/NVIDIA/cuml/python/_skbuild/linux-x86_64-3.9/cmake-build
  Command:
    cmake /home/nicholasb/NVIDIA/cuml/python -G Ninja -DCMAKE_INSTALL_PREFIX:PATH=/home/nicholasb/NVIDIA/cuml/python/_skbuild/linux-x86_64-3.9/cmake-install -DPYTHON_EXECUTABLE:FILEPATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/bin/python -DPYTHON_VERSION_STRING:STRING=3.9.14 -DPYTHON_INCLUDE_DIR:PATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/include/python3.9 -DPYTHON_LIBRARY:FILEPATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib/libpython3.9.so -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib/python3.9/site-packages/skbuild/resources/cmake -DCMAKE_LIBRARY_PATH=/home/nicholasb/NVIDIA/cuml/cpp/build -DCMAKE_MESSAGE_LOG_LEVEL=WARNING -DFIND_CUML_CPP=ON -DCMAKE_BUILD_TYPE:STRING=Release

-- Configuring done
CMake Warning at _skbuild/linux-x86_64-3.9/cmake-build/_deps/rapids-cmake-src/rapids-cmake/cython/create_modules.cmake:109 (add_library):
  Cannot generate a safe runtime search path for target common_base because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcudart.so.11.0] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcurand.so.10] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcusolver.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcublas.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcublasLt.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64
    runtime library [libcusparse.so.11] in /home/nicholasb/miniconda3/envs/all_cuda-115_arch-x86_64/lib may be hidden by files in:
      /usr/local/cuda/lib64

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  cuml/common/CMakeLists.txt:29 (rapids_cython_create_modules)

[cjnolet]

@ajschmidt8 are you okay pushing this 23.02?

[ajschmidt8]

@ajschmidt8 are you okay pushing this 23.02?

Yes. I will update the base branch

[dantegd]

@gpucibot merge