Update to Legate 24.09 nightlies #163

seberg · 2024-10-11T14:19:45Z

This updates to the nightlies based on Jacobs work (i.e. most of the cmake changes and the larger part of the non-stream related C changes).

A slightly bigger change is that there is now a warning for the stream pool API, so we have to pass through the stream more explicitly.

Locally there is a test failing because legate --cpus 2 thinks that --cpus are passed twice. Let's see, the question will be whether we can just remove it (using the default of 4 cpus probably).

seberg · 2024-10-11T14:38:11Z

OK... This is failing to build in CI due to https://github.com/nv-legate/legate.core.internal/issues/1246 will mark as draft until then.

RAMitchell

Thank you for taking this on @seberg! Looks not too bad in terms of code changes. Will take another look when the CI is working.

jameslamb

Thanks for doing this!

I think your diagnosis of why this is failing (#163 (comment)) is exactly correct.

I left some other small packaging-related suggestions for your consideration.

dependencies.yaml

conda/recipes/legate-boost/conda_build_config.yaml

ci/build_python.sh

seberg · 2024-10-16T08:14:26Z

Just to note: the way that install_info.py is now created, it includes the full local build path to search for the library.
I am not clear that is intentional (I don't think the cunumeric install_info.py includes it), OTOH, maybe it helps with local builds?

seberg · 2024-10-16T10:28:31Z

Ping CI to run (close/reopen).

jameslamb · 2024-10-16T19:35:43Z

/ok to test

seberg · 2024-10-16T20:26:06Z

This is getting somewhere: The test failures seem pretty harmless (one new mypy failure, and the --cpus getting passed twice, see below details at end).

Two notes about the setup:

install_info is now fully generated, and I am not sure it was previously ./ci/run_pytests_gpu.sh works around this by moving directory away to not pick up the source package. I am not sure in-place builds work with this (did they before?) Local builds are fine, the file pops up in-tree.
~~install_info.py seems to hardcode the actual build path. That doesn't seem right for a package.~~ This is true, and a potential problem, but it is not new.

Not sure what this is about. Maybe it is picking up the config of the parent process? Not sure that starting legate from a legate process is a good idea :).

== LEGATE ERROR:
== LEGATE ERROR: Duplicate argument --cpus
== LEGATE ERROR:
Usage: LEGATE [--help] [--version] [--cpus INT] [--gpus INT] [--omps INT] [--ompthreads INT] [--utility INT] [--sysmem INT] [--numamem INT] [--fbmem INT] [--zcmem INT] [--regmem INT] [--eager-alloc-percentage INT] [--profile] [--spy] [--logging STRING] [--logdir STRING] [--log-to-file] [--freeze-on-error]
Optional arguments:
  -h, --help                    shows help message and exits 
  -v, --version                 prints version information and exits 
Legate arguments (detailed usage):
  --cpus INT                    number of CPU's to reserve, must be >=0 [default: -1]
  --gpus INT                    number of GPU's to reserve, must be >=0 [default: -1]
  --omps INT                    number of OpenMP processors to reserve, must be >=0 [default: -1]
  --ompthreads INT              number of OpenMP threads to use, must be >=0 [default: -1]
  --utility INT                 number of utility processors to reserve, must be >=0 [default: 2]
  --sysmem INT                  size (in megabytes) of system memory to reserve [default: -1]
  --numamem INT                 size (in megabytes) of NUMA memory to reserve [default: -1]
  --fbmem INT                   size (in megabytes) of GPU (or "frame buffer") memory to reservef [default: -1]
  --zcmem INT                   size (in megabytes) of zero-copy GPU memory to reserve [default: 128]
  --regmem INT                  size (in megabytes) of NIC-registered memory to reserve [default: 0]
  --eager-alloc-percentage INT  percentage of eager allocation [default: 50]
  --profile                     whether to enable Legion runtime profiling 
  --spy                         whether to enable Legion spy 
  --logging STRING              comma separated list of loggers to enable and their level, e.g. legate=3,foo=0,bar=5 [default: ""]
  --logdir STRING               directory to emit logfiles to [default: ""]
  --log-to-file                 wether to save logs to file 
  --freeze-on-error             whether to pause the program on first error 
test_examples.py::test_benchmark FAILED

seberg · 2024-10-16T20:49:32Z

Hmmmmm, had to actually cancel the GPU job, it was still running after an hour, which I don't think is correct... hanging at:

EDIT: Hmmm, no. There are more tests later, I just missed it between all the warnings. ~~So maybe it is working, but just slow for some reason.~~ It's working, but the Neural net tests are ~10x slower than previously

models/test_nn.py::test_nn[float32-0.0-hidden_layer_sizes1-1] [0 - 7f0788d16740]    0.000000 {5}{numa}: mems_allowed: ret=-1 errno=1 mask= count=64
[0 - 7f056827c740]  698.580191 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 0 of task legateboost::BuildNNTask (UID 2743741) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1116
[0 - 7f056827c740]  698.580213 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 1 of task legateboost::BuildNNTask (UID 2743741) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1116
[0 - 7f0568288740]  698.647548 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 0 of task legateboost::BuildNNTask (UID 2743862) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1116
[0 - 7f0568288740]  698.647563 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 1 of task legateboost::BuildNNTask (UID 2743862) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1116
[0 - 7f0568288740]  698.714261 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 0 of task legateboost::BuildNNTask (UID 2743980) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1116
[0 - 7f0568288740]  698.714276 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 1 of task legateboost::BuildNNTask (UID 2743980) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1116
[0 - 7f0568288740]  698.714285 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 2 of task legateboost::BuildNNTask (UID 2743980) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:
http://legion.stanford.edu/messages/warning_code.html#warning_code_1116
[0 - 7f0568288740]  698.714292 {4}{runtime}: [warning 1116] LEGION WARNING: Ignoring request by mapper legate.core on Node 0 to check for collective usage for region requirement 5 of task legateboost::BuildNNTask (UID 2743980) because region requirement has writing privileges. (from file /tmp/conda-croot/legate/work/arch-conda/skbuild_core/_deps/legion-src/runtime/legion/legion_tasks.cc:819)
For more information see:

<similar warnings repeat>

This updates code for 24.09 compatibility based on Jacobs start. Additional/larger changes: * legate::cuda::StreamPool::get_stream_pool().get_stream(); is deprecated so use context.get_task_stream() and rewire things as needed. * We are always including the label/experimental channel. * The `install_info.py` file generation seem slightly different, so the test scripts now move into the test folder and then run the tests. (This could maybe be improved.) This has no effect locally, though. Signed-off-by: Sebastian Berg <sebastianb@nvidia.com>

RAMitchell · 2024-10-23T09:29:13Z

The reason for the slowness of the GPU tests is that they are actually the CPU tests :)

legate-boost/ci/test_python_gpu.sh

Line 13 in c42e0fa

./ci/run_pytests_cpu.sh

@jameslamb a typo?

seberg · 2024-10-23T09:38:27Z

The reason for the slowness of the GPU tests is that they are actually the CPU tests :)

Let me fix that, although I am confused. I thought the new version of legate would guess at using the GPU if anything!?

Signed-off-by: Sebastian Berg <sebastianb@nvidia.com>

seberg · 2024-10-23T09:41:52Z

Well, let's see. Maybe I got the tags wrong also, and we are actually compiling a CPU only version for both or so...

jameslamb · 2024-10-23T14:13:42Z

The reason for the slowness of the GPU tests is that they are actually the CPU tests :)

legate-boost/ci/test_python_gpu.sh

Line 13 in c42e0fa

./ci/run_pytests_cpu.sh

@jameslamb a typo?

ah!!! Yes absolutely a typo! Sorry about that, thank you for fixing it.

If this PR is going to go on much longer, I recommend just putting up a separate one with that change to get it onto main.

seberg · 2024-10-23T14:17:35Z

The CPU tests are taking 43 minutes right now. Maybe that is actually acceptable for the moment?

RAMitchell · 2024-10-23T14:39:10Z

Its acceptable for now, although lets not leave it for too long. From my perspective this PR is fine. @jameslamb will this interfere with your packaging work if we merged it?

jameslamb · 2024-10-23T14:42:51Z

conda/recipes/legate-boost/meta.yaml

-    - cunumeric {{ legate_version }}
-    - legate-core {{ legate_version }}
+    - cunumeric {{ legate_version }} =*_gpu
+    - legate {{ legate_version }} =*_gpu


oh hey great! I'm glad they now added _gpu to the build strings for legate / cunumeric. When I'd asked about that before, the folks maintaining legate / cunumeric were against it: https://github.com/nv-legate/legate.core.internal/pull/1035#discussion_r1702268130

jameslamb

From my perspective this PR is fine. @jameslamb will this interfere with your packaging work if we merged it?

Thanks for the @. I totally support merging this... focusing on just supporting legate / cunumeric 24.09 will help the packaging work towards publishing a legate-boost 24.09.

seberg · 2024-10-23T19:11:09Z

I guess I'll put it in then, thanks! 🤞 that CI will be stable enough (and that slowdown tracked down soon!).

This comment was marked as resolved.

Sign in to view

seberg force-pushed the legate-24.09 branch from a761d4a to c99d2e5 Compare October 11, 2024 14:21

seberg marked this pull request as draft October 11, 2024 14:38

RAMitchell reviewed Oct 11, 2024

View reviewed changes

jameslamb self-requested a review October 11, 2024 15:23

jameslamb reviewed Oct 11, 2024

View reviewed changes

dependencies.yaml Outdated Show resolved Hide resolved

dependencies.yaml Outdated Show resolved Hide resolved

conda/recipes/legate-boost/conda_build_config.yaml Outdated Show resolved Hide resolved

ci/build_python.sh Show resolved Hide resolved

seberg force-pushed the legate-24.09 branch 3 times, most recently from 226e78b to a0981da Compare October 15, 2024 15:08

seberg force-pushed the legate-24.09 branch 2 times, most recently from 69dcf7f to 889ab86 Compare October 16, 2024 10:18

seberg closed this Oct 16, 2024

seberg reopened this Oct 16, 2024

seberg force-pushed the legate-24.09 branch from 895af66 to 889ab86 Compare October 16, 2024 11:59

seberg force-pushed the legate-24.09 branch from e6d4a37 to 7f616fe Compare October 17, 2024 13:09

seberg marked this pull request as ready for review October 17, 2024 15:48

magnatelee mentioned this pull request Oct 21, 2024

Fix for the one pool scheme #170

Draft

CI: Fix typo to ensure we use a GPU in GPU tests

6fb395d

Signed-off-by: Sebastian Berg <sebastianb@nvidia.com>

RAMitchell approved these changes Oct 23, 2024

View reviewed changes

jameslamb reviewed Oct 23, 2024

View reviewed changes

jameslamb self-requested a review October 23, 2024 14:43

jameslamb approved these changes Oct 23, 2024

View reviewed changes

seberg merged commit 6c8a1cb into rapidsai:main Oct 23, 2024
10 checks passed

seberg deleted the legate-24.09 branch October 23, 2024 19:11

jameslamb mentioned this pull request Oct 23, 2024

start v24.09.00 development [skip ci] #171

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Legate 24.09 nightlies #163

Update to Legate 24.09 nightlies #163

seberg commented Oct 11, 2024

This comment was marked as resolved.

seberg commented Oct 11, 2024 •

edited

Loading

RAMitchell left a comment

jameslamb left a comment

seberg commented Oct 16, 2024

seberg commented Oct 16, 2024

jameslamb commented Oct 16, 2024

seberg commented Oct 16, 2024 •

edited

Loading

seberg commented Oct 16, 2024 •

edited

Loading

RAMitchell commented Oct 23, 2024

seberg commented Oct 23, 2024

seberg commented Oct 23, 2024

jameslamb commented Oct 23, 2024

seberg commented Oct 23, 2024

RAMitchell commented Oct 23, 2024

jameslamb Oct 23, 2024

jameslamb left a comment

seberg commented Oct 23, 2024

Update to Legate 24.09 nightlies #163

Update to Legate 24.09 nightlies #163

Conversation

seberg commented Oct 11, 2024

This comment was marked as resolved.

seberg commented Oct 11, 2024 • edited Loading

RAMitchell left a comment

Choose a reason for hiding this comment

jameslamb left a comment

Choose a reason for hiding this comment

seberg commented Oct 16, 2024

seberg commented Oct 16, 2024

jameslamb commented Oct 16, 2024

seberg commented Oct 16, 2024 • edited Loading

seberg commented Oct 16, 2024 • edited Loading

RAMitchell commented Oct 23, 2024

seberg commented Oct 23, 2024

seberg commented Oct 23, 2024

jameslamb commented Oct 23, 2024

seberg commented Oct 23, 2024

RAMitchell commented Oct 23, 2024

jameslamb Oct 23, 2024

Choose a reason for hiding this comment

jameslamb left a comment

Choose a reason for hiding this comment

seberg commented Oct 23, 2024

seberg commented Oct 11, 2024 •

edited

Loading

seberg commented Oct 16, 2024 •

edited

Loading

seberg commented Oct 16, 2024 •

edited

Loading