cuda-modules: apply lessons learned from cross-compilation attempts #301416
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ofborg
bot
added
the
6.topic: cross-compilation
This was referenced Apr 4, 2024
ConnorBaker
force-pushed
the
feat/cudaPackages-lessons-learned-from-cross-compilation-attempt
branch
from
a688406
to
79c50d7
Compare
Outdated `nixpkgs-review` for 79c50d75eb4593169b52154b8c65d896855b6848
✅
|
ofborg
bot
added
11.by: package-maintainer
SomeoneSerge
reviewed
Apr 4, 2024
|
I'm able to run $ nix run -L --override-input nixpkgs "github:NixOS/nixpkgs/79c50d75eb4593169b52154b8c65d896855b6848" .#nix-cuda-test
Seed set to 42
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Files already downloaded and verified
Files already downloaded and verified
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
-----------------------------------------------
0 | criterion | CrossEntropyLoss | 0
1 | model | ViT | 86.3 M
-----------------------------------------------
86.3 M Trainable params
0 Non-trainable params
86.3 M Total params
345.317 Total estimated model params size (MB)
Epoch 9: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:26<00:00, 9.04it/s, v_num=4, train_loss=2.350, val_loss=2.330]`Trainer.fit` stopped: `max_epochs=10` reached.
Epoch 9: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:28<00:00, 8.82it/s, v_num=4, train_loss=2.350, val_loss=2.330] |
samuela
reviewed
Apr 4, 2024
There was a problem hiding this comment.
Thanks for writing up this documentation @ConnorBaker !
I don't currently have a need for cross-compilation support, so I'm fine continuing to pursue the cudaPackages transition, but I propose that we consult with @kmittman to double check that we're not building ourselves into a corner.
@kmittman I'm not sure if you've already been appraised of the cross-compilation work going on here, but tl;dr it looks like cross-compilation doesn't work with the CUDA redistributable packages. We've been investing in migrating away from the runfile-based CUDA towards the redistributables, but this discovery gives us pause. Any insight you may have would be constructive for us and much appreciated!
SomeoneSerge
reviewed
Apr 4, 2024
SomeoneSerge
reviewed
Apr 4, 2024
SomeoneSerge
reviewed
Apr 4, 2024
SomeoneSerge
reviewed
Apr 4, 2024
pkgs/development/cuda-modules/setup-hooks/auto-add-driver-runpath/default.nix
Outdated
Show resolved
Hide resolved
SomeoneSerge
reviewed
Apr 4, 2024
pkgs/development/cuda-modules/setup-hooks/mark-for-cudatoolkit-root/default.nix
Outdated
Show resolved
Hide resolved
SomeoneSerge
reviewed
Apr 4, 2024
.../development/cuda-modules/setup-hooks/mark-for-cudatoolkit-root/mark-for-cudatoolkit-root.sh
Outdated
Show resolved
Hide resolved
SomeoneSerge
reviewed
Apr 4, 2024
.../development/cuda-modules/setup-hooks/mark-for-cudatoolkit-root/mark-for-cudatoolkit-root.sh
Outdated
Show resolved
Hide resolved
|
ConnorBaker
force-pushed
the
feat/cudaPackages-lessons-learned-from-cross-compilation-attempt
branch
2 times, most recently
from
d2963d7
to
4e84386
Compare
See #302694 (which is stacked on this PR).
I'll try this again later and see if I can get it to work. |
added 4 commits
April 10, 2024 14:14
For more information about *why* this is desirable, see NixOS#45717 and NixOS#27069
…ix deprecation warning
14 tasks
|
@samuela @SomeoneSerge @kmittman thank you all for the feedback on the cross-compilation notes -- I've moved that to #303122 to try to unblock this PR. |
ConnorBaker
force-pushed
the
feat/cudaPackages-lessons-learned-from-cross-compilation-attempt
branch
from
80330cd
to
b1158d6
Compare
|
Important This review was run against commit b1158d6ff287d1f9c726a516ab08330643f67a0a Note Template PR="301416"; \
SYSTEM="aarch64-linux"; \
CUDA_SUPPORT="true"; \
CUDA_CAPABILITIES='[ "7.5" ]'; \
nixpkgs-review pr "${PR:?}" \
--system "${SYSTEM:?}" \
--no-shell \
--checkout commit \
--allow aliases \
--build-args "--max-jobs 1" \
--build-graph nix \
--skip-package-regex 'python3(\d){1,2}Packages\.(pytorch-pfn-extras|ffcv)' \
--skip-package-regex 'python3(\d){1,2}Packages\.(shap|mlflow|optuna)' \
--skip-package-regex 'python3(\d){1,2}Packages\.(apricot-select)' \
--extra-nixpkgs-config "{
allowUnfree = true;
allowBroken = false;
checkMeta = true;
cudaSupport = ${CUDA_SUPPORT:?};
cudaCapabilities = ${CUDA_CAPABILITIES:?};
}"The packages Template log archival command: PR="301416"; \
RUN_NUMBER="13"; \
SYSTEM="aarch64-linux"; \
CUDA_CAPABILITIES="7_5"; \
tar -cvf "${SYSTEM:?}-cap-${CUDA_CAPABILITIES:?}-pr-${PR:?}${RUN_NUMBER:+-$RUN_NUMBER}.tar.zst" \
-I "zstd -T0 --long=31 --ultra -22" \
-C "${HOME:?}/.cache/nixpkgs-review/pr-${PR:?}${RUN_NUMBER:+-$RUN_NUMBER}" \
.Logs are made available at https://drive.google.com/drive/folders/1GgABhwa2ooKeZXMqf5He6PcyClNrE6cQ?usp=share_link ✅
|
pkgs/development/cuda-modules/cuda-library-samples/extension.nix
Outdated
Show resolved
Hide resolved
pkgs/development/cuda-modules/setup-hooks/mark-for-cudatoolkit-root-hook.sh
Outdated
Show resolved
Hide resolved
pkgs/development/cuda-modules/cuda-library-samples/extension.nix
Outdated
Show resolved
Hide resolved
added 11 commits
April 18, 2024 16:31
The setupCudaHook always checks for the existence of `$prefix/nix-support/include-in-cudatoolkit-root`, so we need to be sure it always exists. It isn't populated when `strictDeps` is set, but it must exist.
On NixOS, the the return status is that of the last command executed within the function or script. When we're doing tests immediately before the return, the value you end up returning might not be what you want.
…rs, and enable on Jetson post-11.4
…lify src, break out comments in postPatch
ConnorBaker
force-pushed
the
feat/cudaPackages-lessons-learned-from-cross-compilation-attempt
branch
from
af1ffad
to
6208368
Compare
SomeoneSerge
approved these changes
Apr 18, 2024
| @@ -86,7 +86,7 @@ let | |||
| [ | |||
| (import ../development/cuda-modules/setup-hooks/extension.nix) | |||
| (callPackage ../development/cuda-modules/cuda/extension.nix { inherit cudaVersion; }) | |||
There was a problem hiding this comment.
We could still add your comment about which callPackage this is - this would be the place.
| @@ -1,14 +1,25 @@ | |||
| # shellcheck shell=bash | |||
|
|
|||
| # Should we mimick cc-wrapper's "hygiene"? | |||
| [[ -z ${strictDeps-} ]] || (( "$hostOffset" < 0 )) || return 0 | |||
| (( ${hostOffset:?} == -1 && ${targetOffset:?} == 0)) || return 0 | |||
There was a problem hiding this comment.
Side note: I believe that "in practice" the derivation never sees offsets other than {-1, 0, 1}, but I saw other hooks write < 0 instead of == -1... I really can't say what this change means in the larger scheme of things
Comment on lines
+75
to
+80
| brokenConditions = prevAttrs.brokenConditions // { | ||
| "libnvjitlink missing (CUDA >= 12.0)" = | ||
| !(cudaAtLeast "12.0" -> (libnvjitlink != null && libnvjitlink.lib != null)); | ||
| "libcusparse missing (CUDA >= 12.1)" = | ||
| !(cudaAtLeast "12.1" -> (libcusparse != null && libcusparse.lib != null)); | ||
| }; |
There was a problem hiding this comment.
Just noticed this leaks into the .drv file, affecting the hash
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important
To-do:
cmakeFlagsArrayin the setup hooks does not cause problems by (for example) overriding user-specified values incmakeFlagsbecause of the order of string interpolation in the CMake setup hook.Note
Description of changes
Supersedes #279952 and #275560.
This PR instead adds fixes and updates accumulated over the course of work on #279952 and #275560. Work related specifically to rewriting the setup hooks is covered by #302694.
*Platformthroughstdenvrather than directly frompkgssubstituteInPlaceto--replace-failto fix deprecationcmakeCudaArchitecturesandcmakeCudaArchitecturesStringreturnso it does not use the return value of the previously executed command (which would sometimes cause failures when evaluating bash functions)setupCudaHookwherein attempting to read an empty marker file would failncclandsaxpybrokenConditionsThings done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.