Consolidate GPU Error Checking Function #350
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bcaddy
force-pushed
the
dev-iss286and296
branch
2 times, most recently
from
5d4941e
to
42d17ef
Compare
Consolodate all GPU error checking functions and macros into one overloaded function with one overload for CUDA/HIP errors and one for CUFFT/HIPFFT errors. That one doesn't use any macros and supports all the usual usage modes. It does utilize the `experimental::source_location` class. That class is supported on all compilers that we use or expect others to use but if it doesn't work for you then commenting out the relevant lines should be sufficient. Replaced all calls to `CHECK`, `CudaSafeCall`, `CudaCheckError`, and `gpErrchk` with `GPU_Error_Check`.
Some already had error checks on the next line but now they're all wrapped as is standard in the rest of the code
The CUDA_ERROR_CHECK macro that turns on error checking has been deprecated in favor of the new DISABLE_GPU_ERROR_CHECKING macro which disable error checking. Error checking is now on by default unless compiled with the DISABLE_GPU_ERROR_CHECKING macro.
bcaddy
force-pushed
the
dev-iss286and296
branch
from
42d17ef
to
cdbed5d
Compare
mabruzzo
approved these changes
Dec 8, 2023
|
Yep, everywhere that the implicit sync seemed like it might be relevant were on functions that already contain an implicit sync (like moving or allocating). Since we only use 1 GPU stream there's an implicit sync between all kernels. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GPU Error Checking
I replaced the 3 macros and 7 functions for GPU error checking with a single overloaded function; one overload for CUDA/HIP checking and one for CUFFT/HIPFFT checking. The function supports wrapping a CUDA call or being called with no arguments to check the latest error.
I also added error checking to some
cudaMallocsthat were missing them or used them in a non-standard way.The other major change is the deprecation of the
CUDA_ERROR_CHECKmacro. Now error checking is on by default and can be disabled with the newDISABLE_GPU_ERROR_CHECKINGmacro.This should resolve Issue #286 and possibly #296 as well, subject to discussion in that issue.