CUDA runtime error - open3D v0.14.1 #4679
Comments
|
Quadro M1000M is an old card and I suspect One potential fix is to replace all the functions with |
theNded
added
build/install
|
Hi @theNded The denseSlam is now running until a point where I do face another error: an out of memory error It always failed to this 926th RGB-D image. |
|
Quadro M1000M only has 4G GPU memory, so there are not many things we can do with it. One potential change is to increase the voxel size by a factor of say 2, but it will sacrifice the tracking and reconstruction quality. |
|
Ok then it is all solved! |
|
Hi, @theNded I would like to re-open this issue as there are new findings about it. To recap the Digging in the CUDA documentation we can find out that the Stream Ordered Memory Allocator is not available on all NVIDIA GPUs and that support should be verified at runtime, see https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY__POOLS.html In practical terms this would mean that the compile-time check on the driver version performed in Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp Lines 22 to 27 in 59792c2 if (device.cudaSupportsMemoryPools()) {
OPEN3D_CUDA_CHECK(cudaMallocAsync(static_cast<void**>(&ptr), byte_size,
cuda::GetStream()));
} else {
OPEN3D_CUDA_CHECK(cudaMalloc(static_cast<void**>(&ptr), byte_size));
}and the implementation of int driverVersion = 0;
int deviceSupportsMemoryPools = 0;
OPEN3D_CUDA_CHECK(cudaDriverGetVersion(&driverVersion));
if (driverVersion >= 11020) { // avoid invalid value error in cudaDeviceGetAttribute
OPEN3D_CUDA_CHECK(cudaDeviceGetAttribute(&deviceSupportsMemoryPools, cudaDevAttrMemoryPoolsSupported, device));
}
return !!deviceSupportsMemoryPools;I'll try to propose a patch for this, but if someone more familiar with the Open3D codebase wants to anticipate me, please go ahead. Thank you, Antonio |
… time (isl-org#4679) Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11020, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time.
Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11020, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time.
Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11020, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time.
|
Pushed a tentative fix to #6440 |
Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11020, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time.
Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11020, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time.
Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11.2, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time. Still keep the compile time check to support building with CUDA versions older than 11.2.
Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11.2, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time. Still keep the compile time check to support building with CUDA versions older than 11.2.
Some CUDA GPUs, like the Quadro M3000M don't support Memory Pools operations like cudaMallocAsync/cudaFreeAsync even on driver versions newer than 11.2, and this can result in errors like: CUDA runtime error: operation not supported So check for support at runtime instead of compile time. Still keep the compile time check to support building with CUDA versions older than 11.2.
Checklist
masterbranch).Describe the issue
With the following configuration:
I got a runtime error, running the DenseSlam.cpp when I set
"--device CUDA:0"but everything works fine when I use"--device CPU:0"The error is the following
[Open3D INFO] Using device: CUDA:0 terminate called after throwing an instance of 'std::runtime_error' what(): [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /home/ubuntu/Work/Projects/handheld-scanning-prototype/Open3DBox/build/open3d/src/external_open3d/cpp/open3d/core/CUDAUtils.cpp:301: /home/ubuntu/Work/Projects/handheld-scanning-prototype/Open3DBox/build/open3d/src/external_open3d/cpp/open3d/core/MemoryManagerCUDA.cpp:43 CUDA runtime error: operation not supportedI do not understand what is the issue, since testing my cuda install I've run the
deviceQuerycuda application which outputs me the followingbin/x86_64/linux/release/deviceQuery Starting...CUDA Device Query (Runtime API) version (CUDART static linking)Detected 1 CUDA Capable device(s)Device 0: "Quadro M1000M"CUDA Driver Version / Runtime Version 11.6 / 11.6....deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.6, CUDA Runtime Version = 11.6, NumDevs = 1 Result = PASSCould you please provide any support to solve my issue?
Also joining my cmake file to build open3D
Steps to reproduce the bug
Error message
[Open3D INFO] Using device: CUDA:0 terminate called after throwing an instance of 'std::runtime_error' what(): [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /home/ubuntu/Work/Projects/handheld-scanning-prototype/Open3DBox/build/open3d/src/external_open3d/cpp/open3d/core/CUDAUtils.cpp:301: /home/ubuntu/Work/Projects/handheld-scanning-prototype/Open3DBox/build/open3d/src/external_open3d/cpp/open3d/core/MemoryManagerCUDA.cpp:43 CUDA runtime error: operation not supportedExpected behavior
Running DenseSLAM without crashing it is the case when running with the flag "--device CPU:0"
Open3D, Python and System information
Additional information
The text was updated successfully, but these errors were encountered: