What versions of CUDA & GCC are required to compile Marian-nmt?

[Edward205]

I attempted to compile marian-nmt on my Arch installation, which is using GCC 13.1.1 and CUDA 12.1. However, the compilation process failed. I followed these steps:

1. Cloned the marian-nmt repository into a folder named "marian" on my desktop.
2. Tried to install marian-nmt using the instructions provided, but encountered errors.
   Compilation process log

Unfortunately, I'm unable to downgrade GCC on my Arch installation. As a workaround, I created a new partition and installed Ubuntu 18.04. However, I encountered issues with PGP signing of the CUDA 10 packages during the installation.

I consulted the compatibility table available at marian-nmt/marian-dev#526 and discovered that I'm unable to install any CUDA or GCC version that matches the tested configurations.

Could you please provide information on the latest tested versions of GCC and CUDA that are known to work with marian-nmt? Thank you.

[snukky]

Maybe this one https://gist.github.com/ax3l/9489132? It seems CUDA 12 doesn't support GCC version higher than 12.2.1.

[Edward205]

@snukky Downgrading to GCC 12.2.1 and retrying to compile, it did get further but failed with these errors:

[ 58%] Building CXX object src/CMakeFiles/marian_cuda.dir/tensors/gpu/prod.cpp.o
[ 59%] Building CXX object src/CMakeFiles/marian_cuda.dir/tensors/gpu/prod_sparse.cpp.o
In file included from /home/edward/Desktop/marian/marian/src/common/definitions.h:5,
                 from /home/edward/Desktop/marian/marian/src/tensors/tensor.h:3,
                 from /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.h:3,
                 from /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:9:
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTyped(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = float; ComputeType = float]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:449:1:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:24:23: error: pointer used after ‘void operator delete(void*, std::size_t)’ [-Werror=use-after-free]
   24 |     if(x != 0 && --x->references_ == 0) {    \
      |                    ~~~^~~~~~~~~~~
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTyped(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = float; ComputeType = float]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:448:18:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:25:14: note: call to ‘void operator delete(void*, std::size_t)’ here
   25 |       delete x;                              \
      |              ^
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTypedLegacy(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = float; ComputeType = float]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:550:1:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:24:23: error: pointer used after ‘void operator delete(void*, std::size_t)’ [-Werror=use-after-free]
   24 |     if(x != 0 && --x->references_ == 0) {    \
      |                    ~~~^~~~~~~~~~~
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTypedLegacy(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = float; ComputeType = float]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:549:18:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:25:14: note: call to ‘void operator delete(void*, std::size_t)’ here
   25 |       delete x;                              \
      |              ^
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTypedLegacy(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = __half; ComputeType = float]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:550:1:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:24:23: error: pointer used after ‘void operator delete(void*, std::size_t)’ [-Werror=use-after-free]
   24 |     if(x != 0 && --x->references_ == 0) {    \
      |                    ~~~^~~~~~~~~~~
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTypedLegacy(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = __half; ComputeType = float]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:549:18:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:25:14: note: call to ‘void operator delete(void*, std::size_t)’ here
   25 |       delete x;                              \
      |              ^
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTyped(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = __half; ComputeType = __half]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:449:1:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:24:23: error: pointer used after ‘void operator delete(void*, std::size_t)’ [-Werror=use-after-free]
   24 |     if(x != 0 && --x->references_ == 0) {    \
      |                    ~~~^~~~~~~~~~~
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
In function ‘void marian::intrusivePtrRelease(MemoryPiece*)’,
    inlined from ‘IntrusivePtr<T>::~IntrusivePtr() [with T = marian::MemoryPiece]’ at /home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:66:26,
    inlined from ‘void marian::gpu::ProdBatchedTyped(marian::Tensor, marian::Ptr<marian::Allocator>, marian::Tensor, marian::Tensor, bool, bool, ComputeType, ComputeType) [with ElementType = __half; ComputeType = __half]’ at /home/edward/Desktop/marian/marian/src/tensors/gpu/prod.cpp:448:18:
/home/edward/Desktop/marian/marian/src/common/intrusive_ptr.h:25:14: note: call to ‘void operator delete(void*, std::size_t)’ here
   25 |       delete x;                              \
      |              ^
/home/edward/Desktop/marian/marian/src/tensors/memory_piece.h:14:3: note: in expansion of macro ‘ENABLE_INTRUSIVE_PTR’
   14 |   ENABLE_INTRUSIVE_PTR(MemoryPiece)
      |   ^~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [src/CMakeFiles/marian_cuda.dir/build.make:160: src/CMakeFiles/marian_cuda.dir/tensors/gpu/prod.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:392: src/CMakeFiles/marian_cuda.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

[antot]

I got the same error compiling Marian (v 1.12.0) with GCC 12.2, using Boost/1.81.0-GCC-12.2.0, CMake/3.24.3-GCCcore-12.2.0, CUDA/12.1.1:
Software/git/marian_1_12_0/src/common/intrusive_ptr.h:24:23: error: pointer used after ‘void operator delete(void*, std::size_t)’ [-Werror=use-after-free]
24 | if(x != 0 && --x->references_ == 0) {

However, it compilled successfully with GCC 11.3, using Boost/1.79.0-GCC-11.3.0, CMake/3.24.3-GCCcore-11.3.0, CUDA/12.1.1