Neko on LUMI-G

[njansson]

To compile Neko on LUMI-G

(For a general guide on how to build Neko for AMD GPUs please see: #585)

• Make sure have to have the following modules loaded:
  • The Cray software stack (ml CrayEnv)
  • Cray Accelerator Target (ml craype-accel-amd-gfx90a)
  • ROCm Toolkit (ml rocm)
• Two different compiler combinations can be used:

1. Configure Neko to use Cray Fortran + HIPCC and device aware MPI :

./configure  FC=ftn CC=cc --with-hip HIP_HIPCC_FLAGS="-O3 --offload-arch=gfx90a" --enable-device-mpi

2. Configure Neko to use Cray Fortran + CRAY C++ and device aware MPI :

./configure  FC=ftn CC=cc --with-hip HIPCC=CC HIP_HIPCC_FLAGS="-O3 -x hip --offload-arch=gfx90a" --enable-device-mpi

• Build using make && make install
• Make sure to set MPICH_GPU_SUPPORT_ENABLED=1 in the shell/jobscript before running neko (otherwise device aware MPI will cause a segmentation fault), if everything went well backend type should be reported as HIP (but currently not the device type...).

    _  __  ____  __ __  ____ 
   / |/ / / __/ / //_/ / __ \
  /    / / _/  / ,<   / /_/ /
 /_/|_/ /___/ /_/|_|  \____/ 
 
 (version: 0.4.99)
 (build: 2022-10-04 on x86_64-pc-linux-gnu using cray)
 
  
 -------Job Information-------- 
 Start time: 23:01 / 2022-10-04
 Running on: 32 MPI ranks
 Bcknd type: Accelerator (HIP)
 Dev. name :
 Real type : double precision

[ronithstanly]

After the latest Lumi summer updates the following works:

ml CrayEnv cpe/23.09 craype-accel-amd-gfx90a rocm

./configure MPIFC=ftn MPICC=cc MPICXX=CC CFLAGS="-O3" FCFLAGS="-O3" --prefix=/path/to/neko_install HIPCC=hipcc FC=ftn CC=cc --with-hip=${ROCM_PATH} HIP_HIPCC_FLAGS="-O3 --offload-arch=gfx90a" --enable-device-mpi

make && make install