Get code building with IBM XL on Power 9

[AndrewGaspar]

PR Summary

This change gets the code building with IBM XL 16.1.1.7. Only required a couple small CMake changes.

The tests currently fail due to an issue with IBM XL when used with Catch2. When the fix to Catch2 is merged, I will update our Catch2 version: catchorg/Catch2#1907

PR Checklist

• Code passes cpplint
• New features are documented.
• Adds a test for any bugs fixed. Adds tests for new features.

[AndrewGaspar]

@Yurlungur @JoshuaSBrown @forrestglines Could one of you take a look at this?

[nmsriram]

@AndrewGaspar if this is meant to be a temporary workaround till Catch2 gets fixed, could you please capture that in a comment in the file? That way future generations will not wonder why the lines are there in the first place.

[AndrewGaspar]

This is not a temporary workaround for Catch2 - these lines a necessary on their own for XL to work. Additionally, Catch2 needs an update. However, I discovered it’s literally only broken in the 2.11.3 release.

[nmsriram]

OK, if that's the case, do you need to check more than one string for the compiler? Cmake appears to use either XL or XLCLANG for the XL compiler (CMakeFiles/3.16.2/CompilerIdC/CMakeCCompilerId.c). Do you need it for the XLCLANG compiler string as well? Perhaps MATCHES "^XL" instead of STEQUAL "XL" if this is the case.

[JoshuaSBrown]

Why is "qxflag=disable__cplusplustOverride" needed, I'm having trouble finding what it is used for?

[AndrewGaspar]

See the note at the bottom of this page: https://www.ibm.com/support/knowledgecenter/SSXVZZ_16.1.1/com.ibm.xlcpp1611.lelinux.doc/language_ref/standard_features.html

[jlippuner]

I'm trying to compile this on Darwin, but CMake is complaining:

[jlippuner@cn2025 build_xl]$ CXX=xlc CC=xlc cmake ..                                                                                                         
-- The C compiler identification is XL 16.1.1.7
-- The CXX compiler identification is XL 16.1.1.7
-- Check for working C compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc
-- Check for working C compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc
-- Check for working CXX compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
/home/jlippuner/bin/clang-format-8: /home/jlippuner/bin/clang-format-8: cannot execute binary file
CMake Warning at cmake/Format.cmake:36 (message):
  Couldn't determine clang-format version.  clang-format 8.0 is required -
  results on other versions may not be stable
Call Stack (most recent call first):
  CMakeLists.txt:29 (include)


-- Setting build type to 'RelWithDebInfo' as none was specified.
CMake Warning (dev) at CMakeLists.txt:54 (find_package):
  Policy CMP0074 is not set: find_package uses <PackageName>_ROOT variables.
  Run "cmake --help-policy CMP0074" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  Environment variable MPI_ROOT is set to:

    /projects/opt/ppc64le/p9/openmpi/4.0.3-xl_xlc-16.1.1.7-xlf-16.1.1.7

  For compatibility, CMake is ignoring the variable.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found MPI_CXX: /projects/opt/ppc64le/p9/openmpi/4.0.3-xl_xlc-16.1.1.7-xlf-16.1.1.7/lib/libmpi.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1") found components:  CXX 
-- Found OpenMP_CXX: -qsmp=omp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5") found components:  CXX 
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /home/jlippuner/progs/spack/opt/spack/linux-rhel7-power9le/xl-16.1/hdf5-1.10.5-ttt5wbwilmn6pra5wfsztz6k62sjojys/lib/libhdf5.so;/home/jlippuner/progs/spack/opt/spack/linux-rhel7-power9le/xl-16.1/zlib-1.2.11-quasq73tmp3kiadcuq7f274ahmaybxdw/lib/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.5") found components:  C 
-- Setting default Kokkos CXX standard to 11
-- Setting policy CMP0074 to use <Package>_ROOT variables
-- The project name is: Kokkos
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=c++11 for C++11 standard as feature
-- Execution Spaces:
--   Device Parallel: NONE
--     Host Parallel: NONE
--       Host Serial: SERIAL
-- 
-- Architectures:
-- Found TPLLIBDL: /usr/lib64/libdl.so  
-- Building unit tests.
-- Building integration tests.
-- Building regression tests.
-- PAR_LOOP_LAYOUT='SIMDFOR_LOOP' (default par_for wrapper layout)
-- Configuring done
CMake Error in tst/CMakeLists.txt:
  The compiler feature "cxx_alignas" is not known to CXX compiler

  "XL"

  version 16.1.1.7.


-- Build files have been written to: /home/jlippuner/work/parthenon/build_xl

[AndrewGaspar]

You configured like this:

CXX=xlc CC=xlc cmake .. 

But I think you meant this:

CXX=xlc++ CC=xlc cmake .. 

[jlippuner]

I didn't bother saying CXX=xlc++, because xlc++ is a symlink pointing to xlc (I'm on Darwin using the ibm/xlc-16.1.1.7-xlf-16.1.1.7 module). And unfortunately, saying CXX=xlc++ doesn't fix the issue:

[jlippuner@cn2025 build_xl]$ CXX=xlc++ CC=xlc cmake ..                                                                                                       
-- The C compiler identification is XL 16.1.1.7
-- The CXX compiler identification is XL 16.1.1.7
-- Check for working C compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc
-- Check for working C compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc++
-- Check for working CXX compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc++ -- works
[...]
-- Configuring done
CMake Error in tst/CMakeLists.txt:
  The compiler feature "cxx_alignas" is not known to CXX compiler

  "XL"

  version 16.1.1.7.

-- Build files have been written to: /home/jlippuner/work/parthenon/build_xl

[Yurlungur]

No objections from me once @jlippuner's issue is resolved.

[AndrewGaspar]

Do you need it for the XLCLANG compiler string as well? Perhaps MATCHES "^XL" instead of STEQUAL "XL" if this is the case.

This seems to work with just "XL", and I'm using the clang version of XL. They must have some compatibility mode in CMake?

[AndrewGaspar]

@jlippuner I should have specified the build process I followed better. You need to add this CXX flag to your CMake configuration:

cmake -DCMAKE_CXX_FLAGS=-F/projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/etc/xlc.cfg.rhel.7.7.gcc.9.3.0.cuda.10.1 ..

You'll also want to load the gcc/9.3.0 module so that some of the .so files that are needed are found when you try to run the executable.

I don't hard code this into the CMake because it's not portable. It's fully dependent on how the system admin chose to install XL.

[Yurlungur]

I wonder---can we get the power9 architecture integrated into the CI? @pgrete does your gitlab instance support that?

[nmsriram]

Do you need it for the XLCLANG compiler string as well? Perhaps MATCHES "^XL" instead of STEQUAL "XL" if this is the case.

This seems to work with just "XL", and I'm using the clang version of XL. They must have some compatibility mode in CMake?

If it works then that's fine. I was worried you'd get caught with this at some point that's all.

[AndrewGaspar]

At some point I thought we were going to try running the tests here on Darwin. @JoshuaSBrown - do you know what happened with that?

(btw, why does GitHub only support threaded comments on line comments?!)

[JoshuaSBrown]

At some point I thought we were going to try running the tests here on Darwin. @JoshuaSBrown - do you know what happened with that?

(btw, why does GitHub only support threaded comments on line comments?!)

Good question, you can just use quotes. I did not implement because 1. it was not a priority as the external guys already had a designated machine and 2. I think there are issues triggering the ci from a public github repo because darwin is an internal system. I can investigate this further if needed, but we might end up needing the enterprise version of github to make it work.

[pgrete]

I wonder---can we get the power9 architecture integrated into the CI? @pgrete does your gitlab instance support that?

Technically this is easily possible if we get access to a Power9 machine ("just" to install a gitlab-runner on there).
I currently only have access to Power9 machines in larger center where I'm pretty sure running sth external automated using my credential is against the policy.
I'll open a support tickets and ask for directions.
(I'm assuming here that no one else has access to a "private" Power9 machine or could we use Darwin -- I'm happy to take the technical discussion offline just in case)

[jlippuner]

I finally figured out the issue I was having on Darwin. The culprit is CMake. CMake version 3.12 thinks that the XL compiler does not support cxx_alignas. But CMake 3.15 and 3.17 knows that it does (there is no module for other CMake versions in between). I did have to use the XL config file that @AndrewGaspar mentioned so that XL would use the correct GCC headers.

Should we increase the minimum CMake version required for Parthenon to 3.15?

Here's what works for me on Darwin Power 9:

module purge
module load cmake/3.17.0
module load ibm/xlc-16.1.1.7-xlf-16.1.1.7
module load openmpi/p9/4.0.3-xlc_16.1.1.7-xlf_16.1.1.7
module load gcc/9.3.0
module load clang/8.0.1
spack load hdf5%xl # you need to build this yourself with spack

CXX=xlc++ CC=xlc cmake -DCMAKE_CXX_FLAGS=-F/projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/etc/xlc.cfg.rhel.7.7.gcc.9.3.0.cuda.10.1 ..
make -j
make test

All tests pass for me without doing anything about Catch2.

[jlippuner]

Works now for me on Darwin if I use CMake >= 3.15

[Yurlungur]

I finally figured out the issue I was having on Darwin. The culprit is CMake. CMake version 3.12 thinks that the XL compiler does not support cxx_alignas. But CMake 3.15 and 3.17 knows that it does (there is no module for other CMake versions in between). I did have to use the XL config file that @AndrewGaspar mentioned so that XL would use the correct GCC headers.

Here's what works for me on Darwin Power 9:

module purge
module load cmake/3.17.0
module load ibm/xlc-16.1.1.7-xlf-16.1.1.7
module load openmpi/p9/4.0.3-xlc_16.1.1.7-xlf_16.1.1.7
module load gcc/9.3.0
module load clang/8.0.1
spack load hdf5%xl # you need to build this yourself with spack

CXX=xlc++ CC=xlc cmake -DCMAKE_CXX_FLAGS=-F/projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/etc/xlc.cfg.rhel.7.7.gcc.9.3.0.cuda.10.1 ..
make -j
make test

All tests pass for me without doing anything about Catch2.

Perhaps we should document this configuration somewhere. I'd propose we have an extra doc file with build recipes for common machines we all use. It could save us all a lot of headache. I'll start an issue.

Should we increase the minimum CMake version required for Parthenon to 3.15?

I lean towards keeping the cmake requirements down and just documenting this issue. If we require newer versions of cmake it can become a pain on, e.g., an Ubuntu LTS laptop. (Something I've already experienced with our build system.)

[Yurlungur]

I wonder---can we get the power9 architecture integrated into the CI? @pgrete does your gitlab instance support that?

Technically this is easily possible if we get access to a Power9 machine ("just" to install a gitlab-runner on there).
I currently only have access to Power9 machines in larger center where I'm pretty sure running sth external automated using my credential is against the policy.
I'll open a support tickets and ask for directions.
(I'm assuming here that no one else has access to a "private" Power9 machine or could we use Darwin -- I'm happy to take the technical discussion offline just in case)

Thanks for doing that! I think we can use Darwin, which already has a gitlab instance running on it. But I suspect there are some technical and administrative hurdles to overcome.... to, for example, make the CI output public. @gshipman and @JoshuaSBrown are probably the people to ask about that. Also @junghans. I'll start an issue where we can continue the discussion.