Compiler cmake issue

[koecher]

This is a PR Draft to figure out the problems we have with setting the compiler and cmake

Updated packages:

• Trilinos 13.0.1
• (optional SuperLU_dist 6.4.0 seems to be working with trilinos 13.0.1)

Developments

• New detection of BLAS, LAPACK, MKL, openBLAS in trilinos.package
• removes explicit compiler settings for cmake in trilinos.package

Successful tests on freshly set up virtual machines with the recommended installation instructions
@koecher

• ubuntu 21.04, gcc-10, openmpi from OS, cmake 3.18 from OS, candi sets CC,CXX,etc to mpi compilers
• ubuntu 21.04, gcc-10, openmpi from OS, cmake 3.20.5 from candi (unrelated superlu problem for deal.II / linking blas)
• CI / ubuntu 18.04 minimal (pull_request) Successful
• CI / ubuntu 20.04 (pull_request) Successful
• centos 7, gcc-8.1.0/mpich-3.2 self-compiled with all env. variables set, cmake 3.20.5 from candi, Successful
• ubuntu 20.04 (LTS), gcc-9, openmpi from OS, cmake 3.16.3 from OS, Successful
• ubuntu 20.04 (LTS), gcc-9, openmpi from OS, cmake 3.20.5 from candi, Successful
• ubuntu 20.04 (LTS), gcc-10 alternative, openmpi from OS, cmake 3.16.3 from OS
• ubuntu 20.04 (LTS), gcc-10 alternative, openmpi from OS, cmake 3.20.5 from candi
• opensuse 15 / Leap, gcc-7.5, mpich from OS, cmake 3.17.0 from OS, Successful
• CI / OSX gcc (pull_request) Has Failure in a test after install, seems unrelated
• CI / OSX clang (pull_request), Successful
• macOS 11.4.1 (bigsur), gcc-11, openmpi, cmake 3.21.0 from homebrew, Successful

@gfcas

• Ubuntu 18.04 (LTS), gcc-7.5, openmpi from OS, cmake 3.20.5 from OS
• Ubuntu 20.04 (LTS), gcc-9.3, openmpi from OS, cmake 3.20.5 from OS
• Ubuntu 20.04 (LTS), gcc-10.3, openmpi from OS, cmake 3.20.5 from OS, succeded, https://github.com/gfcas/candi/actions/runs/1022810879

Clean ups and TODOs

• remove local.cfg commit
• Trilinos configuration variables in candi
• BLAS and LAPACK dirs
• TODOs in trilinos.package
• SuperLU_dist BLAS/LAPACK linking problem (only for v7.0.0), resolved

[koecher]

Related to #207 #159

[koecher]

@gfcas can you test this branch on your machines?

[ghost]

@gfcas can you test this branch on your machines?

@koecher of course, I will also reconfigure the github actions runners. On my personal laptop maybe it will take some days because I got a new one.

[ghost]

I think it very good that we update the Trilinos package, which is one of the major ones for parallel computing. Should we include also Umfpack and Mumps as described in the dealii readme (https://www.dealii.org/current/external-libs/trilinos.html). For my own work I was recently interested in the trilinos subpackage ShyLU.

[koecher]

I think it very good that we update the Trilinos package, which is one of the major ones for parallel computing. Should we include also Umfpack and Mumps as described in the dealii readme (https://www.dealii.org/current/external-libs/trilinos.html). For my own work I was recently interested in the trilinos subpackage ShyLU.

Let us concentrate on Trilinos and deal.II for now to find a solution to the problem

[koecher]

@gfcas no reviews so far, everything is draft for testing

[ghost]

@koecher:

• Ubuntu 18.04 (LTS), gcc-7.5, openmpi from OS, cmake 3.20.5 from OS
• Ubuntu 20.04 (LTS), gcc-9.3, openmpi from OS, cmake 3.20.5 from OS
• Ubuntu 20.04 (LTS), gcc-10.3, openmpi from OS, cmake 3.20.5 from OS
  just succeded, see https://github.com/gfcas/candi/actions/runs/1022810879

[koecher]

I've cleaned up the discussion. Please only report on the compiler-cmake issue. SuperLU is not of interest anymore here

[ghost]

In order to get this issue ready for v9.3.1 maybe we should consistently test it with DEAL_II_VERSION=v9.3.1?

[koecher]

I think we need a test for intel compilers, the other systems look good.

Testing with the fixed v9.3.1 is okay.

[koecher]

Remarks for macOS:

Packages:

• parmetis 4.0.3
• hdf5 1.10.7
• superlu_dist 6.4.0
• p4est 2.2
• trilinos 13.0.1 (without compiler settings, without -lgfortran)
• deal.II v9.3.1 (without compiler settings)

Special configuration: LC_ALL=C and unset LANGUAGE since step-32 doesn't work without it

export OMPI_CC=gcc-11
export OMPI_CXX=g++-11
export OMPI_FC=gfortran-11
export CC=mpicc
export CXX=mpicxx
export FC=mpifort
export FF=mpifort
export LC_ALL=C
unset LANGUAGE

Tested step tutorials

• step-1
• step-7
• step-29 (using deal.II complex and umfpack as direct solver)
• step-32 (needs LC_ALL=C)
• step-33
• step-40 (only Trilinos installed)
• step-72
• hdf5 file output with own software

Additional system preferences: (if dylib library not found, e.g. for TrilinosWrappers::SolverDirect)

• System Preferences / Security & Privacy / Developer Tools / Terminal

[tjhei]

Do you think it is a good time to go to Trilinos 13 at this point? I am not sure we have tested it much with deal.ii.

[koecher]

Do you think it is a good time to go to Trilinos 13 at this point? I am not sure we have tested it much with deal.ii.

Well this was a try if things are going well so far. I think we should test the parallel features of trilinos 13 a little more.
From the current tests on macos, I'm fine with trilinos 13.

[koecher]

@tjhei do you have the chance to test this with intel compilers?

[tjhei]

Deal.II currently suggests using 12.18, see https://github.com/dealii/dealii/blob/94b2450484e130a74051c7bd7230c8d6f79b98b2/doc/external-libs/trilinos.html#L53
I don't think it is a good idea to go to 13 right now, especially because it does not fix any problem, or does it?

What do you want to me to test, just run with default settings with Intel and see if things compile?

[koecher]

Deal.II currently suggests using 12.18, see https://github.com/dealii/dealii/blob/94b2450484e130a74051c7bd7230c8d6f79b98b2/doc/external-libs/trilinos.html#L53
I don't think it is a good idea to go to 13 right now, especially because it does not fix any problem, or does it?

What do you want to me to test, just run with default settings with Intel and see if things compile?

• can you test if this PR is running smoothly? Here is a local.cfg to set the packages.
• I suggest to stay with trilinos 12 for deal.II v9.3 and
• introduce an additional trilinos13.package for master

This PR isn't useful for a merge, the things we learned here should be used for clean PRs in future

[tjhei]

@zjiaqi2018 can you please check out this PR of candi on palmetto or frontera and compile with current Intel compilers and MKL? Please set MKL=ON in the local.cfg. Let us know what changes you need to do or if you get any errors.

[zjiaqi2018]

@zjiaqi2018 can you please check out this PR of candi on palmetto or frontera and compile with current Intel compilers and MKL? Please set MKL=ON in the local.cfg. Let us know what changes you need to do or if you get any errors.

I tested it on frontera, and it seems to stop at:

[zjiaqi2018]

@zjiaqi2018 can you please check out this PR of candi on palmetto or frontera and compile with current Intel compilers and MKL? Please set MKL=ON in the local.cfg. Let us know what changes you need to do or if you get any errors.

I tested it on frontera, and it seems to stop at:

with -D DEAL_II_COMPONENT_EXAMPLES=OFF, it doesn't work either:

[tjhei]

Is this on Frontera? Rene had to disable gold linker, see "-fuse" in https://github.com/geodynamics/aspect/wiki/Installation-on-Frontera

[tjhei]

Can you post the deal.ii summary.log and your changes to the settings once it works?

[zjiaqi2018]

It works now. For the local changes, I just follow Rene's instructions you mentioned.