Second round of benchmarking on SuperMUC-NG

Second release of SCEMa for benchmarking purposes on SuperMUC-NG.

This new release features two optimisations solving issues found during the first benchmark:

• avoid communication across nodes within a single LAMMPS simulation by making MD allocation size a factor or a multiple of node core count;
• parallelise I/O (mkdir and rmdir calls) by disabling logging of MD simulations

Implementation of compact-tension and dogbone test simulations for large-scale simulations of SCEMa on up to a full partition of SuperMUC-NG.

Input files featuring SCEMa configuration file (.json), FE meshes (.msh) and molecular data (.lammpstrj, .bin, etc...) can be found in the attached tarball named inputs_benchmark_SupermucNG_12.2020.tar.gz.

Specific installation and execution directions for SuperMUC-NG can be found below and in the README.md file included in the tarball.

Building SCEMa on SuperMUC-NG

First deal.ii and LAMMPS need to be compiled with the dependencies listed here: https://github.com/UCL-CCS/SCEMa/blob/master/README.md

Compiling deal.ii

More specifically for deal.ii (version 8.5.0 or above) which can be a bit tricky to compile, the following modules need to be loaded on SuperMUC-NG:

module load python/3.6_intel petsc/3.8.4-impi-shared metis/5.1.0-intel-i64-r64 cmake/3.14.4 tbb/2019 gcc/7 mumps/5.0.1-intel-impi-shared scalapack/mkl netcdf/4.6.1-impi-hdf5v1.10-parallel hdf5/1.10.2-intel-impi-frt-threadsafe

Once these are loaded, configuration of deal.ii Makefiles should be straightforward:

cd /path/to/deal.ii/8.5.0/source
mkdir build
cmake -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_PETSC=ON ..

From there, all the necessary options should be automatically set to 'ON', and deal.ii can be built:

make -j 8 install

Compiling SCEMa

Once both libraries are installed, edit paths in SCEMa CMakeLists configuration file which points toward:

• Deal.ii build directory: /path/to/dealii-8.5.0/
• LAMMPS sources directory: /path/to/lammps/src/
• Python include directory: /path/to/python/include/pythonX.Y/
• LAMMPS shared library: /path/to/lammps/src/liblammps.so
• Python shared library: /path/to/python/lib/libpythonX.Y.so

grep -R "/path/to" /path/to/inputs_benchmark/CMakeLists.txt.generic

and move to SCEMa's directory:

cd /path/to/SCEMa/source/
mkdir build
cd build
cmake /path/to/inputs_benchmark/CMakeLists.txt.generic
make dealammps

Running a simulation of the dogbone test for pure epoxy resin

Before trying to perfom a simulation using SCEMa:

• deal.ii version 8.5 or above must be installed
• LAMMPS (preferably version 17Nov16) must be installed
• dealammps executable (from the SCEMa repository) must be build

Retrieve benchmark inputs tarball archived on the GitHub page of the release and decompress it:

tar -xzvf inputs_benchmark.tar.gz

Create a directory for the simulation data

mkdir dogbone
cd dogbone

Copy the JSON inputs configuration file and the SLURM execution script:

cp ../inputs_benchmark/slurm.dogbone ../inputs_benchmark/inputs_dogbone.json .

Edit generic path to SCEMa directory (/path/to/SCEMa) in both files

Submit the simulation:

sbatch slurm.dogbone 

Verification of graph-based clustering algorithm

Implementation of compact-tension, dogbone (from mesh file or from regular cuboid) and drop-weight test simulations to validate the clustering algorithm relying on splines for dimensionality reduction and graphs for maximum number of MD simulations removal.

The validation can be performed in two steps:

• Replacing MD simulations with a constitutive model approximation of the stress (Hooke's law, or RNN-based surrogate for example) to predict stresses. Based on the number of independent stress evaluations (usually performed via MD simulations), and assuming a theoretical distribution of the CPU resources required by one MD simulation, one can predict the speed-up or the computational cost reduction enabled by the clustering algorithm. Such method is ideal for simulations on large meshes (dogbone_file3D, compact, dropweight_cuboid).
• Once the theoretical computational cost reduction is assessed, one can perform a real multiscale simulation embedding FE coupled with MD to assess the real computational cost reduction enabled by the clustering algorithm. This is meant to be performed on a single case (e.g. dogbone_cuboid)

Input files featuring SCEMa configuration file (.json), FE meshes (.msh) and molecular data (.lammpstrj, .bin, etc...) can be found in the tarball named inputs_validation_clustering_ARCHER_04.2020.tar.gz.

Benchmarking on SuperMUC-NG

Implementation of compact-tension and dogbone test simulations for large-scale benchmarking of SCEMa on up to a full partition of SuperMUC-NG.

Input files featuring SCEMa configuration file (.json), FE meshes (.msh) and molecular data (.lammpstrj, .bin, etc...), but also specific installation directions for SuperMUC-NG (.md) can be found in the tarball named inputs_benchmark_SupermucNG_05.2020.tar.gz.