Building and running SW4 on Summit
SW4 (Seismic Waves, 4th order) implements substantial capabilities for 3-D seismic modeling, with a free surface condition on the top boundary, absorbing super-grid conditions on the far-field boundaries, and an arbitrary number of point force and/or point moment tensor source terms.
SW4 uses 3 domains (i.e., meshes) per rank. Two uniform domains and one curvilinear (structured grid with an explicit coordinate system).
git clone https://github.com/Alpine-DAV/sw4.git
cd sw4
git checkout ascent
These modules load the correct compilers and setup the environment for building.
ml cuda/10.1.168
ml gcc/5.4.0
ml cmake/3.14.2
We have build and installed Ascent in the ALPINE project directory:
/gpfs/alpine/proj-shared/csc340/larsen/ascent/install
This is the default Ascent that SW4 will build against, and no changes to the SW4 build are needed. Note: you will need to be a member of csc340 to access the build.
To build your own Ascent, you can follow the instructions here: https://github.com/Alpine-DAV/ascent/wiki#summit--gcc--cuda--static
To point SW4 at the custom Ascent build, modify line 10 of the SW4 file
configs/make.inc
to point to your install
ASCENT_INSTALL_DIR=/gpfs/alpine/proj-shared/csc340/larsen/ascent/install
cd sw4
make
The sw4 executable is built in the optimize directory and is called sw4.
The current integration of SW4 has a hard coded visualization frequency of 100. This can be changed by altering the line: https://github.com/Alpine-DAV/sw4/blob/c7fb4db0e61dafe9aee347c1c5374557f2213cd3/src/solve.C#L56
SW4 had ghost zones and the field name is ascent_ghosts. The published vector field is named something. I apologize for the name, but the actual variable is dependent on what is in the input deck. Most likely, this is velocity, but could be displacement.
With no ascent_actions file, the default behavior is to save out the mesh.
To run, go to the special place on summit that lets you write to files. In my testing, this is $MEMBERWORK, but I have heard that scratch space works as well.
ml cuda/10.1.168
ml gcc/5.4.0
ml cmake/3.14.2
The provide input deck needs to be copied into the launch directory. They are located at
/gpfs/alpine/proj-shared/csc340/larsen/sw4_deck/hayward-att-h200-ref.in
/gpfs/alpine/proj-shared/csc340/larsen/sw4_deck/m6.5-20.0x13.0.s500.v5.1.srf
The input deck references files located in the same directory as above, but that should be fine.
The rupture file (.srf) defines the earthquake.
By default, the input deck will create about 100M zones, which is the typical problem size for 6 GPUs (1 summit node). To alter the deck for debugging change the h value at line 8:
grid x=100e3 y=100e3 z=30e3 h=200 lat=38.33 lon=-122.075 az=143.6380001671 proj=tmerc datum=NAD83 lon_p=-123.0 lat_p=35.0 scale=0.9996
The higher the value, the smaller the mesh. h=500 creates about 8M zones and a run with 6 GPUs will finish in less than 10 minutes.
Through a process of trial and error the number of mesh points approximately follows this equation: num_points = 678745400000000*h^-2.993227.
For a default set of actions copy this file into the launch directory:
/gpfs/alpine/proj-shared/csc340/larsen/sw4_deck/ascent_actions.yaml
Get yourself an interactive node (bsub -W 30 -nnodes 1 -P CSC340 -Is /bin/bash) or write a batch script
jsrun -n 6 -g 1 path_to_sw4 hayward-att-h200-ref.in