Automated Jenkins Testing
The WRF repository is linked to a Jenkins server that runs on AWS. Whenever a pull request (PR) is made or when a commit to an existing PR is made, a suite of tests are conducted. Those tests take less than an hour to complete. The purpose of this page is to fully describe the tests that are conducted and to assist the user in interpreting the test results.
The Jenkins server was developed and is maintained by ScalaComputing.
The automated tests are referred to as regression tests. A successful completion of the testing indicates that the source code modification has no side effects that break other WRF options. The testing does not specifically verify the proposed modification. The developer is responsible for demonstrating that the new code performs as expected.
The following table lists the broad test areas (currently 10), and the associated 24 parallel build types across those 10 broad test areas.
| Test Number | Mnemonic | Description | Type |
|---|---|---|---|
| 1 | em_real | ARW real data | S O M |
| 2 | nmm_real | NMM real data | S - M |
| 3 | em_chem | Chemistry real | S - M |
| 4 | em_quarter_ss | Ideal Supercell | S O M |
| 5 | em_b_wave | Ideal Baroclinic Wave | S O M |
| 6 | em_real8 | ARW real 8-byte float | S O M |
| 7 | em_quarter_ss8 | Supercell 8-byte float | S O M |
| 8 | em_move | Moving Nest | - - M |
| 9 | em_fire | Ideal Ground Fire | S O M |
| 10 | em_hill2d_x | Ideal 2d Hill | S - - |
Type refers to the requested parallel option:
- S = Serial
- O = OpenMP
- M = MPI
Therefore "S O M" refers to a test that is built for Serial, OpenMP, and MPI. The "S - -" entry refers to a test that is only built for Serial processing.
The regression tests are short. For each of the 24 builds (for example em_real using MPI), one or more simulations are performed. The simulations are approximately 10 time steps in length.
The WRF model requires both the netCDF directory structure (lib, include, and bin), and an MPI implementation (OpenMPI is selected). The compiler is GNU, using both gfortran and gcc. To keep a uniform environment for the testing, the WRF source code, test data, and built libraries are placed inside a Docker container (one container for each of the 24 builds).
The simulation choices for each of the 24 builds are controlled by a script that assigns namelists.
| Test Name | Namelists Tested |
|---|---|
| em_real | 03FD 10 11 20NE |
| nmm_nest | 01 03 04a 06 |
| em_chem | 1 2 5 |
| em_quarter_ss | 02NE 03 03NE 04 |
| em_b_wave | 1NE 2 2NE 3 |
| em_real8 | 14 17AD |
| em_quarter_ss8 | 06 08 09 |
| em_move | 01 02 |
| em_fire | 01 |
| em_hill2d_x | 01 |
The complete set of namelists options are available via wget or curl: tar file of WRF namelists used in the regression tests.
Once a PR is proposed and for each subsequent commit to that PR, automated testing is performed via the Jenkins server (for security purposes, this server is not generally accessible).
When the automated tests are all complete, and if the tests were successful, then multiple green checkmarks appear:
A number of indicators are reviewed to determine SUCCESS vs FAIL, depending on the stage of the automated testing.
-
Build SUCCESS
- Both wrf.exe and the associated pre-processor exist and have executable privilege
-
Individual Simulation Run SUCCESS
- Pre-processor print output has SUCCESS message
- Pre-processor netCDF output exists
- WRF model print output has SUCCESS message
- WRF model netCDF output exists for each domain
- Two time periods exist in each WRF model netCDF output file
- No occurrence of "NaN" in any WRF model netCDF output file
-
Parallel Comparison SUCCESS (assuming same namelist configuration)
- If Serial and OpenMP types exist, compare bit-wise identical results
- If Serial and MPI types exist, compare bit-wise identical results
After approximately an hour from the time of the PR or commit, an email arrives. The email is sent from user jenkins, and due to the contents, many email servers identify the message as SPAM.
The content of the email message lists sufficient information to determine to which PR and commit the test refers, and the email also provides the github ID of the person proposing the change (usually the email recipent).
The email contains an attachment: wrf_output.zip.
- Download the file wrf_output.zip.
- Inflate (decompress) this zip'ed file
unzip wrf_output.zip
- A new directory is created, get into that directory
cd output_testcase
- There are 10 files. Each file is associated with the 10 broad area tests that are conducted. For example, test 1 refers to em_real, test 2 refers to nmm_real, etc.
ls -ls
total 3896
472 -rw-r--r--@ 1 gill 1500 239033 Dec 12 18:57 output_1
408 -rw-r--r--@ 1 gill 1500 206813 Dec 12 18:47 output_10
624 -rw-r--r--@ 1 gill 1500 316727 Dec 12 18:48 output_2
184 -rw-r--r--@ 1 gill 1500 91521 Dec 12 19:09 output_3
624 -rw-r--r--@ 1 gill 1500 319111 Dec 12 18:54 output_4
480 -rw-r--r--@ 1 gill 1500 244403 Dec 12 18:53 output_5
264 -rw-r--r--@ 1 gill 1500 134162 Dec 12 18:54 output_6
464 -rw-r--r--@ 1 gill 1500 235128 Dec 12 18:53 output_7
128 -rw-r--r--@ 1 gill 1500 62690 Dec 12 18:49 output_8
248 -rw-r--r--@ 1 gill 1500 126681 Dec 12 18:55 output_9
Within each file are a number of sections. They can be searched and aggregated information can be displayed. The -a option to grep indicates that the file should be treated as an ASCII text file, regardless of the presence of binary characters being detected.
- There should be 24 total builds that are attempted:
grep -a " START" * | grep -av "CLEAN START" | wc -l
24
- There are 43 bit-for-bit comparisons:
grep -a "status = " * | wc -l
43
All should have status = 0 (success), so NONE should have any other status:
grep -a "status = " * | grep -av "status = 0" | wc -l
0
- There are 71 total simulations:
grep -a " = STATUS" * | wc -l
71
All have status = 0, so NONE have a nonzero status:
grep -a " = STATUS" * | grep -av "0 = STATUS" | wc -l
1
- The top part of each output file contains the print out related to creating the Docker container. This starts with:
Sending build context to Docker daemon 289.3kB
Step 1/6 : FROM davegill/wrf-coop:fifthtry
fifthtry: Pulling from davegill/wrf-coop
and ends with:
Step 6/6 : CMD ["/bin/tcsh"]
---> Running in e911e10a911a
Removing intermediate container e911e10a911a
---> b1233e0731e8
Successfully built b1233e0731e8
- The second part of each file is the output of the real / ideal program and the WRF model, for each of the namelist choices, for each of the parallel build types. The parallel build sections begin with:
==============================================================
==============================================================
SERIAL START
==============================================================
or
==============================================================
==============================================================
OPENMP START
==============================================================
or
==============================================================
==============================================================
MPI START
==============================================================
Within each of these larger sections, the namelists are looped over (32 is the serial build for GNU):
grep "32" output_1 | grep "NML = "
RUN WRF test_001s for em_real 32 em_real, NML = 03FD
RUN WRF test_001s for em_real 32 em_real, NML = 10
RUN WRF test_001s for em_real 32 em_real, NML = 11
RUN WRF test_001s for em_real 32 em_real, NML = 20NE
- The third and final section is provided only for tests that support multiple parallel build types (for example, serial and mpi). The third section details the comparisons between the different parallel build, and those comparisons should always be bit-wise identical.
tail output_1
SUCCESS_RUN_WRF_d01_em_real_32_em_real_03FD vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_03FD status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_03FD vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_03FD status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_10 vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_10 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_10 vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_10 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_11 vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_11 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_11 vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_11 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_20NE vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_20NE status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_20NE vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_20NE status = 0
Fri Dec 13 01:57:35 UTC 2019
Here are the steps to construct a docker container that has the directory structure, data, libraries, and the executable ready for conducting a test run (typically re-running a test case that has failed).
-
Log into the classroom as "150".
-
Get to the right location
cd /classroom/dave/wrf-coop
- We use the SHARED directory as an anchor for the shared volume between Docker and Linux.
set SHARED = `pwd`
- Building the Docker image takes about 3 minutes, regardless of the type of WRF run. The file Dockerfile needs to point to the requested fork and branch.
docker build -t wrf_regtest .
- Making an instance (a Docker container) of the image takes less than 30 s. Notice that you are now IN the container (for example, your prompt has changed). This command depends on which build of the WRF model is to be tested, as the name of the container is explicitly used in most of the following commands. This name is available from the specific test to be re-run.
docker run -it --name test_001m -v $SHARED/OUTPUT:/wrf/wrfoutput wrf_regtest /bin/tcsh
The next steps build and run the WRF executable by putting data in the right places. Once the pump is primed, you'll be able to do 1-line MPI commands with WRF. Building the WRF code (step 6) takes about 3 minutes (Chemistry is about 15 minutes). Running the WRF code is quite variable, from 30 s (most cases) to 15 minutes (for Chemistry). As with the name of the test, the arguments for the build and run are available from the files test_0[01][0-9][som].csh. This example is from test_001m.csh.
6. Build the WRF executables
./script.csh BUILD CLEAN 34 1 em_real -d J=-j@8
- Select the specific test to re-run. This information could be from the email notification of a failure. The complete list is in the chosen test_0[01][0-9][som].csh file.
set t = 03FD
- Run the pre-processor and the WRF model with the script to get the correct data in the right place.
./script.csh RUN em_real 34 em_real $t NP=8
- Start the manual testing by going to the WRF run-time directory.
cd WRF/test/em_real
- Now you can test with WRF repeatedly. Note also, that is is where you can modify a namelist manually to conduct a specific test for a new feature.
mpirun -np 8 -oversubscribe wrf.exe
- If the job is Serial:
./wrf.exe
- If the job is OpenMP
setenv OMP_NUM_THREADS 4
./wrf.exe
- If you exit the container, you can always get back "in" and keep working.
docker start -ai test_001m
cd WRF/test/em_real
mpirun -np 8 -oversubscribe wrf.exe
-
If you need to transfer a file to-from the container, use /wrf/wrfoutput inside the container and use /classroom/dave/wrf-coop/OUTPUT outside of the container.
-
You can always remove the (docker rm) containers and (docker rmi) images. There are no side effects. First exit the container, then from the Linux OS, get the container name and remove it.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
428227e682bd wrf_regtest "/bin/tcsh" 17 minutes ago Exited (0) 5 seconds ago test_001m
docker rm test_001m
- Get the names of the images, and remove them.
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
wrf_regtest latest c182ba751eb0 23 minutes ago 4.72 GB
docker.io/davegill/wrf-coop fifthtry efc665da99ef 7 months ago 4.28 GB
docker rmi c182ba751eb0 efc665da99ef