Automated Jenkins Testing
The WRF repository is linked to a Jenkins server that runs on AWS. Whenever a pull request (PR) is made or when a commit to an existing PR is made, a suite of tests are conducted. Those tests take less than an hour to complete. The purpose of this page is to fully describe the tests that are conducted and to assist the user in interpreting the test results.
The Jenkins server was developed and is maintained by ScalaComputing.
The automated tests are referred to as regression tests. A successful completion of the testing indicates that the source code modification has no side effects that break other WRF options. The testing does not specifically verify the proposed modification. The developer is responsible for demonstrating that the new code performs as expected.
The following table lists the broad test areas (currently 10), and the associated 24 parallel build types across those 10 broad test areas.
| Test Number | Mnemonic | Description | Type |
|---|---|---|---|
| 1 | em_real | ARW real data | S O M |
| 2 | nmm_real | NMM real data | S - M |
| 3 | em_chem | Chemistry real | S - M |
| 4 | em_quarter_ss | Ideal Supercell | S O M |
| 5 | em_b_wave | Ideal Baroclinic Wave | S O M |
| 6 | em_real8 | ARW real 8-byte float | S O M |
| 7 | em_quarter_ss8 | Supercell 8-byte float | S O M |
| 8 | em_move | Moving Nest | - - M |
| 9 | em_fire | Ideal Ground Fire | S O M |
| 10 | em_hill2d_x | Ideal 2d Hill | S - - |
Type refers to the requested parallel option:
- S = Serial
- O = OpenMP
- M = MPI Therefore "S O M" refers to a test that is built for Serial, OpenMP, and MPI. The "S - -" entry refers to a test that is only built for Serial processing.
The regression tests are short. For each of the 24 builds (for example em_real using MPI), one or more simulations are performed. The simulations are approximately 10 time steps in length.
The WRF model requires both the netCDF directory structure (lib, include, and bin), and an MPI implementation (OpenMPI is selected). The compiler is GNU, using both gfortran and gcc. To keep a uniform environment for the testing, the WRF source code, test data, and built libraries are placed inside a Docker container (one container for each of the 24 builds).
The simulation choices for each of the 24 builds are controlled by a script that assigns namelists.
| Test Name | Namelists Tested |
|---|---|
| em_real | 03FD 10 11 20NE |
| nmm_nest | 01 03 04a 06 |
| em_chem | 1 2 5 |
| em_quarter_ss | 02NE 03 03NE 04 |
| em_b_wave | 1NE 2 2NE 3 |
| em_real8 | 14 17AD |
| em_quarter_ss8 | 06 08 09 |
| em_move | 01 02 |
| em_fire | 01 |
| em_hill2d_x | 01 |
The complete set of namelists options are available via wget or curl: tar file of WRF namelists used in the regression tests.
A number of indicators are reviewed to determine SUCCESS vs FAIL, depending on the stage of the automated testing.
-
Build SUCCESS
- Both wrf.exe and the associated pre-processor exist and have executable privilege
-
Individual Simulation Run SUCCESS
- Pre-processor print output has SUCCESS message
- Pre-processor netCDF output exists
- WRF model print output has SUCCESS message
- WRF model netCDF output exists for each domain
- Two time periods exist in each WRF model netCDF output file
- No occurrence of "NaN" in any WRF model netCDF output file
-
Parallel Comparison SUCCESS (assuming same namelist configuration)
- If Serial and OpenMP types exist, compare bit-wise identical results
- If Serial and MPI types exist, compare bit-wise identical results
After approximately an hour from the time of the PR or commit, an email arrives. The email is sent from user jenkins, and due to the contents, many email servers identify the message as SPAM.
The content of the email message lists sufficient information to determine to which PR and commit the test refers, and the email also provides the github ID of the person proposing the change (usually the email recipent).
The email contains an attachment: wrf_output.zip.
- Download the file wrf_output.zip.
- Inflate (decompress) this zip'ed file
unzip wrf_output.zip
- A new directory is created, get into that directory
cd output_testcase
- There are 10 files. Each file is associated with the 10 broad area tests that are conducted. For example, test 1 refers to em_real, test 2 refers to nmm_real, etc.
ls -ls
total 3896
472 -rw-r--r--@ 1 gill 1500 239033 Dec 12 18:57 output_1
408 -rw-r--r--@ 1 gill 1500 206813 Dec 12 18:47 output_10
624 -rw-r--r--@ 1 gill 1500 316727 Dec 12 18:48 output_2
184 -rw-r--r--@ 1 gill 1500 91521 Dec 12 19:09 output_3
624 -rw-r--r--@ 1 gill 1500 319111 Dec 12 18:54 output_4
480 -rw-r--r--@ 1 gill 1500 244403 Dec 12 18:53 output_5
264 -rw-r--r--@ 1 gill 1500 134162 Dec 12 18:54 output_6
464 -rw-r--r--@ 1 gill 1500 235128 Dec 12 18:53 output_7
128 -rw-r--r--@ 1 gill 1500 62690 Dec 12 18:49 output_8
248 -rw-r--r--@ 1 gill 1500 126681 Dec 12 18:55 output_9
Within each file are a number of sections. They can be searched and aggregated information can be displayed. The -a option to grep indicates that the file should be treated as an ASCII text file, regardless of the presence of binary characters being detected.
- There should be 24 total builds that are attempted:
grep -a " START" * | grep -av "CLEAN START" | wc -l
24
- There are 43 bit-for-bit comparisons:
grep -a "status = " * | wc -l
43
All should have status = 0 (success), so NONE should have any other status:
grep -a "status = " * | grep -av "status = 0" | wc -l
0
- There are 71 total simulations:
grep -a " = STATUS" * | wc -l
71
All have status = 0, so NONE have a nonzero status:
grep -a " = STATUS" * | grep -av "0 = STATUS" | wc -l
1
- The top part of each output file contains the print out related to creating the Docker container. This starts with:
Sending build context to Docker daemon 289.3kB^M^M
Step 1/6 : FROM davegill/wrf-coop:fifthtry
fifthtry: Pulling from davegill/wrf-coop
and ends with:
Step 6/6 : CMD ["/bin/tcsh"]
---> Running in e911e10a911a
Removing intermediate container e911e10a911a
---> b1233e0731e8
Successfully built b1233e0731e8
- The second part of each file is the output of the real / ideal program and the WRF model, for each of the namelist choices, for each of the parallel build types. The parallel build sections begin with:
==============================================================
==============================================================
SERIAL START
==============================================================
or
==============================================================
==============================================================
OPENMP START
==============================================================
or
==============================================================
==============================================================
MPI START
==============================================================
Within each of these larger sections, the namelists are looped over (32 is the serial build for GNU):
grep "32" output_1 | grep "NML = "
RUN WRF test_001s for em_real 32 em_real, NML = 03FD
RUN WRF test_001s for em_real 32 em_real, NML = 10
RUN WRF test_001s for em_real 32 em_real, NML = 11
RUN WRF test_001s for em_real 32 em_real, NML = 20NE
- The third and final section is provided only for tests that support multiple parallel build types (for example, serial and mpi). The third section details the comparisons between the different parallel build, and those comparisons should always be bit-wise identical.
tail output_1
SUCCESS_RUN_WRF_d01_em_real_32_em_real_03FD vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_03FD status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_03FD vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_03FD status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_10 vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_10 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_10 vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_10 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_11 vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_11 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_11 vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_11 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_20NE vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_20NE status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_20NE vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_20NE status = 0
Fri Dec 13 01:57:35 UTC 2019