gitlab-ci: Adds (command-line) tool to simplify .gitlab-ci.yml

[adcroft]

This is a re-factor of the gitlab pipeline scripts. The main change is to encapsulate in bash functions the long list of commands for each job/stage in the pipeline yml file. The bash functions can be invoked succinctly at the command line (see .gitlab/README.md) and most jobs in the pipeline are now one-liners. The build process still replicates FRE by using the MRS scripts but the functions are named so that we can deliberately add other build processes when they start being used for production.

This re-factor was triggered because the logic in the previous scripts was not catching all the parameter doc string changes. In fixing that issue, I took the opportunity to clean up the output and have tested various cases to make sure the reporting is correct. Example output:

• Broken restarts: https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/jobs/87411
• Change in doc strings: https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/jobs/87345
• Regression fail: https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/jobs/87438

Breakdown of changes:

• Adds .gitlab/pipeline-ci-tool.sh to enact most of the stages of the gitlab CI pipeline
  • enables interactive/command-line reproduction of the pipeline
  • .gitlab/pipeline-ci-tool.sh is documented in .gitlab/README.md with instructions on how to use it at the command line and what each function is doing
• All commands formerly in .gitlab-ci.yml are now one-line invocations of .gitlab/pipeline-ci-tool.sh so .gitlab-ci.yml is now considerably smaller and easier to read with statements like .gitlab/pipeline-ci-tool.sh mrs-compile debug_gnu or .gitlab/pipeline-ci-tool.sh check-params gnu
• Previously, all results were compared against the stored regression answers. This meant that any error (e.g. layout) would show up as a fail of all types. Hereafter, we use the regression answers to check the repro-symmetric mode and then compare everything else to repro-symmetric or other results as appropriate. This allows us to distinguish between types of errors. The GH actions do the comparisons this way, and we originally did this in the first forms of the pipeline, but in the last re-factor when I sped up the process, I lazily switched to using the regression answers and this was a mistake.

Future:

• The experiment suite(s), e.g. MOM6-examples, should provide some sort of API (bash tool?) or follow a protocol (yaml file?) that we could use to define executables and models to test. Similarly for the stats repo. Currently, the MOM6 pipeline has to know the details of MOM6-examples/gaea-stats in order to run the regression tests and we really need to be able to run/test multiple suites (e.g. ESMG, CPT, etc) and not just MOM6-examples/gaea-stats. The functions that do things like “build an executable” or “compare stats files” would then be moved out of this script and to scripts/tools in those repos.
• Environment variables vs command-line args: currently the scripts require certain variables setup by the CI. We could have provided the same info as command line args. However, when I tried it this way, it became harder to code for multiple stages on the same command-line (a helpful feature when trying to do stuff on the command-line). I could go either way but given this is meant for the CI and the variables therefore exist, the lines in the yml file is much simpler to read.
• I would like to avoid these scripts becoming the routine way to operate interactively; they are necessarily destructive In the temporary job directory and if you use “./” for that directory your would clobber your repo working directory. Naming the script pipeline-ci-tool.sh I hope will avoid anyone thinking this is for users.

[codecov]

Codecov Report

Merging #226 (3e2da9a) into dev/gfdl (b525932) will decrease coverage by 0.55%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           dev/gfdl     #226      +/-   ##
============================================
- Coverage     37.78%   37.23%   -0.56%     
============================================
  Files           263      263              
  Lines         71012    73060    +2048     
  Branches      13149    13605     +456     
============================================
+ Hits          26834    27201     +367     
- Misses        39279    40841    +1562     
- Partials       4899     5018     +119     

Impacted Files	Coverage Δ
src/core/MOM_open_boundary.F90	25.56% <0.00%> (-22.83%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Hallberg-NOAA]

This looks like a very valuable refactoring and cleanup of the gitlab-CI code. Apart from noting the occasional spelling error (unless "eport" on line 15 means something technical that I have not heard of) it all looks good to me. However, rather than approving this myself, I would like to give Marshall or anyone else who speaks fluent yml to also take a look.

[marshallward]

Gaea regression: https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/pipelines/17282 ✔️

(Hopefully this used the new gitlab config file)

[marshallward]

Looks good. Typos etc can be sorted out in the next refactor.