Add CI Reproducibility Testing

[aidanheerdegen]

As a team member I want automated reproducibility testing of code I add to the repository. This removes a burdensome task from me, but it also means the checking is consistent across the team, and any updates or improvements to the automated testing will benefit everyone.

As a Model Release team member, an ACCESS-NRI team member and a model user I want automated reproducibility testing of code to ensure code being merged into codebase will be consistent with previous versions and not change model answers. This testing will need to be run on the correct HPC hardware (gadi@NCI). If it does change model answers I want it clearly documented why.

As a Model Release team member I want these tests to use the reproducibility framework and tools developed by the NRI.

Edit 9/9/22 12:17: Typos

[aidanheerdegen]

We don't want to release models without doing run testing first. In order to test a model we need to deploy it to HPC in some way first.

This is a chicken and egg or perhaps a cart before the horse problem.

We can solve this by having a pre-release staging area where we automatically deploy models without run testing, and then trigger a pull-request to https://github.com/ACCESS-NRI/access-om2-configs/ with updated model versions (exe paths) to the deployed model in pre-release.

[CodeGat]

Update on this issue: We now have a prerelease environment, and reproducibility testing. We don't yet have the functionality to open a PR in access-om2-configs, but it is possible.

[aidanheerdegen]

For an automated check we should have the ability to nominate a small subset of specific configs to create PRs against (perhaps generate a matrix job to accomplish this). They should probably be relatively resource light so as not to burn through too many compute resources, but also so they can get through the queue in good time.

Once the config check PRs are merged we could generate PRs against alll relevant configs to update to the latest model release.