Add access class for access-esm repro testing

[jo-basevi]

• Added an access model class (access is the name of the payu driver for the access-esm1.5 model).

• Refactor extracting mom5 checksums to a separate file and function so it can be reused in access-om2 and access. I used just a helper function, rather than model class of mom5 with extract_checksums() method to easily allow for different checksums be added to access-om2/access models in future.

• Added a test access.out and expected checksums to test resources so it passes the existing extract_checksums() tests for each model class.

• Set default model runtime for access to 24hr for access (as the minimum runtime is day), and keep access-om2/om3 to 3hrs. This default runtime is used in the historical repro checksums tests.

• Refactored the test_bit_repro_historical a little as the test was getting long with the added logic..

• Added a start of an integration-like test for test_bit_repro_historical. This is still draft while I am finishing off adding tests. But if there's any feedback on the changes is welcome

[codecov]

Codecov Report

Attention: Patch coverage is 87.20000% with 16 lines in your changes missing coverage. Please review.

Project coverage is 54.85%. Comparing base (c9d6184) to head (13472bf).
Report is 9 commits behind head on main.

Files	Patch %	Lines
src/model_config_tests/exp_test_helper.py	47.36%	10 Missing ⚠️
src/model_config_tests/models/accessesm1p5.py	91.42%	3 Missing ⚠️
src/model_config_tests/test_bit_reproducibility.py	94.28%	2 Missing ⚠️
src/model_config_tests/util.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #28       +/-   ##
===========================================
+ Coverage   14.44%   54.85%   +40.41%     
===========================================
  Files          11       13        +2     
  Lines         547      618       +71     
===========================================
+ Hits           79      339      +260     
+ Misses        468      279      -189     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jo-basevi]

• Added more tests for test_bit_repro_historical. It's somewhat ironic having a lot of test code for tests.. The tests probably can be improved and broken up to test more smaller functions that are moved outside of test_bit_reproducibility.py and just have one or 2 integration style tests. In future, I want to test the other repro and config pytests, and other models rather than just access. Though that might be moving out of the scope of this PR.
• Added a quick improvement to the logging when the payu run fails in test_bit_repro_historical. I could add more logging, and use the logging library but I'll save that for an PR of this issue: Further Logging of pytest States #5

[CodeGat]

Wow, from all of that I only have one comment 😄 Looks great.

Have you tested the test code?

[jo-basevi]

I've tested the code with the pre-industrial configuration: https://github.com/ACCESS-NRI/access-esm1.5-configs/tree/pre-industrial, but using the executable paths from coecms: https://github.com/coecms/access-esm/blob/main/config.yaml, as the exe paths on the current pre-industrial branch weren't working.

Though I think the pre-industrial configuration is getting updated with spack executables, so I'll be able to do another test run.

I've tested it with the broken 025 access-om2 configs tests to see what logs are printed when things go wrong..

[jo-basevi]

Ok, so the spack executables in this PR: ACCESS-NRI/access-esm1.5-configs#26, pass the historical bit reproducibility test using checksums from the coecms executables as a reference.

Restart reproducibility fails, because the model is not in a good state to restart from after a model runtime of one day - it's minimum runtime might be a month (?). Talked with @aidanheerdegen about whether restart reproducibility should be tested as part of Reproducibility CI checks for configs (as they fail for the access-om2-bgc configs too), and decided that bit historical reproducibility was sufficient testing for now. So I've changed the test marker to checksum_slow, and will create a separate issue for refactoring the restart repro tests. Does that sound an ok plan?

[aidanheerdegen]

Ok, so the spack executables in this PR: ACCESS-NRI/access-esm1.5-configs#26, pass the historical bit reproducibility test using checksums from the coecms executables as a reference.

Hey that's cool!

Restart reproducibility fails, because the model is not in a good state to restart from after a model runtime of one day - it's minimum runtime might be a month (?). Talked with @aidanheerdegen about whether restart reproducibility should be tested as part of Reproducibility CI checks for configs (as they fail for the access-om2-bgc configs too), and decided that bit historical reproducibility was sufficient testing for now. So I've changed the test marker to checksum_slow, and will create a separate issue for refactoring the restart repro tests. Does that sound an ok plan?

I believe ESM1.5 has never been restart reproducible. @MartinDix mentioned that CABLE (the land model embedded in the UM in ESM1.5) also prevented restart reproducibility, so best to not have it turned on as a check. Which also means we can just use 1 day runs for historical reproducibility.

[aidanheerdegen]

Just a warning, this was a pretty light review, I didn't have time to check all the refactoring, but it looks great to me. Thanks for all the hard work!

[jo-basevi]

I reran the historical repro test with ACCESS-ESM1.5 pre-industrial config and an ACCESS-OM2 config, just to make sure the latest changes didn't break the tests.

[CodeGat]

Just one thing... 😄