set up regressions tests to be run automatically every night on all WCOSS machines

[edwardhartnett]

The regression tests involve large test files. I don't think we will ever get these large file tests running on GitHub actions.

So we need to have a cron job run the tests on develop branch, and email the results to George.

This is similar to what we now do with hpc-stack. As we develop unit tests, we will add them to the github CI system, and the regression tests will continue to be run every night.

[edwardhartnett]

@kgerheiser any progress with this?

[edwardhartnett]

Some more notes:

• THis needs to run on both the develop and public-v2 branches.
• This should check the hash on github and only run when there have been changes.
• The test should be run every 10 minutes or so, if there have been changes. This way we can see any breakage immediately.
• The results should be emailed to us, with PASSED or FAILED in the subject of the email (bonus for making the subject line green or red text, depending on pass/fail).

There is a way to automatically sort the mail into a named group with gmail. I will see if I can remember it. Then these emails will not clutter our inboxes.

[edwardhartnett]

Any progress here? It would be great to get this going this week...

[kgerheiser]

Not much. I'll make this my priority now.

[edwardhartnett]

Any progress here?

[kgerheiser]

Yes, I have a script that clones, checks the hash, builds, runs the regression tests and emails the results.

[kgerheiser]

See https://github.com/kgerheiser/UFS_UTILS/blob/feature/auto-reg-tests/reg_tests/cron.sh

[edwardhartnett]

Awesome. Is this running as a cron job on the machines @GeorgeGayno-NOAA uses for testing?

How often should it be run? Every hour perhaps?

[edwardhartnett]

How is this coming? Can we get this working at least on some machines, this week?

[GeorgeGayno-NOAA]

@kgerheiser changes were merged with #426. (Good job! @kgerheiser). And I have cron'd them on all officially supported machines. So can this issue be closed?