Build RL8+OFED image in CI

[MoteHue]

Uses OFED for both RockyLinux 8 and 9 image builds.

• Fixes fact gathering in the ofed role; facts were only gathered once due to smart gathering which happens before updating host packages. As such, the ofed_distro_version was e.g. 8.9 due to the base image despite the dnf update step having upgraded the distro to something later.
• Changes both RL8 and RL9 to LTS version of OFED (now this is supported for RL9).
• Fixes bug where package update wouldn't correctly trigger a reboot.
• Image build workflow now always runs R8+OFED, R9+OFED, RL9+OFED+CUDA versions.
• Test workflow always runs on RL8+OFED and RL9+OFED images.
• Changes skeleton, caas and stackhpc ansible.cfg to fail if any inventory can't be parsed.

[MoteHue]

Might want to switch to forcing a reboot. /var/run/reboot-required isn't populated, despite a slightly newer kernel getting installed:

[rocky@openhpc-ofed-rl8-240814-1513-eeb1f9fc ~]$ dnf list --installed kernel
Installed Packages
kernel.x86_64                                                                                                                                         4.18.0-553.el8_10                                                                                                                                              @anaconda
kernel.x86_64                                                                                                                                         4.18.0-553.16.1.el8_10                                                                                                                                         @baseos  

[jovial]

Might want to switch to forcing a reboot. /var/run/reboot-required isn't populated, despite a slightly newer kernel getting installed:

[rocky@openhpc-ofed-rl8-240814-1513-eeb1f9fc ~]$ dnf list --installed kernel
Installed Packages
kernel.x86_64                                                                                                                                         4.18.0-553.el8_10                                                                                                                                              @anaconda
kernel.x86_64                                                                                                                                         4.18.0-553.16.1.el8_10                                                                                                                                         @baseos  

Might want to switch to forcing a reboot. /var/run/reboot-required isn't populated, despite a slightly newer kernel getting installed:

[rocky@openhpc-ofed-rl8-240814-1513-eeb1f9fc ~]$ dnf list --installed kernel
Installed Packages
kernel.x86_64                                                                                                                                         4.18.0-553.el8_10                                                                                                                                              @anaconda
kernel.x86_64                                                                                                                                         4.18.0-553.16.1.el8_10                                                                                                                                         @baseos  

If I understood your point correctly, you were suggesting that why might want to reboot prior to building ofed? I don't think this is necessary as we can compile the kernel modules for a kernel we are not running (as long as we have the kernel headers). The new kernel and modules will be loaded when we boot the new image.

Or is this during the ansible run of site.yml afterwards?

[sjpb]

necessary as we can compile the kernel modules for a kernel we are not running (as long as we have the kernel headers). The new kernel and modules will be loaded when we boot the new image.

Hmm, possibly, but to me that seems too risky/confusing. Especially given we may be e.g. installing CUDA afterwards which has kernel-* and ofed dependencies. We have to reboot on selinux changes anyway, so we are expecting a reboot somewhere during the build process. We just don't want to do it twice, if possible.

I think doing this is appropriate:

1. rebooting if the update step has changed, rather than just if the reboot sentinel file exists (i.e. don't trust latter as it's been wrong at least once, probably a packaging bug/oversight I guess)
2. always dumping facts (b/c meta: can't be conditional)
3. always rerunning setup (b/c 2.)

[sjpb]

Actually there's another reason: boostrap.yml runs both from site.yml and from fatimage.yml, which is good (as it reduces code duplication). So really, if we need a reboot we should do it, rather than having to determine we're running in a build and therefore can ignore that as we will get a "reboot" when creating an instance with the image ...

Make sense?

[jovial]

Actually there's another reason: boostrap.yml runs both from site.yml and from fatimage.yml, which is good (as it reduces code duplication). So really, if we need a reboot we should do it, rather than having to determine we're running in a build and therefore can ignore that as we will get a "reboot" when creating an instance with the image ...

Make sense?

Yep, I missed that you run this against live nodes too. That makes more sense. Isn't /var/run/reboot-required a Debian/Ubuntuism? Redhat seem to suggest using: https://access.redhat.com/solutions/27943

[MoteHue]

Isn't /var/run/reboot-required a Debian/Ubuntuism?

Looks like you're right, explains why it wasn't working then :P
I'll get this to always reboot, as per Steve's reasoning.

[jovial]

Seems to match what steve specified to me 👍

[sjpb]

@MoteHue @jovial this has been approved but the first comment in the PR makes it look like this is draft, can someone clarify/set status/comments appropriately please?

[MoteHue]

@MoteHue @jovial this has been approved but the first comment in the PR makes it look like this is draft, can someone clarify/set status/comments appropriately please?

It's no longer a draft

[sjpb]

Checked at 53baab2 that a reboot didn't happen after the CI reimage - i.e the conditional on the reboot appears to be working OK again.

[sjpb]

Build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/10736952422

[sjpb]

Checked reboot did happen during image build above

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[jovial]

Logic looks good to me