Add local ssd RAID0 startup script

[alyssa-sm]

This PR adds a startup script that will check for existing local ssds and raid if there are 2 or more.

This script has been tested manually with slurm-gcp v6 and vm-instance with centos, debian, rocky, and ubuntu.

If there are at least 2 local ssds attached, a raid device for them will be created. Using lsblk -o NAME,FSTYPE,LABEL,UUID,MOUNTPOINT shows the following.

NAME    FSTYPE            LABEL               UUID                                 MOUNTPOINT
sda                                                                                
├─sda1  ext4                                  7be27af0-f080-46f6-bdf8-bc495c775f18 /
├─sda14                                                                            
└─sda15 vfat                                  4FFB-6B16                            /boot/efi
nvme0n1 linux_raid_member ssd-test-debian-0:0 0823949c-a4f7-b097-8712-cbbb5a50066a 
└─md0   ext4              LOCALSSD            6d3ba06d-b59c-4df4-ae5e-29f49ccc5e7a /mnt/localssd
nvme0n2 linux_raid_member ssd-test-debian-0:0 0823949c-a4f7-b097-8712-cbbb5a50066a 
└─md0   ext4              LOCALSSD            6d3ba06d-b59c-4df4-ae5e-29f49ccc5e7a /mnt/localssd

When the number of local ssds present is 0 or 1, the script will exit early.

Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: TASK [Exit if less than 2 local ssd] *******************************************
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script:
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: PLAY RECAP *********************************************************************
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: localhost                  : ok=6    changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script:
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: stderr=Wed Jun 26 00:21:26 +0000 2024 Info [2370]: === start executing runner: 99-running-script-warning.sh-54ee ===
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: Wed Jun 26 00:21:26 +0000 2024 Info [2370]: === 99-running-script-warning.sh-54ee finished with exit_code=0 ===
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: Wed Jun 26 00:21:29 +0000 2024 Info [2370]: === start executing runner: install_ansible_automatic.sh-c9b1 ===
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: Wed Jun 26 00:23:28 +0000 2024 Info [2370]: === install_ansible_automatic.sh-c9b1 finished with exit_code=0 ===
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: Wed Jun 26 00:23:31 +0000 2024 Info [2370]: === start executing runner: setup-raid.yml-4df9 ===
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: Wed Jun 26 00:23:37 +0000 2024 Info [2370]: === setup-raid.yml-4df9 finished with exit_code=0 ===
Jun 26 00:23:37 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script:
Jun 26 00:23:38 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: INFO: Check status of cluster services
Jun 26 00:23:39 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script: INFO: Done setting up compute
Jun 26 00:23:39 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: startup-script exit status 0
Jun 26 00:23:39 ssdtest-computenodeset-0 google_metadata_script_runner[1555]: Finished running startup scripts.
Jun 26 00:23:39 ssdtest-computenodeset-0 systemd[1]: google-startup-scripts.service: Succeeded.

Manual Testing for rebooting.

• Deployed a blueprint that boots up a vm-instance for each of the OS as well as a Slurm-GCP v6 controller and login node.
• After the instances are setup, verified that the raid0 was set up with lsblk -o NAME,FSTYPE,LABEL,UUID,MOUNTPOINT and saved the output.
• Used the command sudo reboot to reboot the instance
• When the instances finish setting back up, verified the raid0 was re-established with lsblk -o NAME,FSTYPE,LABEL,UUID,MOUNTPOINT and compared the UUIDs of the raid0 local ssds before and after reboot.

A reboot is successful if it can successful recreate the raid device with the local ssd that were up prior to reboot. Rebooting was successful for all the instances.

Manual Testing for restarting.

• Deployed a blueprint that boots up a vm-instance for each of the OS as well as a Slurm-GCP v6 controller and login node.
• After the instances are setup, verified that the raid0 was set up with lsblk -o NAME,FSTYPE,LABEL,UUID,MOUNTPOINT.
• Manually shut down the instances and then started them back up.
• When the instances finish setting back up, verified the raid0 was created with lsblk -o NAME,FSTYPE,LABEL,UUID,MOUNTPOINT.

A restart is successful if it can setup the raid device with the local ssds. Restarting was successful for all the instances except for the Slurm-GCP instances. The current problem is that the Slurm setup verifies that Slurm is already installed and exits early which leads to the startup script not getting executed.

Cases covered by this PR:

• Centos, Rocky, Ubuntu, Debian VM instances reboot and restart
• Slurm-GCP reboot

Cases that require more thought:

• Slurm-GCP restart

[tpdownes]

I want to review the implementation further for refining. I have tested the following blueprint:

# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

---
blueprint_name: startup-vm-instance

vars:
  project_id:  ## Set project id here
  deployment_name: testfix
  region: asia-southeast1
  zone: asia-southeast1-b
  bandwidth_tier: gvnic_enabled

deployment_groups:
- group: first
  modules:
  - id: network1
    source: modules/network/vpc
  - id: script
    source: modules/scripts/startup-script
    settings:
      setup_raid: true
  - id: vm1
    source: modules/compute/vm-instance
    use:
    - network1
    - script
    settings:
      machine_type: g2-standard-24
      local_ssd_count: 2

Which is running CentOS 7. I need to test other OSes that are not EOL, and I understand there may be an issue with Debian behavior upon reboot. I have successfully tested this blueprint with:

• rebooting preserved data and is remounted ("WAI")
• power off loses data but new disks are formatted and mounted ("WAI")
• power off with preview flag to preserve data preserved data and is remounted ("WAI")

[tpdownes]

I did some research into the behavior of mdadm on modern Linux systems. They will be auto-assembled with a predictable name at /dev/md/raid_name if you supply a name during creation. Additionally, the hostname of the machine must match the hostname it was created on. In the real world, I believe there are boot-time race conditions which prevent hostname matching from being 100% reliable even on the same host. We can work around this by supplying --homehost=any during RAID creation.

If you follow these "rules" then you can rely on the system to assemble the RAID array at "/dev/md/raid_name" and ignore the numeric device names. If that file does not exist, then it means that the RAID array must be created. This occurs when the VM is booted for the first time or powered off and powered on without disabling local SSD discard.

This won't handle a Slurm-GCP power off/on cycle but it's a good first step.

[tpdownes]

4 very small changes that I'd like to see. I'm going to begin testing with those changes in place.

[tpdownes]

LGTM. I want @nick-stroud to weigh in on input variable format and then done.

[tpdownes]

We missed a change that causes the accidental inclusion of the runner in all invocations of startup-script