Improve Azure image support (#436)

* Add support for specifying image payment plans when creating VMs or VMSS.
  This allows using certain images which require payment plan information.
  For example, the Rocky Linux image in Azure requires a plan name to be
  provided, even though that plan is the "free" plan.
* Add support for using custom Azure images with Muchos.
* Make the cloud-init file name configurable.
* Add optional parameters for the proxy image configuration (image name,
  payment plan, cloud-init filename).
* Allow specifying the OS disk size for Azure VMs.
* Remove the deprecated `warn` attribute when using the `shell` module.

README.md

@@ -165,13 +165,13 @@ Under the `azure` section, edit following values as per your configuration:
 * `subnet` to provide a name for the subnet within which the cluster resources will be deployed
 * `use_multiple_vmss` allows you to configure VMs with different CPU, memory, disk configurations for leaders and workers. To
   know more about this feature, please follow the [doc](docs/azure-multiple-vmss.md).
-* `azure_image_reference` allows you to specify the CentOS image SKU in the format as shown below. To configure
-  CentOS 8.x, please follow [these steps](docs/azure-image-reference.md).
+* `azure_image_reference` allows you to specify the Azure image SKU in the format as shown below.
   ```bash
-  offer|publisher|sku|version|
-  Ex: CentOS|OpenLogic|7_9|latest|
+  offer|publisher|sku|version|image_id|
+  Ex: CentOS|OpenLogic|7_9|latest||
   ```
-* `azure_proxy_image_reference` allows you to specify the CentOS image SKU that will be used for the optional proxy machine.
+  For more information on using other images, refer to [Azure images](docs/azure-image-reference.md).
+* `azure_proxy_image_reference` allows you to specify the Azure image SKU that will be used for the optional proxy machine.
   If this property is not specified, then the value of `azure_image_reference` will be used instead.
 * `numnodes` to change the cluster size in terms of number of nodes deployed
 * `data_disk_count` to specify how many persistent data disks are attached to each node and will be used by HDFS.

ansible/roles/azure/tasks/create_multiple_vmss.yml

@@ -30,7 +30,23 @@
     file: "{{ deploy_path }}/conf/azure_multiple_vmss_vars.yml"
     name: azure_multiple_vmss_vars
 
+- name: "Create image plan related facts"
+  set_fact:
+    azure_image_plan_name: "{{ azure_image_plan.split('|')[0] }}"
+    azure_image_plan_dict:
+      name: "{{ azure_image_plan.split('|')[0] }}"
+      product: "{{ azure_image_plan.split('|')[1] }}"
+      publisher: "{{ azure_image_plan.split('|')[2] }}"
+
 - name: Create Scale Set
+  vars:
+    image_offer: "{{ azure_image_reference.split('|')[0] }}"
+    image_publisher: "{{ azure_image_reference.split('|')[1] }}"
+    image_sku: "{{ azure_image_reference.split('|')[2] }}"
+    image_version: "{{ azure_image_reference.split('|')[3] }}"
+    image_id: "{{ azure_image_reference.split('|')[4] }}"
+    accnet_capable: "{{ True if item.sku in accnet_capable_skus else False }}"
+    osdisk_sku: "{{ 'Premium_LRS' if item.sku in premiumio_capable_skus else 'Standard_LRS' }}"
   azure.azcollection.azure_rm_virtualmachinescaleset:
     resource_group: "{{ resource_group }}"
     location: "{{ location }}"
@@ -49,6 +65,7 @@
     upgrade_policy: Manual
     tier: Standard
     managed_disk_type: "{{ osdisk_sku }}"
+    os_disk_size_gb: "{{ os_disk_size_gb if os_disk_size_gb else omit }}"
     os_disk_caching: ReadWrite
     enable_accelerated_networking: "{{ accnet_capable }}"
     image:
@@ -57,23 +74,16 @@
       sku: "{{ image_sku if image_sku else omit }}"
       version: "{{ image_version if image_version else omit }}"
       id: "{{ image_id if image_id else omit }}"
+    plan: "{{ azure_image_plan_dict if azure_image_plan_name else omit }}"
     data_disks: |
       {%- set data_disks = [] -%}
       {%- for lun in range(item.data_disk_count) -%}
         {%- set _ = data_disks.append({'lun': lun, 'disk_size_gb': item.data_disk_size_gb, 'managed_disk_type': item.data_disk_sku, 'caching': item.data_disk_caching|default('ReadOnly') }) -%}
       {%- endfor -%}
       {{ data_disks }}
-    custom_data: "{{ lookup('file', 'cloud-init.yml') }}"
+    custom_data: "{{ lookup('file', azure_image_cloud_init_file) if azure_image_cloud_init_file }}"
   with_items:
     - "{{ azure_multiple_vmss_vars.vars_list }}"
-  vars:
-    image_offer: "{{ azure_image_reference.split('|')[0] }}"
-    image_publisher: "{{ azure_image_reference.split('|')[1] }}"
-    image_sku: "{{ azure_image_reference.split('|')[2] }}"
-    image_version: "{{ azure_image_reference.split('|')[3] }}"
-    image_id: "{{ azure_image_reference.split('|')[4] }}"
-    accnet_capable: "{{ True if item.sku in accnet_capable_skus else False }}"
-    osdisk_sku: "{{ 'Premium_LRS' if item.sku in premiumio_capable_skus else 'Standard_LRS' }}"
   register: _create_clusters
   async: 600
   poll: 0

ansible/roles/azure/tasks/create_optional_proxy.yml

@@ -60,12 +60,21 @@
     enable_accelerated_networking: "{{ True if azure_proxy_host_vm_sku in accnet_capable_skus else False }}"
   when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None
 
-- name: Create azure proxy virtual machine # noqa schema[tasks]
+- name: "Create image plan related facts"
+  set_fact:
+    azure_proxy_image_plan_name: "{{ azure_proxy_image_plan.split('|')[0] }}"
+    azure_proxy_image_plan_dict:
+      name: "{{ azure_proxy_image_plan.split('|')[0] }}"
+      product: "{{ azure_proxy_image_plan.split('|')[1] }}"
+      publisher: "{{ azure_proxy_image_plan.split('|')[2] }}"
+
+- name: Create azure proxy virtual machine
   vars:
     image_offer: "{{ azure_proxy_image_reference.split('|')[0] }}"
     image_publisher: "{{ azure_proxy_image_reference.split('|')[1] }}"
     image_sku: "{{ azure_proxy_image_reference.split('|')[2] }}"
     image_version: "{{ azure_proxy_image_reference.split('|')[3] }}"
+    image_id: "{{ azure_proxy_image_reference.split('|')[4] }}"
     osdisk_sku: "{{ 'Premium_LRS' if azure_proxy_host_vm_sku in premiumio_capable_skus else 'Standard_LRS' }}"
   azure_rm_virtualmachine:
     resource_group: "{{ resource_group }}"
@@ -78,16 +87,19 @@
     ssh_public_keys:
       - path: /home/{{ cluster_user }}/.ssh/authorized_keys
         key_data: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
+    os_disk_size_gb: "{{ os_disk_size_gb if os_disk_size_gb else omit }}"
     os_disk_caching: ReadWrite
     image:
       offer: "{{ image_offer if image_offer else omit }}"
       publisher: "{{ image_publisher if image_publisher else omit }}"
       sku: "{{ image_sku if image_sku else omit }}"
       version: "{{ image_version if image_version else omit }}"
+      id: "{{ image_id if image_id else omit }}"
+    plan: "{{ azure_proxy_image_plan_dict if azure_proxy_image_plan_name else omit }}"
     managed_disk_type: "{{ osdisk_sku }}"
     data_disks:
       - lun: 0
         disk_size_gb: 64
         managed_disk_type: "{{ data_disk_sku }}"
-    custom_data: "{{ lookup('file', 'cloud-init.yml') }}"
+    custom_data: "{{ lookup('file', azure_proxy_image_cloud_init_file) if azure_proxy_image_cloud_init_file }}"
   when: azure_proxy_host is defined and azure_proxy_host and azure_proxy_host != None

ansible/roles/azure/tasks/create_vmss.yml

@@ -33,12 +33,28 @@
     luns_dict: "{{ luns_dict | default ([]) + [{ 'lun': item, 'disk_size_gb': disk_size_gb, 'managed_disk_type': data_disk_sku, 'caching': data_disk_caching } ] }}"
   with_sequence: start=0 end={{ data_disk_count-1 if data_disk_count > 0 else 0 }}
 
+- name: "Create image plan related facts"
+  set_fact:
+    azure_image_plan_name: "{{ azure_image_plan.split('|')[0] }}"
+    azure_image_plan_dict:
+      name: "{{ azure_image_plan.split('|')[0] }}"
+      product: "{{ azure_image_plan.split('|')[1] }}"
+      publisher: "{{ azure_image_plan.split('|')[2] }}"
+
 - name: Set single placement group to correct value
   set_fact:
     single_placement_group: False
   when: numnodes > 100
 
 - name: Create Scale Set
+  vars:
+    image_offer: "{{ azure_image_reference.split('|')[0] }}"
+    image_publisher: "{{ azure_image_reference.split('|')[1] }}"
+    image_sku: "{{ azure_image_reference.split('|')[2] }}"
+    image_version: "{{ azure_image_reference.split('|')[3] }}"
+    image_id: "{{ azure_image_reference.split('|')[4] }}"
+    accnet_capable: "{{ True if vm_sku in accnet_capable_skus else False }}"
+    osdisk_sku: "{{ 'Premium_LRS' if vm_sku in premiumio_capable_skus else 'Standard_LRS' }}"
   azure.azcollection.azure_rm_virtualmachinescaleset:
     resource_group: "{{ resource_group }}"
     location: "{{ location }}"
@@ -56,6 +72,7 @@
     upgrade_policy: Manual
     tier: Standard
     managed_disk_type: "{{ osdisk_sku }}"
+    os_disk_size_gb: "{{ os_disk_size_gb if os_disk_size_gb else omit }}"
     os_disk_caching: ReadWrite
     enable_accelerated_networking: "{{ accnet_capable }}"
     single_placement_group: "{{ single_placement_group | default(omit) }}"
@@ -64,15 +81,10 @@
       publisher: "{{ image_publisher if image_publisher else omit }}"
       sku: "{{ image_sku if image_sku else omit }}"
       version: "{{ image_version if image_version else omit }}"
+      id: "{{ image_id if image_id else omit }}"
+    plan: "{{ azure_image_plan_dict if azure_image_plan_name else omit }}"
     data_disks: "{{ luns_dict if data_disk_count > 0 else omit }}"
-    custom_data: "{{ lookup('file', 'cloud-init.yml') }}"
-  vars:
-    image_offer: "{{ azure_image_reference.split('|')[0] }}"
-    image_publisher: "{{ azure_image_reference.split('|')[1] }}"
-    image_sku: "{{ azure_image_reference.split('|')[2] }}"
-    image_version: "{{ azure_image_reference.split('|')[3] }}"
-    accnet_capable: "{{ True if vm_sku in accnet_capable_skus else False }}"
-    osdisk_sku: "{{ 'Premium_LRS' if vm_sku in premiumio_capable_skus else 'Standard_LRS' }}"
+    custom_data: "{{ lookup('file', azure_image_cloud_init_file) if azure_image_cloud_init_file }}"
   tags: create_vmss
 
 # SECTION 4: Automatically populate entries in the hosts file and in the muchos.props file, based on the VMSS node details

ansible/roles/proxy/tasks/get-asf-mirror.yml

@@ -2,7 +2,6 @@
   shell: set -o pipefail && curl -sk https://www.apache.org/dyn/closer.cgi?as_json | jq -r .preferred
   args:
     executable: bash
-    warn: no
   register: apache_mirror
   retries: 10
   delay: 10

conf/muchos.props.example

@@ -127,14 +127,30 @@ use_multiple_vmss = False
 # Note that for multiple-VMSS deployments, this setting can be overridden at a 
 # per-VMSS level by specifying vmss_priority in the conf/azure_multiple_vmss_vars.yml file.
 vmss_priority = None
-# Azure image reference defined as a pipe-delimited string in the format offer|publisher|sku|version|
+# Azure image reference defined as a pipe-delimited string in the format offer|publisher|sku|version|image_id|
 # Please refer 'Launching an Azure cluster' section of the README before making changes
-azure_image_reference = CentOS|OpenLogic|7_9|latest|
-# Azure image reference defined as a pipe-delimited string in the format offer|publisher|sku|version|
+azure_image_reference = CentOS|OpenLogic|7_9|latest||
+# Image payment plan information - values required only if the image requires payment plan info.
+# The format of this configuration line is plan_name|product|publisher|
+azure_image_plan = |||
+# Cloud init file to use when creating VMs with the above image reference.
+# Currently, a CentOS 7.9 specific cloud-init file is used. In the future, different files can be used
+# as needed for different images like Alma / Rocky / Fedora.
+azure_image_cloud_init_file = cloud-init-centos79.yml
+# Azure image reference defined as a pipe-delimited string in the format offer|publisher|sku|version|image_id|
 # This is the image that will be used for the proxy machine (if specified by azure_proxy_host). If
 # this is not set, then the value of azure_image_reference will be used on the proxy.
 # Please refer 'Launching an Azure cluster' section of the README before making changes
-#azure_proxy_image_reference = CentOS|OpenLogic|7_9|latest|
+#azure_proxy_image_reference = CentOS|OpenLogic|7_9|latest||
+# Proxy image payment plan information - required only if the proxy image requires payment plan info.
+# If this is not set, the value of azure_image_plan will be used on the proxy.
+# The format of this configuration line is plan_name|product|publisher|
+# azure_proxy_image_plan = |||
+# Optional cloud init file to use when creating VMs with the above proxy image reference. If this is not set,
+# the value of azure_image_cloud_init_file will be used.
+# azure_proxy_image_cloud_init_file =
+# Optional OS disk size in GB. If not specified, the size of the OS disk will be as defined in the VM image
+# os_disk_size_gb = 48
 # Size of the cluster to provision.
 # A virtual machine scale set (VMSS) with these many VMs will be created.
 # The minimum allowed size for this is 3 nodes for non-HA & 4 nodes for HA setup

docs/azure-image-reference.md

@@ -1,25 +1,68 @@
-Configure Azure image with CentOS 8.x
---------------------------------------
+# Azure VM images
+Muchos can be configured to use any Azure Marketplace image, including those that need a payment plan. In addition, Muchos can be configured to use a custom VM image which has been uploaded to an Azure subscription, or a VM image which exists in a shared image gallery.
+
+# Cluster image configuration
+The sections below describe the various image related configurations in `muchos.props`.
+
+## azure_image_reference
+`azure_image_reference` is a pipe-delimited string in the format `offer|publisher|sku|version|image_id|`. The trailing pipe character is intentional.
+
+* For Azure Marketplace images, the values for the fields `offer|publisher|sku|version` can be obtained from the Azure portal, or by using the Azure CLI commands as shown later. For example, the CentOS 7.9 image currently used as the default in Muchos is specified as:
+
+    `CentOS|OpenLogic|7_9|latest||`
+
+  In the above case, since it's a marketplace image, the last value of `image_id` is empty.
+
+* It is also possible to use a [custom Azure image](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/imaging) with Muchos. For using an image from an Azure Compute Gallery ("Shared Image Gallery"), the full resource ID of the image should be specified for `image_id` and the other fields should not be specified. For example:
+
+    `||||//subscriptions/AZURE_SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Compute/galleries/SHARED_GALLERY_NAME/images/SHARED_IMAGE_NAME/versions/SHARED_IMAGE_VERSION|`
+
+  For more information on creating an image in an Azure compute gallery, see [Tutorial: Create a custom image of an Azure VM with the Azure CLI](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/tutorial-custom-images). Some Linux distributions like Fedora make available [a VHD file specifically for Azure](https://fedoraproject.org/cloud/download) which can then be [converted to a fixed-size disk](https://learn.microsoft.com/en-us/azure/virtual-machines/windows/prepare-for-upload-vhd-image#convert-the-virtual-disk-to-a-fixed-size-vhd) and uploaded to Azure for creating a custom image. In the case of the Fedora image, the `Fedora-Cloud-Base-Azure-39-1.5.x86_64.vhd` needs to be resized to a fixed-size VHD of size 6,442,450,944 bytes before uploading to Azure.
+
+* For legacy managed images (not from a gallery), the full resource ID of the image should be specified in place of `image_id`, and the other fields should not be specified. For example:
+
+    `||||/subscriptions/AZURE_SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Compute/images/CUSTOM_IMAGE_NAME|`
+
+  For more information on legacy managed images see [Create a legacy managed image of a generalized VM in Azure](https://learn.microsoft.com/en-us/azure/virtual-machines/capture-image-resource).
+
+## azure_image_plan
+`azure_image_plan` is only needed when working with images which require payment plan information to be supplied when a VM or VMSS is being created using that image. The format of this configuration is `plan_name|product|publisher|`. Plan information for the images published by a given publisher can easily be queried by using the Azure CLI. For example, to query the plan information for a Rocky Linux image in Azure:
+
+    `az vm image show --urn "erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:latest" --query "plan"`
+
+Then using that information, `azure_image_plan` can be configured as below in muchos.props:
+
+    `azure_image_plan = free|rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|`
+
+## azure_image_cloud_init_file
+`azure_image_cloud_init_file` is used to optionally specify the name of a cloud-init file to be used. Only specify the filename here, and make sure that the file exists under the `ansible/roles/azure/files` directory in this repo.
+
+# Proxy image configuration
+If needed, there are optional configurations to allow using a different image for the optional proxy. The following configurations follow exactly the same formats as their cluster image counterparts.
+* `azure_proxy_image_reference`
+* `azure_proxy_image_plan`
+* `azure_proxy_image_cloud_init_file`
+
+By default, these configurations are commented out in the muchos.props file, and the corresponding values from the cluster image configurations are used.
+
+# Other useful commands
+You can run the below Azure CLI command to determine the list of SKU's available for a given product and publisher in a given region:
 
-To configure CentOS 8.x, simply modify the `sku` as shown below. This will install CentOS 8.2 Generation 2 Azure VMs. To know more about Generation 2 VMs on Azure, please visit the [doc](https://docs.microsoft.com/en-us/azure/virtual-machines/generation-2)
-```bash
-CentOS|OpenLogic|8_2-gen2|latest|
-```
-Run the Azure CLI command to determine the list of SKU's available for CentOS in a given region
 ```bash
-az vm image list-skus -l <region> -f CentOS -p OpenLogic -o table
+az vm image list-skus -l <region> -f AlmaLinux -p AlmaLinux -o table
 ```
-For illustration, provided a sample output that displays the sku list from `westus2` region. The sku name `8_1, 8_2` implicitly refers to Generation 1 Azure VMs.
+For illustration, provided a sample output that displays the sku list from `westus2` region. The sku name `8-gen1, 8-gen2` refer to [Azure VMs generations](https://learn.microsoft.com/en-us/azure/virtual-machines/generation-2). 
+
 ```bash
-$ az vm image list-skus -l westus2 -f CentOS -p OpenLogic -o table
+$ az vm image list-skus -l westus2 -f AlmaLinux -p AlmaLinux -o table
 Location    Name
---------   ------
-westus2     7_8-gen2
-westus2     8.0
-westus2     8_0-gen2
-westus2     8_1
-westus2     8_1-gen2
-westus2     8_2
-westus2     8_2-gen2
+----------  --------
+westus2     8-gen1
+westus2     8-gen2
+westus2     8_4
+westus2     8_4-gen2
+westus2     8_5
+westus2     8_5-gen2
+westus2     9-gen1
+westus2     9-gen2
 ```
-

docs/azure-ubuntu-1804.md

@@ -1,7 +1,7 @@
 Tips for running Muchos for Azure under Ubuntu 18.04
 ----------------------------------------------------
 
-Muchos sets up a Centos cluster, but it does not have to run in Centos.  If you
+Muchos sets up a Centos cluster by default, but it does not have to run in Centos.  If you
 wish to run Muchos under Ubuntu 18.04 and have it set up an Azure cluster, then
 the following steps can get you on your way.