Automation for updating templates

[AkihiroSuda]

It is really hard for me to create a PR like #1236 to update the template image digests.

We have to have a tool for updating these templates automatically.
The tool must retain comment lines and indentation styles in the YAMLs.

---

Frequently updated images:

• Ubuntu Add hack/update-template-ubuntu.sh #2702
• Debian Refactor update-template-ubuntu.sh and add update-template-debian.sh #2731
• ArchLinux
• CentOS Stream

Less frequent ones:

• AlmaLinux
• Rocky Linux
• Oracle Linux
• Fedora
• openSUSE Leap

[afbjorklund]

Maybe also some means of sharing them, with some kind of FROM system ?

• Inherit system images and share them between examples #824

[AkihiroSuda]

Maybe also some means of sharing them, with some kind of FROM system ?

• Inherit system images and share them between examples #824

Yes, but that is a separate issue

[jlm0x017]

Do you already have the link to the updated image? Or is that step needed as well?
I expect you grab the checksum provided at the source, rather than generating it yourself, but please do confirm.

[AkihiroSuda]

Do you already have the link to the updated image?

No, e.g., we have to detect the latest version 20230124-1270 from https://cloud.debian.org/images/cloud/bullseye/ , but I'm not sure what is the robust way to do this.

w3m | grep might be enough, but seriously we should also consider adopting some machine learning stuff.

[lobshunter]

What about a naive bash/python script to replace placeholder strings in yaml? Something like:

ubuntu_image = `w3m | grep -e "xxx"`
sed -i "s/UBUNTU_IMAGE/$ubuntu_image/g" ubuntu.yaml

The robustness solely relies on consistent file naming of upstreams(counting on hyrum's law). Sadly I didn't find any public tool to retrive latest released images.

[AkihiroSuda]

I'm now planning to use yq

[balajiv113]

How about using libosinfo (osinfo-db os) ??

I could see their db has info of different os variants
https://gitlab.com/libosinfo/osinfo-db/-/tree/main/data/os

We might need to write a python wrapper on top of this library (the tool osinfo-db is not giving out info on image download URL's by architecture that is present in the xml file)

[afbjorklund]

yq can read xml too

yq -p xml -P

[afbjorklund]

The library and database are licensed under the terms of the GNU LGPL version 2 or later.

https://libosinfo.org/

[AkihiroSuda]

osinfo-db

Doesn't seem to contain permalinks: https://gitlab.com/libosinfo/osinfo-db/-/blob/ea8a7974a1f7189953c80fa9b1478b1ff8a75f8e/data/os/ubuntu.com/ubuntu-23.04.xml.in

    <image arch="x86_64" format="qcow2" cloud-init="true">
      <url>https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img</url>
    </image>

[balajiv113]

yq can read xml too

True, but if we can use API it would be great. Else with yq we might need to read all xml files under each folder that we are interested in. With API i think it will be more managable

GNU LGPL version 2 or later

I thought since we are going to use this more of a build tool (Mostly a github actions workflow) this should not be a problem.

[afbjorklund]

It was mostly referring to the "and database", most of the tools actually seem to be GPL v2 (and to require glib)

[balajiv113]

Doesn't seem to contain permalinks

True :(
Supported examples are as below

• almalinux-8.yaml
• almalinux-9.yaml
• alpine.yaml (we can support this as its our variant of alpine)
• archlinux.yaml (Not present)
• centos-stream-8.yaml
• centos-stream-9.yaml
• debian.yaml (Present, Using only latest version)
• fedora.yaml
• opensuse.yaml (Present, But qcow2 image not present)
• oraclelinux-8.yaml (Not present)
• oraclelinux-9.yaml (Not present)
• rocky-8.yaml
• rocky-9.yaml
• ubuntu.yaml (Present, Using only latest version)
• ubuntu-lts.yaml (Present, Using only latest version)
• experimental/opensuse-tumbleweed.yaml (Present, But qcow2 image not present)

[AkihiroSuda]

I guess we can consider using GPT

[afbjorklund]

For Ubuntu, this is implemented in the ironically named "simple streams" (it's 14M):

sudo apt install simplestreams ubuntu-keyring

sstream-query --json --max=1 --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson release='noble' ftype='disk1.img' | jq -r '.[] | [.item_url,.arch,.sha256]'

https://philroche.net/2018/02/12/ubuntu-cloud-images-and-how-to-find-the-most-recent-cloud-image-part-1-of-3/

The JSON+GPG file is: http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson

---

There is also a highlevel command:

sudo snap install image-status

$ image-status cloud-release
focal    amd64  20240626  disk1.img
jammy    amd64  20240627  disk1.img
mantic   amd64  20240619  disk1.img
noble    amd64  20240622  disk1.img

Where "disk1.img" is the old spelling of QCOW.

[AkihiroSuda]

Thanks @norio-nomura 🎉

• Add hack/update-template-ubuntu.sh #2702

[AkihiroSuda]

For Debian, probably we can parse this JSON: https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.json

sha512 isn't encoded in hex though: "cloud.debian.org/digest": "sha512:2oTWCdfsVkXa4d9QPqcgN7KoMUAdG0LOLn7CqEC2mfB8qK6mMIU6PVQwg5Jowr0ze+RdiUmCZMNqm14ShyxZ7g"

Probably it should be just grepped from https://cloud.debian.org/images/cloud/bookworm/latest/SHA512SUMS instead.

[AkihiroSuda]

For ArchLinux, curl -fsSL https://gitlab.archlinux.org/api/v4/projects/archlinux%2Farch-boxes/packages | jq '.[-1].version' can be used for retrieving the latest version

[norio-nomura]

sha512 isn't encoded in hex though: "cloud.debian.org/digest": "sha512:2oTWCdfsVkXa4d9QPqcgN7KoMUAdG0LOLn7CqEC2mfB8qK6mMIU6PVQwg5Jowr0ze+RdiUmCZMNqm14ShyxZ7g"

It appears that this is a base64-encoded hash binary with the trailing "==" removed.
The following steps will convert it into hex-encoded format.

$ debian_sha512="2oTWCdfsVkXa4d9QPqcgN7KoMUAdG0LOLn7CqEC2mfB8qK6mMIU6PVQwg5Jowr0ze+RdiUmCZMNqm14ShyxZ7g"
$ echo "${debian_sha512}=="|base64 -d|xxd -p -c -
da84d609d7ec5645dae1df503ea72037b2a831401d1b42ce2e7ec2a840b699f07ca8aea630853a3d5430839268c2bd337be45d89498264c36a9b5e12872c59ee