Refresh Flatpak build workflow

[oskirby]

Description

I've been thinking of giving the flatpak build infrastructure an update to try and pave the way for proper releasing on flathub. This PR attempts to lay the groundwork to build Flatpaks using the same environment that flathub uses, and develop a workflow that can update the pip/cargo dependencies and file pull requests to flathub.

Eventually the idea is that we can have a release job that automatically pushes tagged releases to flathub, and maybe even release candidates to the flathub beta.

For a quick tour of what's changed:

• We create a new workflow that focuses just on dealing with Flatpaks. After this lands, we should probably deprecate the old Linux workflow.
• Get the Flatpaks building directly from a git reference (eg: no more scripts/linux/script.sh), this requires that:
  • We use the flatpak-cargo-generator.py script to generate a manifest for the Rust crates used by the VPN project.
  • We build the glean_parser using Flatpak manifests. This was a pain because the rpds dependency is a mixed Python/Rust project and it requires some hacks to get it building offline.
  • Remove flatpak-pydeps.yaml as it doesn't add anything of value to the build.
• Update to Qt 6.6
• Write an Appstream metainfo file which is required for submission to Flathub.
• Write some linter jobs to ensure our Flatpak manifests and Appstream files will pass Flathub submission requirements.

Some future work:

• Flathub requires us to include a <releases> element in the Appstream data, but we really don't have a single source of truth for our releases. At present, this is dumping the tags off github and gluing it together with the Github releases API. A better approach would be to host the releases file using something like this and then replace it with a URL reference.
• The release descriptions on Github are badly formed, and break the quality guidelines on Flathub (eg: too much boilerplate, includes URLs and so on). We would benefit from sitting down and re-writing our release descriptions on Github so they are better formatted so that they can be used to auto-generate the release metainfo file.
• The next step after this lands would be to submit our app to Flathub and then cobble together some automation workflows to push updates automatically whenever we cut a release on Github.
• After reflecting a bit on how Flathub wants their deployments to work, we probably don't need the taskcluster jobs to build Flatpaks. In fact, some automation to push the releases/x.y.z branches to Flathub beta would be better.

Reference

JIRA issue: VPN-6302

Checklist

• My code follows the style guidelines for this project
• I have not added any packages that contain high risk or unknown licenses (GPL, LGPL, MPL, etc. consult with DevOps if in question)
• I have performed a self review of my own code
• I have commented my code PARTICULARLY in hard to understand areas
• I have added thorough tests where needed

[hneiva]

I don't see anything concerning. I don't fully understand the logic in scripts/linux/script.sh, but I assume you have tested it out.
Just a few nitpicks, but I don't see anything that should block merging.

[hneiva]

I'm a bit worried about using ubuntu-latest. In releng stuff, we usually pin it to a specific major version.
Not a blocker, unless if you think this could cause problems in the future.

[oskirby]

I don't anticipate this would cause problems. Both the build and linter jobs are downloading and running containerized environments to do the real work, so we're not really making any significant use of the host Ubuntu environment.

[hneiva]

nitpick: Maybe move this outside the file content into a variable so it's more readable/easier to understand?

[oskirby]

The output of appstream-releases.py is a fairly messy, multiline XML file which I'm not a fan of passing around as environment variables. And this script is really just a temporary workaround until something like this PR lands - after which we can replace all of this with a static XML file that links to a hosted releases file.

The gist of the problem that led to this hack, is that we are required to provide a <releases> XML tag to pass Flatpak's submission guidelines - but we don't really have a good source of that release information. So we are trying to generate it dynamically from the git tags.

[oskirby]

The PR to mozillavpn-product-details has landed, so I have updated the PR to turn this into a static XML file that just links to the hosted releases file.

[hneiva]

should we use -9 to improve compression?

Edit: Also, how come we take in a file with extension .tar.gz, but output to .tar only?

[oskirby]

There is a little subtlety going on here, but we are making use of the -r or --append option to tar here in an attempt to minimize the possible changes to the pip package's tarball. This simply appends a record to the archive to insert a single file without touching anything else.

The downside is that this option only works on uncompressed archives, which means we have to decompress the archive first (into a plain .tar file) even though we don't need to extract the archive. Which is what these case statements are doing.

[oskirby]

The modification to this tarball is only done locally to the flatpak build directory, and the modified archive is never uploaded or stored anywhere. Therefore there isn't really any benefit to a more aggressive compression level.

[hneiva]

Awesome, thanks for explaining 😃