Use in automatic RPM building

Goal

Automatically build RPMs for project releases on GitHub.

Specific goals

You want to create a system that automatically builds new RPM packages, against a given GitHub repository. Simply declaring Version: obtained from lastversion and checking if it's newer than previously built is not sufficient to make the auto-build system future-proof.

You need to also consider that the Source0 URL structure becomes different if the upstream decides to tag a new release in an unusual way. This happens more often than you can't imagine. Release URLs are git tags dependent!

GitHub specifics

Source tarball URLs on GitHub are always git tag-specific, that is the key component of a release source tarball's URL is the git tag and only that. Standard structure:

https://github.com/author/name/archive/tag/name-tag.tar.gz

This is the default format used by lastversion also when you use --source.
lastversion makes sure that the URL you get contains a stable, well-formatted version inside it. But it's still important to understand that the varying component of URLs in between different releases on GitHub, will always be a git tag.

Example, lastversion https://github.com/apache/incubator-pagespeed-ngx:

https://github.com/apache/incubator-pagespeed-ngx/archive/v1.13.35.2-stable/incubator-pagespeed-ngx-v1.13.35.2-stable.tar.gz

Here, v1.13.35.2-stable is the release tag.

Say, you had in your spec, to make it clean and pretty:

Version: 1.13.35.2
URL: https://github.com/apache/incubator-pagespeed-ngx
Source0: %{url}/archive/v%{version}-stable.tar.gz#/incubator-pagespeed-ngx-%{version}-stable

Then you're auto-updating Version: with the help of lastversion. It will work most of the time. But any time a project's author decides on a different release tag format (e.g. v1.2.3 becomes release-1.2.3), you will have broken automatic builds.

Another specific is in the way release tarballs are extracted. For the following link:

I'm talking about the continuous automatic build of newer versions, as they are released.

tl;dr RPM recipe for building a project from GitHub

Say, we want to build new releases of lorem/ipsum from GitHub.

1. Create a Jinja-based .spec template:

Take an existing .spec where the version is hardcoded, convert it to a template, e.g. rename to .j2. Then the essential change is change hardcoded Version: and and URL to template variables, like so:

Name: ipsum
Version: {{ version }}
URL: https://github.com/lorem/ipsum
Source0: %{url}/archive/{{ spec_tag }}/%{upstream_name}-{{ spec_tag }}.tar.gz
...
%prep
%autosetup -n %{name}-%{version}
...

Because release URLs are dependent on tags, we must template them. lastversion will pass them to the template

2. Write a .yml with extra info

This is optional, but you may need to get some extra info aside from the version to make be passed to spec template, e.g. ipsum.yml:

repo: lorem/ipsum
description: |
    What a great package it is!

coupled with these lines added to the template:

%description
{{ description }}

3. Generate your spec

lastversion --format json lorem/ipsum | jinja2 --format=json ipsum.j2 --strict - > ipsum.spec

or, if you using the .yml file:

lastversion --format json /path/to/ipsum.yml | jinja2 --format=json ipsum.j2 --strict - > ipsum.spec

The result is a .spec file containing the latest release version and correct URL for it!

How does this work?

• lastversion fetches information about the latest release as a json, this includes release's fields:
  • spec_tag which is specifically prepared for .spec. The value would be, e.g. v%{version} or foo-%{version}, etc.
  • version - the well-formatted version number, e.g. 1.0.1
• we piped this data to jinja2 and specify the template file and the output file, ipsum.spec.

If you use .yml file, then lastversion takes repo: value inside it, and uses as REPO argument. Then any other keys from there are passed for templating.

Our generated .spec file is clean and ready to build:

Name: ipsum
Version: 1.0.1
URL: https://github.com/lorem/ipsum
Source0: %{url}/archive/v%{version}/%{name}-v%{version}.tar.gz
...
%prep
%autosetup -n %{name}-%{version}
...

Now time to build stuff! :-)

Example

Next time, they release a new version but this time with v1.14.0.0 tag (no stable) and the clean spec has to be:

Version: 1.14.0.0
Source0: https://github.com/apache/incubator-pagespeed-ngx/archive/v%{version}.tar.gz#/incubator-pagespeed-ngx-%{version}

See how Source0 had to be different?

So you if you want to keep your spec files clean with Version: parametrized and having it reliable, you need to pass both tag and version to "something" that generates your spec file, or replace both.

Basically, you have to always have the release tag included in your SourceX tag URL somehow just because GitHub source URLs have to include the tag always.

How I solve it have a jinja template for spec, and then have:

%global upstream_version {{ version }} 
%global upstream_name incubator-pagespeed-ngx
URL: https://github.com/apache/incubator-pagespeed-ngx
Source0: %{url}/archive/{{ spec_tag }}/%{upstream_name}-{{ spec_tag }}.tar.gz
....
%autosetup -n %{upstream_name}-{{ spec_tag_no_v }}

Where spec_tag is what I get from running lastversion --format json REPO | jq -r .spec_tag and spec_tag_no_v is just the same with leading v stripped (because of how GitHub packages are).

lastversion --format json  apache/incubator-pagespeed-ngx | jq -r .spec_tag

In that case, lastversion says it's v%{upstream_version}-stable.

So the generated spec file is "human-friendly":

%global upstream_version 1.13.35.2
%global upstream_name incubator-pagespeed-ngx
URL: https://github.com/apache/incubator-pagespeed-ngx
Source0: %{url}/archive/v%{upstream_version}-stable/%{upstream_name}-v%{upstream_version}-stable.tar.gz
....
%autosetup -n %{upstream_name}-%{upstream_version}-stable

The v%{version} maybe would make more sense for most users maybe, but I am building modules for NGINX, so I mostly have things like:

Version: %{main_version}.%{upstream_version}

So the spec_tag returned by lastversion is the key point to the reliability, considering the human inconsistency when they file releases... I haven't documented it elsewhere because not sure if there is a better way to return the tag info for consuming in spec files.