Migrate away from Ubuntu repositories for 22.04?

[myfirstnameispaul]

As 18.04 approaches EOL I think Mail-in-a-Box is suffering from some of the constraints that occur with the way the Ubuntu project manages its repositories.

OpenDKIM

OpenDKIM, as a a project, appears mostly dead. The current version running in Mail-in-a-Box is a 6 year old alpha of a now 3 year old beta (latest on their GitHub). The project seems weakly supported and key developers have moved on. Notably, Scott Kitterman is contributing to a fork of pydkim called dkimpy. Although dkimpy is in PyPI and lists a Launchpad home page, the project doesn't appear to be creating any PPAs or backports. However, a pip install seems like an option that may be better than the Ubuntu repo or OpenDKIM.

The issue with OpenDKIM demonstrates how new standards cannot be adopted when we the project is bound to the Ubuntu repos. RFC8463, published September 2018 states "Signers SHOULD implement and verifiers MUST implement the Ed25519-SHA256 algorithm." I am of the opinion that Ed25519 support is important because there are still many receiving servers, ISPs, and registrars that do not retrieve or server records longer than 255 characters, an issue Ed25519 handily resolves. Implementation of this standard in Mail-in-a-Box would represent a real increase in the quantity of receiving servers supporting the standard.

NGINX

Ubuntu 18.04 repos use NGINX 1.14.0, while the latest version of NGINX stable is 1.20.1. If the timing of Mail-in-a-Box OS upgrades had been different by one LTS cycle, there would have been an extended critical period of not having TLSv1.3 support, which first appeared in 1.14 of the stable branch. Options for nginx are not great. There is the "Nginx" team PPA that was historically well maintained, but has recently gone dormant at version 1.18.0. There is an nginx PPA managed by Ondřej Surý, which comes with its own considerations.

PHP

The PHP version 7.2 is EOL according to the PHP Group, and I believe it is holding back at least one major component of Mail-in-a-Box, maybe it was Nextcloud? My observation has been that nearly every popular PHP project that publishes installation instructions for Ubuntu will include adding the Ondřej Surý php PPA. However, this brings additional considerations as either some OpenSSL packages or all of them are included in the php PPA. The PPA in the past did present issues when it went through major overhaul which required significant user interaction with the install options. I have found it since to be quietly reliable for PHP packages, but ultimately if this is selected it likely requires evaluating the impact on openssl and NGINX.

Certbot

As discussed in #1971, certbot is deprecated and likely the best alternative is to use pip for installation.

SpamPD

Purely for historical and reference purposes, in #1523 it was discovered that the repo version of spampd was causing painful bug. Even though the package is updated everywhere else, from Debian on forward, Ubuntu repo maintainers cannot even offer a backport to resolve the issue for easily 20K+ users of the package.

[cdp1337]

Have any recommendations? This is a difficult situation to solve; with old packages comes stability in knowing exactly what features are supported for a long duration, but also comes with software being outdated and missing out on new features. Server administrators prefer if their systems are set-and-forget, whereas developers prefer the newest shiny.

[myfirstnameispaul]

I agree with you on this, and I am generally in the category of set-and-forget, but with the caveat that, at lest for my own servers, it support standards I determine should be supported and not exceptionally hold back upgrading stable versions of services or packages.

I opened the issue more for discussion but peppered with hints of suggestion:

Use pip install of dkimpy.

For NGNIX, my admittedly intermediate-level novice experience recommendation is to use the Ondřej Surý PPA. He is historically very good at communicating his plans, when such plans require communicating. His PPAs are very widely used, so don't usually have any bugs in them. I have been running the Nginx team PPA for a long time. I did just send an email to the maintainer in regards to the status of the PPA, but IIRC he is in the U.K. and haven't heard back anything. He's responded before. I think the Nginx team PPA might be just the debian version, or something similar, which was why I'm guessing it wasn't a huge deal for him to maintain the PPA.

For PHP use the Ondřej Surý PPA. Other than the one time I had to navigate his major change to how his PPAs were structured, the only issue was when I did not notice the PPA had added openssl packages and so updated my server, which then triggered a minor bug in NGINX related specifically to that version of OpenSSL, although the bug did not impact the services, it was unexpected and would have occurred no matter which PPA of NGINX stable I used. So I think the consideration with PHP and the Surý PPA is probably how it impacts openssl.

As a side note, I didn't just go searching for all of the features I would like to have and reverse-engineer reasons to get them into the project. There is a Postfix feature I would like to have, but based on the project I don't see it as a reason seek alternatives to the Ubuntu repos.

[Synchro]

The significant change in those repos was really that he removed openssl from his PPA. This was sensible because the base version in Ubuntu got updated to a version that was good enough, however, it caused problems because it was still older than the versions previously provided in the PPA, so necessitated a downgrade, which was confusing to many, but was in fact safe. FWIW Ondřej Surý also has an nginx PPA that may well be better supported than the official nginx one, though I'm also rather partial to this one because it has brotli support.

[myfirstnameispaul]

One thing I forgot to mention is that with the Surý PHP PPA it becomes possible to run multiple versions of PHP, to the extent that is desired, the option is at least there. And there is a downside in that Surý doesn't support versions once they are EOL, unlike the Ubuntu repos which my understanding is will receive security updates, so project updates will have to upgrade PHP versions with the PHP group schedule.

[nomandera]

I would like to suggest that containerisaiton be considered again as a solution to this issue. My experience is limited to docker but snap may be more applicable.

Previous endevours into this have been "all or nothing" but there is absolutely no reason and many benefits to deploying a container to fulfill a specific function or subsystem.

Other than this, containers would allow the user to run exactly what the developers have tested, the ability to roll back should there be an issue and should it be necessary use packages from alternate base OS.

In essence using containers in much the same logical manner as PPA to cherry pick solutions to specific problems.

[JoshData]

I really don't think I will ever have the time to a) lead the work to make any radical changes to this project and then b) take the responsibility for maintenance for it.

I'm not ruling out those sorts of changes, but someone else would have to take over as a primary maintainer of the project to do it (and the community would have to agree with that).

[cdp1337]

That too; corporate sponsorship would need to get involved for advanced maintenance of the repositories. Proxmox does a nice job where they bundle and maintain their own repo, but they have financial backing. (Additionally, there may be support in the community to pay a little to help maintain it; Google already charges for branded gmail.)

@nomandera I don't know how I'd feel about Docker for the purpose of this application, though different use-cases may provide different benefits. To me, MIAB is meant to be ran on a virtual server all by its lonesome; adding a virtual environment for a single appliance on a virtual server just seems excessive. The hypervisor should already provide rollbacks, migrations, etc.

[kiekerjan]

Instead of moving away from the Ubuntu repositories it might also be considered to not skip an LTS and try to keep up with a 2 year upgrade cycle. That way mailinabox can depend on the Ubuntu ecosphere to provide a consistent set of packages. Also, the risk of a ppa becoming stale, or even disappearing altogether, is avoided. At the same time, Mail-in-a-Box can be kept relatively up to speed with newer version of the used packages. The packages mentioned have newer versions packaged in later releases of Ubuntu LTS. e.g. nginx is 1.18 in 20.04
dkimpy even has an official ubuntu package https://packages.ubuntu.com/source/focal/dkimpy

When I look at e.g. Power mailinabox by @ddavness (just taking one, there seem to be multiple efforts to support 20.04) the changes between 18.04 and 20.04 strictly necessary to support the newer Ubuntu version seem relatively minor. However, that is only one version change. I have no idea how much work was necessary between e.g. 14.04 and 18.04, or how much will be necessary to support 22.04, so maybe I'm biased.

I'm aware of the discussion in #1751, but I feel that if it is considered to add PPAs and other sources for newer packages, it should also be considered to keep up with LTSs. That is, if you want newer versions of the packages I think you're better of depending as much as possible on LTS versions instead of a collection of different PPAs, pip installs, github sources etc.

[JoshData]

I'm extremely unlikely to agree to switch anything to a community-supported PPA, primarily for security reasons.

[ddavness]

Now that I was pinged into this conversation, my opinion is that PPA's are not exactly the best way to go. From my anecdotical experience (which, to be honest, is not much) is that once they're installed, they can be sometimes hard to remove without breaking something with apt/dpkg - thus making upgrades in-place a big pain at best.

From my experimentation with Power Mail-in-a-Box, though, I was able to install all the packages that were needed without recourse to PPA's (on Ubuntu 20.04) - though again, this was more out of necessity as the plan in the first place was to make it installable on Debian.

I do support the idea of using pip where feasible - because 1) It already comes with python, and we're already using it; 2) It's more flexible across versions (i.e. not limited to Ubuntu); 3) It is otherwise functionally equivalent to a PPA.

[myfirstnameispaul]

Great to have all of the feedback!

I've definitely revised some of my suggestions from above:

Generally, it may be best to stick with Ubuntu repos so long as they are maintained. However, when they are not maintained (e.g., spampd, certbot, etc.), finding a different source should be evaluated.

When evaluating different sources, the sources actively maintained by project maintainers should be the first consideration. For example, certbot is maintained on snap and PIP. Currently, Mail-in-a-Box does not use snap, so PIP might make more sense for certbot.

When an Ubuntu maintained package is creating some serious issue, migrating to a different source should be evaluated. As I mentioned previously, if Mail-in-a-Box were one LTS cycle different and was installed to 16.04, we would have been suffering a new "support TLSv1.3" issue every other day or maybe only once per week until migrating to 20.04 due to limitations of the Ubuntu repo version of nginx.

It should be considered that occasionally there may be sufficient reason to migrate to a new source to support a feature not otherwise available. A scenario could arise with PHP, where the Ubuntu repo version is below the minimum upstream supported version of a project, or it could be that a feature otherwise very valuable to Mail-in-a-Box as a project is available only with a newer version.

When evaluating a new source for packages not available directly from maintainers or they have limitations on their version (e.g., nginx), an alternative source should be evaluated for security. For example, the "Nginx" team PPA I previously mentioned is maintained by Thomas Ward. We can easily discover he is an Ubuntu Core Developer with positions on Ubuntu management boards, so we are trusting him already.

Packages I feel we should look for alternative sources include ones that end up not being maintained in the Ubuntu repos, and to do this even when first migrating to a new LTS. This includes certbot, and since I also feel we should move away from opendkim as the project optimistically appears to be on life support, dkimpy through PIP. And possibly spampd as that has already resulted in one critical bug that required direct involvement of the maintainer and a fix outside of the repos, which are still on the broken version, but I think that one would require direct package management (download from GitHub).

As a side note, I do not feel moving to migration every LTS is valuable enough to warrant the hassle. I, for one, would seek out a different solution for my mail. The goal for me is the time I might spend maintaining something that otherwise sits there and does its job I can put into making it better, instead of just working to have it do what it is already doing. For example, I am lagging on reconfiguring for domain sockets, although partly because I want to also figure out install and configuration of dkimpy and to include Ed25519 support, among other things.