Should Paketo consider a backport policy?

[ryanmoran]

A recent RFC has sparked a renewed conversation about the expectations users might have about keeping their buildpacks up-to-date while limiting risk of consuming updates that break their build.

To start the discussion, I'd like to hear what some of those expectations are.

I think it might also make sense to take the time to review what types of changes were included in what types of releases.

[fg-j]

A backport policy would increase release / branch management complexity, or at least require some additional upfront automation cost. Are Paketo users requesting backports right now? Do we think that some potential users opt not to use Paketo buildpacks because we don't do backports? I'd want answers to those questions before we decide to invest in the additional complexity of backporting.

[sophiewigmore]

I'm with @fg-j on this one. While I broadly support what's outlined in the RFC, I don't think it would be worth the investment of time/additional complexity without having users expressing interest in backporting.

[dmikusa]

I think what the RFC is asking seems reasonable. My read of it is that the request is to change how we tag images, making it similar to what you see with most docker images.

For example, I can ask for buildpack-a:3 and get the latest version on the 3.x branch or buildpack-a:3.3 and get the latest patch release on the 3.3.x branch.

My main concern with this RFC is that it goes against one of the goals of buildpacks, which is to push folks to update. If we start tagging images to buildpack-a:3.3 a user could quickly fall behind and miss critical security updates, which are not going to come out in a patch release (at least for Java-related buildpacks, we bump minor version for dependency updates). I don't want users to have a false sense of security.

I think a reasonable compromise would be to tag images only with the major branch. That gives you all the benefits of security updates, but would isolate you from large and breaking changes (like removing dependencies).

I didn't get the impression the RFC is asking to manage and backport changes to branches. I could see where, once we tag images as requested in the RFC, the conversation could turn to backports. If you have those branches, then why not? But tagging images in this way is drastically different from managing code branches & having support lifetime policies people can count on.

I would be strongly -1 for anything regarding backports. I think with the rate at which buildpacks move, that would be nearly impossible to manage at the code level. I feel like it would also be complicated for users to understand. What is supported? What isn't? What we have now is much easier to follow. Updates & bug fixes always go out in a new release, and the latest release always has the latest updates.

---

I would be curious to understand more about possible use cases for both the RFC & backports though. Is it the buildpack that people want to hold back (i.e. changes to the actual buildpack code) or is it the dependencies that buildpacks install people want to hold back?

Right now we have a very tight coupling between the two. Updating buildpacks will get you updated dependencies. It's not easy to hold back a dependency, besides holding back the buildpack. If users need more control over dependencies, perhaps we can solve that by decoupling the two and, in the process, avoid a complicated branching/backporting strategy.

[loewenstein]

Sorry for getting at this a little later than I hoped for.

Let me first of all strongly point out that the recent RFC indeed is limited to additional tagging and doesn't request actual backporting.

I also agree however, that the RFC might trigger a discussion on backporting. Actually, I triggered this discussion in a short conversation with @ryanmoran over Slack.

Looking at semver.org as a consumer, I would like to have some lead time to adapt to major version bumps, so I can plan for the efforts involved. Similarly, but to a lesser extent, I would like to have some lead time to adapt to minor version bumps, so that I can plan for potential efforts involved.
During that lead time I would of course not want to be exposed to already fixed security vulnerabilities.

This expectation doesn't seem to be met by how the Paketo project uses semver though. If I rephrase the above, removing semver from the picture, I'd state the following points:

• I would like to be able to get fixes without getting features (low risk)
• I would like to be able to get fixes without getting breaking changes (high risk)

Is there a way to achieve that with the Paketo buildpacks versioning scheme?

I suppose a part of the problem is that not all breaking changes affect everyone, but I cannot know easily if a breaking change will affect me. E.g. by the time the removal of a JRE major version, that has reached end of life, triggers a major update, I have hopefully migrated to a newer one. I definitely agree that the buildpack cannot do anything about it regardless. But the same major version bump or other major version bumps might affect me through an incompatible change in a dependency or the buildpack itself.

Is it the buildpack that people want to hold back (i.e. changes to the actual buildpack code) or is it the dependencies that buildpacks install people want to hold back?

Concluding my statements I'd say it is neither. It is about holding back risk (in terms of incompatible changes) while embracing fixes (which also reduces risk).

[dmikusa]

Is there a way to achieve that with the Paketo buildpacks versioning scheme?

Speaking for the Java buildpacks only.

1. New buildpack functionality goes into minor versions so you could get buildpack fixes without buildpacks features by only upgrading patch releases. Similarly, you could get buildpack fixes without breaking changes (to API/interface) by only upgrading minor/patch versions. That is for the buildpack itself.

2. For dependencies, new versions branches (like adding Java 17) and updates to existing version branches (like updating Java 11), would all come in a minor release. Removing a dependency branch triggers a new major version (like removing Java 15).

So taking both into account to your questions:

I would like to be able to get fixes without getting features (low risk)

You can get buildpack fixes by only upgrading patch releases, but you cannot get dependency fixes this way. Since dependency bumps come in new minor versions, which would also subject you to new buildpack functionality.

You could not achieve this presently without forking and modifying buildpack.toml, because dependencies are tightly coupled to buildpacks. If we relaxed that coupling, you could probably achieve this scenario.

I would like to be able to get fixes without getting breaking changes (high risk)

This should be achievable now if you do not upgrade major branches. The minor bumps will pull in both dependency and buildpack fixes, but should not break API/interface or remove dependencies.

For what it's worth, the CF JBP has dependencies less tightly coupled. You can specify an alternative repo from which it will fetch dependencies and you can hold back/publish new dependencies independent of the buildpack. Does this model work better for you?

[loewenstein]

Thanks @dmikusa-pivotal.

I am getting a little confused. You say

1. New buildpack functionality goes into minor versions so you could get buildpack fixes without buildpacks features by only upgrading patch releases. Similarly, you could get buildpack fixes without breaking changes (to API/interface) by only upgrading minor/patch versions. That is for the buildpack itself.

What I thought I had understood from the discussion around backports so far is that with a major version bump to lets' say 6.0.0 there would no longer be any releases of 5.x or 5.16.x. While I could prevent features or breaking changes by pinning 5 or 5.15, I would not get any fixes, including security fixes for buildpacks or dependencies.

2. For dependencies, new versions branches (like adding Java 17) and updates to existing version branches (like updating Java 11), would all come in a minor release. Removing a dependency branch triggers a new major version (like removing Java 15).

While I don't understand the details of delivering patch level updates of dependencies (like JRE 16.0.1 -> 16.0.2) as minor level updates of the buildpacks - I don't see this as a huge problem, as minor versions are still supposed to be compatible. Hence an update seems a comparably low risk.

Regarding the major version bumps, I guess I would prefer to see the bump some time before the end-of-life. Seeing the bump as kind of a clear warning signal that it's time to use a newer JRE. This way we could still deliver fixes for the old major version, including for the JRE that approaches end-of-life.

I agree with @fg-j, that there will be at least some upfront investment into automation necessary, if we wanted to deliver e.g. Java Buildpack 5.x with security fixes, after 6.0 has been released. Without having looked into the details much, I'd guess that a huge part of the additional effort can be solved with automation.

[ghost]

1. New buildpack functionality goes into minor versions so you could get buildpack fixes without buildpacks features by only upgrading patch releases. Similarly, you could get buildpack fixes without breaking changes (to API/interface) by only upgrading minor/patch versions. That is for the buildpack itself.

What I thought I had understood from the discussion around backports so far is that with a major version bump to lets' say 6.0.0 there would no longer be any releases of 5.x or 5.16.x. While I could prevent features or breaking changes by pinning 5 or 5.15, I would not get any fixes, including security fixes for buildpacks or dependencies.

🤦 Yes, sorry I misspoke there. You're right. So if you follow the advise I wrote you can limit the risk of new versions, but you cannot continue to get any updates on the old branches. Because we don't backport anything.

If we are were to divorce the dependency versions from the buildpack (i.e. make a way you can bump dependencies without needing to re-release the buildpack), you could continue to get dependency updates (not buildpack fixes) on older branches though. In terms of risk, security and CVEs, this would probably be what a user needs 99% of the time. To bump the dependencies, not the buildpack.

1. For dependencies, new versions branches (like adding Java 17) and updates to existing version branches (like updating Java 11), would all come in a minor release. Removing a dependency branch triggers a new major version (like removing Java 15).

While I don't understand the details of delivering patch level updates of dependencies (like JRE 16.0.1 -> 16.0.2) as minor level updates of the buildpacks - I don't see this as a huge problem, as minor versions are still supposed to be compatible. Hence an update seems a comparably low risk.

Honestly, this system is just how things had been working when I joined the project. I don't fully understand the choices either. I know there was a Paketo RFC for formalize this process. I was just looking through it and from the examples, it seems like we should only be bumping the patch release for new patches, like 16.0.1 -> 16.0.2. This may be something we can adjust.

I can take an action item to look at our CI and reconcile it with the RFC.

Regarding the major version bumps, I guess I would prefer to see the bump some time before the end-of-life. Seeing the bump as kind of a clear warning signal that it's time to use a newer JRE. This way we could still deliver fixes for the old major version, including for the JRE that approaches end-of-life.

I agree with @fg-j, that there will be at least some upfront investment into automation necessary, if we wanted to deliver e.g. Java Buildpack 5.x with security fixes, after 6.0 has been released. Without having looked into the details much, I'd guess that a huge part of the additional effort can be solved with automation.

I understand what you're saying. It would be nice to have some time to migrate versions, especially for a major change. What we have been trying to do to combat this is to ensure that the final release before we bump is as up-to-date as possible. So if we are going to go from 5.5.2 -> 6.0.0. We would cut a new 5.6.0 (or 5.5.3, whatever makes sense) that has the latest set of dependencies in it. The 6.0.0 release would then have the same set of dependencies plus the breaking change. That way users can get as much milage as possible out of the final 5.x release. It's not perfect, but with Java releases coming quarterly it does buy users a few months to push through a change.

In regards to cost, I think it's overly optimistic to say backports is just an issue of automation. Sure it'll help, but there are other costs as well.

Some concerns off the top of my head:

1. If we were to backport, we would have to maintain multiple code branches, we would potentially have to to merge fixes
   and we'd definitely have to merge dependency updates across these branches. This increases cognitive load and development time, like if there are merge conflicts or complicated features that perhaps don't backport as easily.
2. We also have to modify and maintain a more complicated CI system to facilitate this. Further cognitive load to support this goal.
3. It would also increase the amount of images we would have to store because a single code release could result in two or more actual images being cut (new release, plus however many backports we have to cut). Ongoing storage costs are important to consider.
4. Because of the rate at which we release new buildpacks, I also worry about how many branches that we'd have to maintain. If we have a six month backport policy (so we'd keep backporting changes to a release for six months from the time it's released), and we're releasing weekly or more often, that's 26 release (not counting backports) during before one version rolls out of support. If 1/3rd of those are minor releases, which seems conceivable since we do a lot of new feature work, that's roughly eight branches to be supporting at any given time. Maybe it's not this drastic, but given the rate that we release this is a concern.

I think what also makes me against backports is the fact that probably 99% of the reason to have backports, most of the backport releases we'd end up cutting, would be to update dependencies provided by the buildpack (how often are there CVE's in the buildpack itself?). If we instead focus on decoupling dependencies from buildpacks, we can provide greater control for users. A user that needs time, can stick on an older buildpack but still pull in the dependencies they are comfortable with consuming. Ultimately, IMHO, that seems more on mission for buildpacks, because even in a case where the user wants to proceed slowly with buildpack updates we are still facilitating a way for those users to update their dependencies.

[loewenstein]

I think what also makes me against backports is the fact that probably 99% of the reason to have backports, most of the backport releases we'd end up cutting, would be to update dependencies provided by the buildpack (how often are there CVE's in the buildpack itself?). If we instead focus on decoupling dependencies from buildpacks, we can provide greater control for users. A user that needs time, can stick on an older buildpack but still pull in the dependencies they are comfortable with consuming. Ultimately, IMHO, that seems more on mission for buildpacks, because even in a case where the user wants to proceed slowly with buildpack updates we are still facilitating a way for those users to update their dependencies.

I agree that the vast majority of reasons for a backport would be security fixes in dependencies like JREs. I am not so sure however if a decoupling of buildpacks and dependencies would be what users are looking for. Wouldn't it put the burden of testing a set of buildpacks and dependencies for soundness to the consumers?

Ideally, as a user that doesn't want to care, I use the Java buildpack, will get the default Java version and rely on the fact that all available security fixes are applied.

On the cost side, putting everything on automation might be overly optimistic, but I am also not sure the worst case picture of branching is realistic.

If we were to backport, we would have to maintain multiple code branches, we would potentially have to to merge fixes
and we'd definitely have to merge dependency updates across these branches.

Given we expect mainly dependency updates to be interesting - I guess it would be fair to only deliver security fixes for the last minor version of one or tow old major versions. That would limit the number of branches and the complexity of merges significantly, wouldn't it? I guess most merges could be done by a CI job.

Because of the rate at which we release new buildpacks, I also worry about how many branches that we'd have to maintain.

Maybe this could be handled by giving guarantees only in number of major releases and not in time. Not ideal maybe for some consumers, but still much more than no fixes after a single major version bump.

[dmikusa]

I agree that the vast majority of reasons for a backport would be security fixes in dependencies like JREs. I am not so sure however if a decoupling of buildpacks and dependencies would be what users are looking for.

I think it depends on how it would be implemented. Looking at something like the Java Buildpack for CF, it can just update automatically independent of a buildpack update (in "online" mode). It relies on external indexes to pull in the most recent version of a dependency. A system like this would decouple buildpack release from dependency release, while still retaining control for the user to pin a version if needed. All of it could be done without requiring the user to do any more work since they would be pulling dependencies from locations managed by the buildpack team (unless a user wants to take complete control and run their own mirror).

Ideally, as a user that doesn't want to care, I use the Java buildpack, will get the default Java version and rely on the fact that all available security fixes are applied.

I totally agree. What I'm suggesting by default, the buildpacks keep updating dependencies. No additional involvement is required for most users. At the same time, there are additional control knobs for those users that want/need them.

Wouldn't it put the burden of testing a set of buildpacks and dependencies for soundness to the consumers?

While buildpacks are coupled to dependencies at the moment, it's not for testing reasons. It's simply the mechanism that's used to provide metadata. That metadata is part of the buildpack itself, i.e. buildpack.toml, so you literally can't change it without issuing a PR, making a code change & cutting a release.

In regards to testing, I would not say that the testing we do relates to the soundness of the dependencies. Our testing focus is on what the buildpack itself is doing and how it integrates with the dependencies. We make sure that the buildpacks detect properly, that they install files into the proper locations, configure environments correctly, pass arguments through, configure correct start commands. We don't test the fitness of the dependencies that we install.

To illustrate what I mean, we pass arguments through to the JVM to configure how it behaves. You do this through a standard interface, the java arguments. For this example, let's say an argument to configure the heap size. We will validate in our tests that we're passing these arguments through to the JVM in the documented way (i.e. -Xmx). If a new version of Java has a bug and when you request over 2G of heap space, the JVM crashes. That's not something we'd necessarily catch in our tests (in this contrived example, it's possible cause it's a very obvious bug). This is what I mean that we don't test the fitness of the dependencies we install. I'm not sure exactly that's what you're getting at here, but I do want to be very clear about what we do and don't test.

Probably more to your point, by decoupling dependencies there is a risk, I would say very low (probably just with major dependency bumps), that the buildpack and its dependencies could stop working. If Java changes the argument for setting the max heap size, then you would need a buildpack update to work with those new versions of Java. Decoupling dependencies gives you a way to delay updating buildpack versions, not permanently live with a single buildpack version. I think it would give you as much time as we could offer if we were updating dependencies through backports.