feat: Stages

[gmpinder]

Stages

A new property (stages) is being added to the recipe file schema. This property will allow users to define a list of Containerfile stages each with their own modules. Stages can be used to compile programs, perform parallel operations, and copy the results into the final image without contaminating the final image.

Module Support

Currently the only modules that work out-of-the-box are copy, script, files, and containerfile. Other modules are dependent on the programs installed on the image. In order to better support some of our essential modules, a setup script is ran at the start of each stage that is not scratch. This script will install curl, wget, bash, and grep and use the package manager for the detected distributions.

At this time, the following distributions are supported:

• Debian
• Ubuntu
• Fedora
• Alpine

Contributions to increase the size of this list is welcome!

Syntax

• Required
  • from - The full image ref (image name + tag). This will be set in the FROM statement of the stage.
  • name - The name of the stage. This is used when referencing the stage when using the from: property in the copy module.
  • modules - The list of modules to execute. The exact same syntax used by the main recipe modules: property.
• Optional
  • shell - Allows a user to pass in an array of strings that are passed directly into the SHELL instruction.

Example

stages:
- name: ubuntu-test
  from: ubuntu
  modules:
  - type: files
    files:
    - usr: /usr
  - type: script
    scripts:
    - example.sh
    snippets:
    - echo "test" > /test.txt
  - type: test-module
  - type: containerfile
    containerfiles:
    - labels
    snippets:
    - RUN echo "This is a snippet"

Tasks

• from-file: - Allows the user to store their stages in a separate file so it can be included in multiple recipes
• no-cache: - This will be useful for stages that want to pull the latest changes from a git repo and not have to rely on the base image getting an update for the build to be triggered again.
• Add setup script to be able to install necessary programs to run bluebuild modules in stages
• Check for circular dependencies and error out

copy module

This is a 1-1 for the COPY instruction. It has the ability to copy files between stages, making this a very important addition to complete functionality for the stages feature. Each use of this "module" will become its own layer.

Syntax

• Required
  • src - The source directory/file from the repo OR when from: is set the image/stage that is specified.
  • dest - The destination directory/file inside the working image.
• Optional
  • from - The stage/image to copy from.

Example

modules:
- type: copy
  from: ubuntu-test
  src: /test.txt
  dest: /

Tasks

• make from: optional
• Add README.md and module.yml

Feature gating

Gating this feature until we release for v0.9.0. The plan will be to build all features (including this one) for main branch builds. This means that these features will be available when using the main image and consequently the use_unstable_cli: option on the GitHub Action. All future v0.9.0 features will be gated as well to allow for patches to v0.8.

Tasks

• Build --all-features on non-tagged builds
• Add stages and copy features

[xynydev]

In my mind I was initially critical of this feature, since it would be more efficient to just build a new container image for the "stage", as this approach rebuilds the stage every day. But as users seem delighted, and caching will make it better, I guess it's good. It's always optional, too.

Most of our modules aren't designed to be used in stages, and would probably produce odd results there. I guess I should work on structuring the docs for modules better wrt that soon.

[gmpinder]

Most of our modules aren't designed to be used in stages, and would probably produce odd results there. I guess I should work on structuring the docs for modules better wrt that soon.

Yeah, I was thinking that we could just mention that not all modules will work oob with every build image and we make no guarantees. The script and files modules are good examples of modules that are still very useful for stages though.

But as users seem delighted, and caching will make it better, I guess it's good. It's always optional, too.

Yeah I'm going to be using this to compile the latest versions of some of my favorite programs rather than having to compile them every so often locally.

[xynydev]

Can't review this now, would be nice to get a corresponding docs PR before merge.

[gmpinder]

Can't review this now, would be nice to get a corresponding docs PR before merge.

Sure, I can get that out later today.

[xynydev]

Aight, thanks. I just read the docs, and there are still a couple things that require discussion IMO.

First of all, are we sure it's a good idea to include a collection of setup scripts needed for the modules for different distributions, as there's an infinite amount of distributions, and not all modules are intended to be used in stages... Would it be best to just supply a package list required for most modules, to not have a growing list of scripts to maintain, and not imply official support for some distros in stages over others?

Also, the stage configuration has an overlapping set of top-level keys with the recipe itself, but the from: key is named differently (from: vs base-image: and image-version:). One could argue that from: is better, but that's a bigger problem if that's the case (actually I'ma make a related issue, edit: #180).

And one thing I noticed, which is not related to this PR, is alt-tags:. My mental model of this parameter was "the default is latest+timestamp, but you can override if you want to", while the alt-tags: name sounds more like "latest+timestamp is the recommended way to tag, but you can swap it out for an alternative method". I'll make an issue about this too. Edit: #179

[gmpinder]

First of all, are we sure it's a good idea to include a collection of setup scripts needed for the modules for different distributions, as there's an infinite amount of distributions, and not all modules are intended to be used in stages... Would it be best to just supply a package list required for most modules, to not have a growing list of scripts to maintain, and not imply official support for some distros in stages over others?

I think we should support the base types of modules just so that use of stages is similar to the final image. I don't think it's out of the question to make sure a few tools are installed. The way we check for the package manager could probably be adjusted to instead check for the program rather than /etc/os-release.

Also, the stage configuration has an overlapping set of top-level keys with the recipe itself, but the from: key is named differently (from: vs base-image: and image-version:). One could argue that from: is better, but that's a bigger problem if that's the case (actually I'ma make a related issue, edit: #180).

Yeah I originally was trying to follow a similar pattern with it being image: originally. After trying to write the documentation for it, I realized that it was difficult to describe the connection between the value provided and what's being templated into the Containerfile. That made me decide to switch to from: cause it was WAY easier to explain what that property was doing. I think we should keep it for stages.

[xynydev]

I think we should support the base types of modules just so that use of stages is similar to the final image. I don't think it's out of the question to make sure a few tools are installed. The way we check for the package manager could probably be adjusted to instead check for the program rather than /etc/os-release.

If we need to, we could copy yq and such from containers. I kinda dislike the idea of writing scripts for different package managers and repos due to reasons above, and now that you took up environment detection, that too might prove challenging.

[gmpinder]

If we need to, we could copy yq and such from containers. I kinda dislike the idea of writing scripts for different package managers and repos due to reasons above, and now that you took up environment detection, that too might prove challenging.

We copy in yq into the image during the setup script run already just like we do for the main images. There are basic utilities that are needed in order to run the files or script modules. Alpine is a widely used base image because of it's small size, so we can very easily assume that users will attempt to build with an alpine image. The thing about Alpine is that it does NOT come with bash installed (thus the reason the setup script just runs POSIX sh) or a gnu compatible grep. If we didn't do this, users would try to use some of our existing modules to perform actions in their stages and run into errors when trying to run our 1st party modules. This creates a disconnect and a sense of stages not being an easy to use feature. Maintaining a single script to install the basics is not a huge deal imho (especially since I'm the one maintaining this particular repo). I'm keeping it in and I'm not budging from that stance.

[gmpinder]

Doing some testing with this made me find out the buildah/podman currently doesn't support the --link on COPY containers/buildah#4325. I'll be removing that for now.