extend packaging support for Docker/Singularity definition files & images (WIP)

[boegel]

After a nice discussion with a couple of the Singularity developers (@gmkurtzer, @GodloveD, @bauerm97) and some EasyBuilders who have been playing around with the combo of EasyBuild & Singularity (@pescobar, @shahzebsiddiqui), we came up with some ideas to extend the existing packaging support in EasyBuild (cfr. easybuild.readthedocs.io/en/latest/Packaging_support.html) to (Docker &) Singularity containers.

The idea is to add support for both generating container definition files (which can then be fed into docker or singularity the create the actual container image or uploaded to https://singularityhub.github.io/) and building actual container images by letting EasyBuild call out to the tools itself. Both use cases make sense.

Additional things to figure out:

• allowing to pick between different procedures to build the container image, e.g. letting eb build the software and then just shove than in a container image, letting eb generate RPMs which can then be used to populate the container image, etc.
• a flexible way to steer specifics, e.g. which bootstrap mechanism to use, which OS to use in the container image, whether an existing (Docker) image should be used as a base, etc.

Inspiration can be taken from earlier experiments:

• https://github.com/pescobar/singularity-easybuild (@pescobar & @shahzebsiddiqui)
• https://github.com/GodloveD/EasyBuild (@GodloveD)

In parallel, the Singularity developers plan to look into supporting EasyBuild as a bootstrap agent in Singularity, which we could then leverage here.

Current status: very much work-in-progress; only the rough structure is in place, actual packaging support (i.e. fleshing out the package_with_docker & package_with_singularity functions) is still TODO.

[stefanoberri]

Hi. I am very interested in this effort and see how other people approach and combine EasyBuild and Docker/Singularity.

I have made a similar effort (limited to building EasyBuild and the toolchain goolf-1.7.20 for now) available here
https://github.com/stefanoberri/docker-easybuild
and the corresponding images here
https://cloud.docker.com/app/sberri/repository/docker/sberri/easybuild/general
and here
https://cloud.docker.com/app/sberri/repository/docker/sberri/eb-goolf-1.7.20/general

[shahzebsiddiqui]

interesting work @stefanoberri, I see some similarities in our builds, your work is more focused on Docker. I think this would be useful for container support with docker. I have also started building some packages in singularity.

Please see https://github.com/shahzebsiddiqui/eb-singularity
https://github.com/shahzebsiddiqui/eb-singularity/tree/eb-rpms

I have an easybuild bootstrap process via Docker and I use RPM to satisfy dependency to speed up the build process.

[boegel]

I discussed this a bit with @shahzebsiddiqui, @jordiblasco and @omula, and we agreed that the first step for this should be to let EasyBuild generate Singularity definition files that use eb --robot in it, which can then be passed to Singularity to create an image.

The work done by @shahzebsiddiqui in https://github.com/shahzebsiddiqui/eb-singularity can be leveraged for this, in particular examples like https://github.com/shahzebsiddiqui/eb-singularity/blob/master/singularity/centos/def_files/g/GCCcore/GCCcore-5.4.0.def .

Creating the image as a part of eb is more troublesome since that requires root privileges (sudo).

Aspects like speeding up the process (by layering images on top of each other or installing RPMs for dependencies rather than building everything from scratch), or trimming down the image size (getting rid of unnecessary (build) dependencies, static linking, build directory outside of image, etc.). can then be looked at later.

Potentially useful in this context: https://spack.readthedocs.io/en/latest/workflows.html#using-spack-to-create-docker-images .

[boegel]

superseded by changes in #2332