Unable to bind directory during build to location that does not already exist

[luator]

Version of Singularity:

$ singularity --version
singularity version 3.8.3

Expected behavior

I am using --bind=/opt/foo to bind a directory during build. The expected behaviour is that /opt/foo is accessible during build (i.e. that bind works the same way as when running an image).

Actual behavior

It fails with

FATAL:   container creation failed: mount /opt/foo->/opt/foo error: while mounting /opt/foo: destination /opt/foo doesn't exist in container
FATAL:   While performing build: while running engine: exit status 255

Steps to reproduce this behavior

Assume this minimal definition file:

Bootstrap: docker
From: ubuntu:20.04

%post
    echo "Hi!"

Create a directory /opt/foo and run

singularity build --fakeroot --bind=/opt/foo minimal.sif minimal.def

Note that if I bind to a location that already exists in the base image (e.g. --bind=/opt/foo:/mnt) it works.

What OS/distro are you running

$ cat /etc/os-release

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

How did you install Singularity

Installed using a deb package that I created from source, using this custom tool.

[DrDaveD]

This is intentional. The mount point is expected to be created in the .def file.

[luator]

Oh, okay. How do I do this? Creating the directory in %post doesn't work as it is already too late there.

It seems the documentation is wrong in this case:

This flag allows you to mount a directory, a file or an image during build, it works the same way as --bind for shell, exec and run and can be specified multiple times, see user defined bind paths.

It is also not mentioned in singularity build --help.

[DrDaveD]

This is really not my area of knowledge, I'm sorry about that. In fact I tried a couple of things to create the directory and it didn't seem to help, so I may in fact be wrong in what I already said. Hopefully somebody else will respond, or maybe you'll have better luck asking on the google group or in slack.

[eburgueno]

I can trace this working on 3.7.4, so this is probably a regression introduced after that:

These are my test files:

$ cat Singularity 
Bootstrap: docker
From: alpine:latest
%post
apk update
apk add bash

$ cat run.sh 
mkdir -p /input
rm -f test.sif
unset SINGULARITY_BINDPATH
singularity build test.sif Singularity
echo
rm -f test.sif
export SINGULARITY_BINDPATH=/input
singularity build test.sif Singularity

Testing that with Singularity 3.7.4:

$ docker run --privileged -ti --rm -v $(pwd):/data --entrypoint /bin/bash centos:7
# yum -y install epel-release
# yum -y install https://kojipkgs.fedoraproject.org//packages/singularity/3.7.4/1.el7/x86_64/singularity-3.7.4-1.el7.x86_64.rpm
# cd /data; ./run.sh
INFO:    Starting build...
(...)
INFO:    Creating SIF file...
INFO:    Build complete: test.sif

INFO:    Starting build...
(...)
INFO:    Creating SIF file...
INFO:    Build complete: test.sif

Testing with 3.8.0:

$ docker run --privileged -ti --rm -v $(pwd):/data --entrypoint /bin/bash centos:7
# yum -y install epel-release
# yum -y install https://kojipkgs.fedoraproject.org//packages/singularity/3.8.0/1.el7/x86_64/singularity-3.8.0-1.el7.x86_64.rpm
# cd /data; ./run.sh
INFO:    Starting build...
(...)
INFO:    Creating SIF file...
INFO:    Build complete: test.sif

INFO:    Starting build...
Getting image source signatures
Copying blob a0d0a0d46f8b [--------------------------------------] 0.0b / 0.0b
Copying config 696d33ca15 done  
Writing manifest to image destination
Storing signatures
2021/10/06 23:13:16  info unpack layer: sha256:a0d0a0d46f8b52473982a3c466318f479767577551a53ffc9074c9fa7035982e
INFO:    Running post scriptlet
FATAL:   container creation failed: mount /input->/input error: while mounting /input: destination /input doesn't exist in container
FATAL:   While performing build: while running engine: exit status 255

[eburgueno]

You can work around this by unsetting SINGULARITY_BINDPATH (or setting it to an empty string), or by creating the mountpoint before %post in your definition file:

$ cat Singularity
Bootstrap: docker
From: alpine:latest

%setup
mkdir $SINGULARITY_ROOTFS/input

%post
apk update
apk add bash

$ export SINGULARITY_BINDPATH=/input
$ singularity build test.sif Singularity
INFO:    Starting build...
(...)
INFO:    Running setup scriptlet
+ mkdir /tmp/build-temp-106646087/rootfs/input
INFO:    Running post scriptlet
(...)
INFO:    Creating SIF file...
INFO:    Build complete: test.sif

But neither of these are ideal. The build process should not make assumptions about the runtime environment. And forcing end-users to unset the bind paths defined by the admin in the global config defeats the purpose of the global config.

[DrDaveD]

The --bind option was added in pr #5911.

[DrDaveD]

@eburgueno I see now that in your 3.7.4 test all you did was show that there was no error message, but you did not show that it was actually bind-mounting anything. @cclerget said that SINGULARITY_BINDPATH was ignored prior to 3.8.0. Can you confirm that?

[eburgueno]

@DrDaveD yes, it appears to be ignored:

$ cat Singularity 
Bootstrap: docker
From: alpine:latest

%post
ls -l /input
apk update
apk add bash

$ docker run --privileged -ti --rm -v $(pwd):/data --entrypoint /bin/bash centos:7
# yum -y install epel-release
# yum -y install https://kojipkgs.fedoraproject.org//packages/singularity/3.7.4/1.el7/x86_64/singularity-3.7.4-1.el7.x86_64.rpm
# mkdir -p /input
# touch /input/me
# export SINGULARITY_BINDPATH=/input
# cd /data
# singularity build test.sif Singularity
INFO:    Starting build...
Getting image source signatures
Copying blob a0d0a0d46f8b done  
Copying config 696d33ca15 done  
Writing manifest to image destination
Storing signatures
2021/10/09 23:56:42  info unpack layer: sha256:a0d0a0d46f8b52473982a3c466318f479767577551a53ffc9074c9fa7035982e
INFO:    Running post scriptlet
+ ls -l /input
ls: /input: No such file or directory
FATAL:   While performing build: while running engine: exit status 1

[sorenwacker]

Would be nice to have a clean, simple, and complete example for this.

[Jeffjewett27]

I had the same issue, but this snippet from the docs solved it for me.

https://apptainer.org/docs/user/main/build_a_container.html#bind

Beware that the mount points must exist in the built image prior to executing post and test. So if you want to bind --bind /example and it doesn’t exist in the bootstrap image, you have to workaround that by adding a setup section:

%setup
  mkdir $APPTAINER_ROOTFS/example

In my case, I did mkdir -p $APPTAINER_ROOTFS/nested/path/to/bind, and built with apptainer build --bind /nested/path/to/bind container.sif Singularity.def