OCI mode and --no-home

[emi80]

Version of Singularity
What version of Singularity are you using? Run:

$ singularity --version
singularity-ce version 4.0.1-jammy

Describe the bug
Cannot run any command in OCI mode if --no-home is specified.

To Reproduce
Steps to reproduce the behavior:

❯ singularity exec --no-home --oci docker://alpine echo Hello
INFO:    Using cached OCI-SIF image
2023-10-26T14:17:16.000200755Z: chdir: No such file or directory

It works as expected if I remove the --oci flag:

❯ singularity exec --no-home docker://alpine echo Hello
INFO:    Using cached SIF image
Hello

Expected behavior
A clear and concise description of what you expected to happen.

❯ singularity exec --no-home --oci docker://alpine echo Hello
INFO:    Using cached OCI-SIF image
Hello

OS / Linux Distribution
Which Linux distribution are you using?

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Installation Method
SingularityCE has been installed using the released deb package for Ubuntu 22.04

Additional context
I apologize if this is expected or planned for OCI mode. Thank you!

[emi80]

I just saw #1780

[dtrudg]

Hi @emi80 - I believe this behaviour is due to the version of crun that's incuded on jammy, and is used by SingularityCE as the low-level runtime in --oci mode. It doesn't handle $HOME directories quite the same as runc until a later version than the 1.5 in Ubuntu Jammy.

I'll try to spend some time soon to verify this. In the meantime, if you swap the crun package for runc, or install a newer version of crun manually, you may find that this works.

[emi80]

Hi @dtrudg,

thanks a lot for your quick reply. I was not aware of this. Is there any information about this behavior of crun in the Singularity docs? I think it would be really helpful for users.

I might then switch to runc and see how it goes.

Many thanks!

Cheers,
Emilio

[emi80]

Indeed switching to runc works

Thanks!

[dtrudg]

You could also try adding --cwd / to the command line and see if that is a workaround with crun.

I think it's actually the CWD not being created (and our CWD being set to $HOME) which is the pathway to the issue here.

We're noting some issues with crun vs runc as --oci mode gets more use, and we will ensure things are documented in the near future.

[dtrudg]

I shoukd clarify that this isn't an issue with crun generally. It gets bumped to newer versions in Red Hat (and derived) distros regularly, and newer versions don't have this problem.

Ubuntu sticks rigidly to older crun versions throughout a release's lifecycle, so it's missing various fixes.

[emi80]

@dtrudg, thanks for the clarification.

As Ubuntu is not the main Linux distro used in HPC environments this might not be a real issue. Though it's important to be aware of this.