ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument ABORT : Retval = 255s

[enjras]

I'm very new to the use of my institute's HPC. I need do use a singularity image I downloaded as software.simg file.
Singularity was already installed on the cluster.

Version of Singularity:

$ singularity version
singularity --version
2.6.1-dist

Expected behavior

singularity exec --nv software.simg nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.54                 Driver Version: 396.54                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro GP100        Off  | 00000000:03:00.0 Off |                  Off |
| 33%   50C    P0    37W / 235W |      2MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

Actual behavior

singularity exec --nv software.simg nvidia-smi

/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec/singularity/functions: line 491: grep: command not found
/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec/singularity/functions: line 491: grep: command not found
/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec/singularity/functions: line 493: grep: command not found
/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec/singularity/functions: line 493: grep: command not found
/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec/singularity/functions: line 497: ldconfig: command not found
/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec/singularity/functions: line 497: grep: command not found
WARNING: Could not find any NVIDIA libraries on this host!
WARNING: You may need to edit /gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/etc/singularity/nvliblist.conf
WARNING: Could not find any NVIDIA binaries on this host!
WARNING: You may need to edit /gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/etc/singularity/nvliblist.conf
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

Also in case it gives you useful information:

singularity -vvv shell hello-world.simg

Increasing verbosity level (4)
Singularity version: 2.6.1-dist
Exec'ing: /gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec/singularity/cli/shell.exec
Evaluating args: 'hello-world.simg'
VERBOSE: Set messagelevel to: 4
VERBOSE: Initialize configuration file: /gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/etc/singularity/singularity.conf
VERBOSE: Got config key allow setuid = 'yes'
VERBOSE: Got config key max loop devices = '256'
VERBOSE: Got config key allow pid ns = 'yes'
VERBOSE: Got config key config passwd = 'yes'
VERBOSE: Got config key config group = 'yes'
VERBOSE: Got config key config resolv_conf = 'yes'
VERBOSE: Got config key mount proc = 'yes'
VERBOSE: Got config key mount sys = 'yes'
VERBOSE: Got config key mount dev = 'yes'
VERBOSE: Got config key mount devpts = 'yes'
VERBOSE: Got config key mount home = 'yes'
VERBOSE: Got config key mount tmp = 'yes'
VERBOSE: Got config key mount hostfs = 'no'
VERBOSE: Got config key bind path = '/etc/localtime'
VERBOSE: Got config key bind path = '/etc/hosts'
VERBOSE: Got config key user bind control = 'yes'
VERBOSE: Got config key enable overlay = 'try'
VERBOSE: Got config key mount slave = 'yes'
VERBOSE: Got config key sessiondir max size = '16'
VERBOSE: Got config key allow container squashfs = 'yes'
VERBOSE: Got config key allow container extfs = 'yes'
VERBOSE: Got config key allow container dir = 'yes'
VERBOSE: Got config key memory fs type = 'tmpfs'
VERBOSE: Got config key always use nv = 'no  '
VERBOSE: Set messagelevel to: 4
VERBOSE: Initialize configuration file: /gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/etc/singularity/singularity.conf
VERBOSE: Got config key allow setuid = 'yes'
VERBOSE: Got config key max loop devices = '256'
VERBOSE: Got config key allow pid ns = 'yes'
VERBOSE: Got config key config passwd = 'yes'
VERBOSE: Got config key config group = 'yes'
VERBOSE: Got config key config resolv_conf = 'yes'
VERBOSE: Got config key mount proc = 'yes'
VERBOSE: Got config key mount sys = 'yes'
VERBOSE: Got config key mount dev = 'yes'
VERBOSE: Got config key mount devpts = 'yes'
VERBOSE: Got config key mount home = 'yes'
VERBOSE: Got config key mount tmp = 'yes'
VERBOSE: Got config key mount hostfs = 'no'
VERBOSE: Got config key bind path = '/etc/localtime'
VERBOSE: Got config key bind path = '/etc/hosts'
VERBOSE: Got config key user bind control = 'yes'
VERBOSE: Got config key enable overlay = 'try'
VERBOSE: Got config key mount slave = 'yes'
VERBOSE: Got config key sessiondir max size = '16'
VERBOSE: Got config key allow container squashfs = 'yes'
VERBOSE: Got config key allow container extfs = 'yes'
VERBOSE: Got config key allow container dir = 'yes'
VERBOSE: Got config key memory fs type = 'tmpfs'
VERBOSE: Got config key always use nv = 'no  '
VERBOSE: Initializing Singularity Registry
VERBOSE: Adding value to registry: 'BINDIR' = '/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/bin'
VERBOSE: Adding value to registry: 'LIBEXECDIR' = '/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/libexec'
VERBOSE: Adding value to registry: 'VERSION' = '2.6.1-dist'
VERBOSE: Adding value to registry: 'LOCALSTATEDIR' = '/var'
VERBOSE: Adding value to registry: 'SYSCONFDIR' = '/gnu/store/dziimc076y8psdcyrg6c7f16fx9fdw2m-singularity-2.6.1/etc'
VERBOSE: Adding value to registry: 'MESSAGELEVEL' = '4'
VERBOSE: Adding value to registry: 'IMAGE' = 'hello-world.simg'
VERBOSE: Adding value to registry: 'COMMAND' = 'shell'
VERBOSE: Set home (via getpwuid()) to: /home/nsurname
VERBOSE: Running NON-SUID program workflow
VERBOSE: Invoking the user namespace
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

What OS/distro are you running

uname --all
Linux [hostname] 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core) 

How did you install Singularity

I did not install Singularity myself. It is already installed on the HPC.

[dtrudg]

The paths to Singularity here beginning /gnu/store suggest it is installed via the GNU Guix package manager. This isn't something we can easily support as an installation in Guix involves patches. Also note that Singularity 2.6.1 is old, and doesn't receive updates / support in the open-source community.

It looks like some of the executables are not 'setuid root' as required, and the system does not allow using unprivileged user namespaces. This means Singularity cannot work as installed. Either the system administrator will need to ensure the required executables are setuid root, or unprivileged user namespaces will need to be enabled on the system. Guix should have a mechanism to ensure setuid is set properly for Singularity via the singularity-service-type.

https://github.com/guix-mirror/guix/blob/master/gnu/services/docker.scm#L185

I'm going to close this issue due to the caveats above. Please feel free to report any problems you have with a current version of Singularity.

[enjras]

Ok, thank you very much for your help!