oci: fallback to fuse-overlayfs if kernel doesn't support unprivileged overlays and oci: support for writable extfs img overlay via fuse-overlayfs, from sylabs 1730 & 1740 #335

In preparation for sylabs/singularity#1021, where we will add a placeholder OCI runtime launcher, refactor the existing code so that: * The internal/pkg/runtime/launcher package contains common option handling, utility functions, and a Launcher interface. * There is a launcher.native package containing the existing launch code for the native singularity runtime. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Accepts no options, fails if options provided. Implements an Exec method which does nothing. Fixes sylabs/singularity#1021 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Handle empty structs (not just nils). Handle always-set network, config file. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Replace invocations of SingularityCE's OCI runtime engine with wrapped invocations of runc. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

As noted in review, the `runc` call for `oci exec` was not actually being made. E2E test for `oci exec` was only checking return code, not for output so it passed. Fix these issues. In `e2e/suite.go` also move registry creation so that it is only performed for the man tests in PID+mount NS. It is not needed for the OCI/CGROUPS e2e tests and wastes CI time. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

As suggested in review the streams can be straight `io.Writer` / `io.Reader`. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

In preparation for further OCI runtime work, move the code that calls out to runc/conmon from internal/app/singularity into internal/pkg/runtime/launcher/oci The oci.Launcher will make use of the basic OCI operations to run containers from its Exec function, so this is a good location while that work proceeds. The functions will be likely be modified considerably, and potentially moved again in future, as the design/implementation of the OCI runtime interaction is developed. The internal/app/singularity OCI* functions are left as a minimal shim layer, between the CLI layer and the launcher, at this time. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

As a first step toward run/shell/exec actions on native OCI images, implement a minimal `singularity run --oci mybundle` which: * Requires an on-disk bundle with appropriate `config.json`. * Runs this bundle using `crun` or `runc`. * Makes no attempt to handle any arguments or options. * Does not modify the `config.json` - i.e. it must match namespace / mapping requirements for rootless execution etc. At this stage, the functionality is essentially equivalent to `singularity oci run` and is not yet useful. The primary purpose of the PR is to refactor some of the code that passes args for launching a container. In addition, we now use `crun` in preference to `runc` if available. `crun` supports e.g. single uid->uid mapping in a usernamespace (without root mapping). Closes sylabs/singularity#598 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Add subtest information to the large multi-step OCI test functions for easier debugging. Move the test of `oci help` out of the SEQ/ordered section. It can run in parallel, and only once regardless of cgroups managers available. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

The Ubuntu packaged crun is very old. Use a release from GitHub instead. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When running containers in `--oci` mode, the argument to run is now an image reference corresponding to a native OCI format handled by containers/image, i.e. * docker:// * docker-archive: * docker-daemon: * oci-archive: * oci: The source image is extracted into a temporary OCI bundle, with a minimally valid configuration that: * Runs the process specified by CMD & ENTRYPOINT only. * Sets the environment specified by the image ENV only. The approach is very naive - we pull through Singularity's OCI blob cache into a temporary oci layout dir, before creating the bundle from it. Auth handling for registries is not yet wired up. There is duplication of various pieces of code from the build / SIF OCI flows as these are not easily exposed to the area we are working in. The intent of the PR, at this stage, is simply to allow e.g. singularity run --oci docker://sylabsio/lolcow Closes sylabs/singularity#1036 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Simple test to create a native oci bundle from the supported sources, and verify it is valid with runtime-tools/validate. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When using `run` or `exec` with the `--oci` runtime mode, accept arguments on the command line. For `run`, the arguments override any CMD specified by the image. For `exec`, the arguments replace ENTRYPOINT/CMD entirely, bypassing the process configuration in the image config. This mirrors the behavior of Singularity images today, via the exec and run runscripts - but is implemented in the OCI bundle config, rather than a script in the container. Closes sylabs/singularity#1024 Closes sylabs/singularity#1092 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Enable `singularity shell --oci ...` with behavior matching native runtime, i.e. * Run shell set with SINGULARITY_SHELL or --shell * If not set, try /bin/bash --norc * If not available, use /bin/sh Closes sylabs/singularity#1025 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Ensure e2e tests for oci actions use user profile. Set uid/gid mappings explicitly. We need to do this anyone, going forward, but here it works around: containers/crun#1072 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

runc -> crun Need uidmap on Debian Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Tidy up the rpm spec file. * Remove the manual handling of build root, GOPATH etc. Not needed as we are using go modules now. * Remove redundant explicit deps. * Fix crun -> runc dep for EL7. * Use rpm make_xxx macros instead of direct make calls. * Ensure all directories created are owned by package. Fixes sylabs/singularity#1142 Fixes sylabs/singularity#1141 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Co-authored-by: Mike Frisch <mikef17@gmail.com> Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

If `crun` is not available, require `runc`. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Move ID mapping and process setup code to the launcher, out of the oci bundle package. Add a minimal config for `--oci` mode, rather than starting with the default OCI config. Add explicit configuration of mounts, with `tmpfs` mount locations matching the native runtime with `--containall / --compat`. Ensure ID mapping is always explicit (including for root). Add tests touching the tmpfs locations. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Initial --fakeroot support for --oci mode. Mirrors behavior with --compat / --contain. Closes sylabs/singularity#1035 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Use the configuration file sessiondir max size value for --oci mode tmpfs mounts. Increase the default from 16M -> 64M. The 16M default is very low, and has periodically caused issues running programs that create even small amounts of temporary data on --contained filesystems. Fixes sylabs/singularity#1140 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Support namespace request CLI options. * --ipc - no effect, always used in --oci mode. * --net - only supported with --network none. * --pid - no effect, always used in --oci mode. * -u / --userns - only effective for root, non-root always uses user ns. * --uts Add info logging where the option is redundant. Closes sylabs/singularity#1026 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

If we are running such that stdin is not a terminal, then the OCI runtime config should have Process.Terminal=false to avoid errors. https://github.com/opencontainers/runc/blob/main/docs/terminals.md#issues Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Implement support for bind mounts (rw & ro) specified using -B/--bind and --mount on the singularity command line. Fixes sylabs/singularity#1027 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* Merge image config ENV and env vars requested by user with the --env CLI option. * Set default SINGULARITY_CONTAINER and SINGULARITY_NAME env variables. * Set default LD_LIBRARY_PATH to be used later for library injection (this is a singularity default). Fixes sylabs/singularity#1029 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Pass SINGULARITYENV_ prefixed environment variables into container in --oci mode. Fixes sylabs/singularity#1031 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Allow --env-file to be used to provide environment variables in a file, when running a container in --oci mode. We use the same approach as the native runtime for compatibility. The env file is evaluated in the embedded shell interpreter, but starting with an empty environment. This handles quoting, comments etc. for us, and keeps maximum compatibility with the existing handling. Fixes sylabs/singularity#1030 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Now that sufficient functionality is in place for `--oci` mode, add a CHANGELOG entry for the current state. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Under --oci mode, allow --rocm, which will bind ROCm devices, libraries, and binaries into the container. Fixes sylabs/singularity#1034 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

`rocminfo` now needs `lsmod` and libdrm libdrm_amdgpu. Bind the former in tests, the libraries from rocmliblist.conf. We can now use Ubuntu 22.04 for ROCm tests. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Closes sylabs/singularity#1033 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Bring all handling of the runtime spec out of the native oci bundle package, up into the oci launcher. We obtain the bundle's image spec after it is downloaded / extracted. The launcher then computes the correct Process config and updates the bundle with it. This is required so that we can handle an image USER in the launcher. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When a USER is specified in the image config: * If running unprivileged, ensure the inner uid / gid mapping results in the container process running as the USER, by default. * If running privileged, run as the USER, by default. Fixes sylabs/singularity#77 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

In the --oci launcher, When a minimal container doesn't have an `/etc/passwd` or `/etc/groups` file then don't fail with a fatal error. Instead warn like the native runtime does. Fixes sylabs/singularity#1286 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Honour the DOCKER_HOST, `--no-https`, and auth configuration. Fixes sylabs/singularity#1220 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Honor the --apply-cgroups and individual cgroups resources flags when running in OCI mode. The launcher instructs runc/crun to create a named cgroup with specified LinuxResources in the config.json. runc/crun must be called with the `--systemd-cgroup` flag when using systemd as cgroup manager. Closes sylabs/singularity#1032 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

For the `OCI` command group, we need cgroups v2 unified mode in order to call `runc` with `--systemd-cgroup` as a non-root user. Fall back to cgroupfs operation if this constraint is not satisified. This fixes a failure when running a container using `runc`. `crun` was falling back to cgroupfs itself, even when `--systemd-cgroup` was specified. Fixes sylabs/singularity#1408 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Drop the warning that not all OCI functionality is supported with runc. Some things turned out to be SingularityCE bugs. Some things are very version specific. A new enough runc will work. Fixes sylabs/singularity#1425 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* On SLES we intend to require runc, not crun. * squashfs had a duplicate Requires, instead of a BuildRequires. Fixes sylabs/singularity#1453 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Refactor test image handling so that: * We create the OCI archive, and Docker archive, used in the e2e tests with a container/image Copy from our local registry busybox image. * We use DockerHub credentials for e2e.CopyOCIImage. * We replace a couple of uses of `library://` images with our local registry ORAS SIF, avoiding some additional network access. Fixes sylabs/singularity#1364 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* oci: implemented CDI device mapping * first batch of post-comments revisions * revisions following elezar's comments * added `--device` to CHANGELOG * fixed out-of-order calls to addCDIDevices & Update * added sync.Once getting & refreshing CDI registry * cdi unit-test stub * added some more cdi unit-tests * more unit-tests * better testing for empty mounts lists * finishing touches on cdi unit-test * first stab at cdi e2e-test * DT's temp fix for userns mapping limitation * ignore VSCode debugging build targets * renamed CDI json template file * added initial deviceNode testing * changed flag to --cdi-dirs (from --cdidirs) Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* oci: support --hostname * generalize --hostname inferring --uts across native & oci Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* added --cdi-dirs to changelog * improved comments in actionOciCdi e2e-test Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When running in --oci mode as a normal user, Singularity sets up an outer userns that provides an unprivileged root. When --fakeroot is not also requested, runc / crun then map back to the user's own uid/gid via an inner reverse mapping specified in config.json. This mapping was not being applied correctly, as a userns was not requested in the config.json for all situations where a reverse mapping was requested. Ensure that if we specify a mapping in config.json then we request a userns so that it is applied. Cover this behavior in an e2e-test by writing to a host dir from the container, and checking ownership of the file on the host. Fixes sylabs/singularity#1517 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

In `getReverseUserMaps`,we were taking `targetUID` and `targetGID` parameters that define the UID and GID the container will be entereed as. However, the mappings returned were based on the current host UID and GID, and not the target IDs. Ensure that the user maps use the `targetUID` and `targetGID`. Split the function up so that the core computation of the user maps can be tested. The tests help to explain what the intended functionality here is... which is beneficial as it's somewhat complex. Fixes sylabs/singularity#1519 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When executed from a root-owned cgroup, such as the session scope resulting from a bare ssh login, crun will fail to create our requested container cgroup. If we are running as non-root, create and move into a user-owned cgroup, so that there's a common user-owned ancestor. This avoids the `crun` error. Note that no workaround is needed for `runc` as it is able to create the requested container cgroup without any issue. Fixes sylabs/singularity#1538 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Ensure that the `config passwd` and `config group` directives in `singularity.conf` are honoured in `--oci` mode. Add e2e config tests for OCI mode that test this. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When mount proc/sys/tmp/home directives are set to no in singularity.conf, ensure we don't mount at those locations. Also enables a user bind control test, for which the implementation is already in place and working. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

…ingularity#1562) * pull HashingListSubtract into (new) high-level util package * to-be-squashed: addressing DT's comments Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

If we are running under cgroups v1 or with the cgroupfs manager (i.e. not systemd as cgroup manager), do not attempt to enter a cgroup at startup with crun. We cannot create a cgroup unprivileged in this situation. Under cgroups v1, crun will not perform the cgroups manipulation that leads to the issue we worked around in apptainer#1539. Any other issue with the cgroup that we are in at launch cannot be rectified, either. Fixes sylabs/singularity#1569 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When we run a container in `--oci` mode, we have a temporary rootfs that must be removed when the container exits. A container can contain files / dirs with restrictive permissions that prevent direct removal. Use fs.ForceRemoveAll, which will chmod & remove anything that os.RemoveAll is unable to remove. Fixes sylabs/singularity#1586 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

The `--oci` mode intends to follow behaviour that the native runtime implements when run with `--compat`. One missing aspect is that `--compat` sets `--writable-tmpfs`, where the container rootfs is made writable with a tmpfs backed overlay. This PR: - Introduces a simple wrapping of the `oci run` sub-command as `oci run-wrapped`. This hidden command implements prep / cleanup steps that must take place in a userns for non-root `--oci` execution. - Switches the oci launcher to calling `oci run-wrapped` instead of `oci-run`. - Adds a tmpfs based overlay creation function for OCI bundles. - Includes the tmpfs overlay creation in the `oci run-wrapped` flow. - Copies the native runtime `--compat` e2e tests to OCI mode. Fixes sylabs/singularity#1621 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* oci: support --pwd * replaced --pwd with --cwd (leaving former as synonym), adjusted code accordingly throughout * to-be-squashed: addressing DT's Mar 30 comments Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

…1530) Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

…st of this) Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* oci: support --overlay * support for multiple overlays, other revisions * fixup: deduplicate RunWrapped logic * lots of refactoring and cleanup * remove leftover debug-related panic call * cleanup comments, fix small issues w/erroring --------- Co-authored-by: David Trudgian <david.trudgian@sylabs.io> Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

(sylabs/singularity#1678) Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When runc/crun is called from oci_runc_linux.go, Stdin was incorrectly connected for the various runc/crun operations. * Non-interactive operations such as resume / kill don't need Stdin. * Interactive operations (run/exec) had cmd.Stdin incorrectly set to os.Stdout. This prevented OCI containers from receiving input from pipes, redirection, etc. Fixes sylabs/singularity#1712 Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

These were missed, resulting in sylabs/singularity#1712 not being caught by the e2e suite. Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

OCI-mode: support mounting of squashfs images (read-only) and extfs images (read-write or read-only) with --overlay, alongside the already existing functionality of directory-based overlays. Squashed merge of: * prepare data structures for non-dir overlays * fix e2e test (no more "auto-create overlay dir" functionality) * switch to using image.Init() to analyze image files * added caching mechanism to FindBin * first working version with squashfs support * refactor funcs into methods of OverlayItem where appropriate * standardized naming + added lots of comments * addressing first round of review comments * refactor: move non-OCI-dep. code to internal/pkg/util/fs/overlay * initial support for extfs overlay (readonly) * resurrect testing of write to persistent overlay * fix bug in writable overlay, introduced in course of refactor * removed caching mechanism from FindBin * addressing second round of review comments, except tests * move calling of prepareWritableOverlay() into Item.Mount() * removed redundant error-string wrapping * overlay.Item unit-test * overlay.Set unit-test * e2e tests for oci image overlays, misc. testing improvements Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

fix rel. path --workdir with --scratch, add oci e2e tests, from sylabs 1694

…d overlays (sylabs/singularity#1730) * fallback to fuse-overlayfs if kernel doesn't support unpriv overlays * addressed first round of review comments * fix code-flow in UnprivOverlaysSupported() for unsupported case Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

(sylabs/singularity#1740) * oci: support for writable extfs img overlay via fuse-overlayfs * added unit- and e2e-tests * fixed exposure of "upper" & "work" subdir in readonly overlays * added fix for file-ownership in FUSE-mounted images + e2e test of fix * fix handling of "upper" in :ro overlay dirs, adjust e2e tests Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Dave Dykstra <2129743+DrDaveD@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oci: fallback to fuse-overlayfs if kernel doesn't support unprivileged overlays and oci: support for writable extfs img overlay via fuse-overlayfs, from sylabs 1730 & 1740 #335

oci: fallback to fuse-overlayfs if kernel doesn't support unprivileged overlays and oci: support for writable extfs img overlay via fuse-overlayfs, from sylabs 1730 & 1740 #335

Commits on Jul 4, 2023

Commits on Jul 6, 2023

Commits on Jul 7, 2023