oci: fallback to fuse-overlayfs if kernel doesn't support unprivileged overlays and oci: support for writable extfs img overlay via fuse-overlayfs, from sylabs 1730 & 1740

.github/workflows/ci.yml

@@ -238,7 +238,12 @@ jobs:
           go-version: 1.20.5
 
       - name: Fetch deps
-        run: sudo apt-get -q update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential squashfs-tools squashfuse fuse-overlayfs fakeroot fuse2fs libseccomp-dev cryptsetup dbus-user-session
+        run: sudo apt-get -q update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential squashfs-tools squashfuse fuse-overlayfs fakeroot fuse2fs libseccomp-dev cryptsetup dbus-user-session conmon
+
+      - name: Install crun
+        run: |
+          sudo curl -L -o /usr/local/bin/crun https://github.com/containers/crun/releases/download/1.6/crun-1.6-linux-amd64
+          sudo chmod +x /usr/local/bin/crun
 
       - name: Build and install Apptainer
         run: |
@@ -268,7 +273,12 @@ jobs:
           go-version: 1.20.5
 
       - name: Fetch deps
-        run: sudo apt-get -q update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential squashfs-tools libseccomp-dev cryptsetup dbus-user-session
+        run: sudo apt-get -q update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential squashfs-tools libseccomp-dev cryptsetup dbus-user-session conmon
+
+      - name: Install crun
+        run: |
+          sudo curl -L -o /usr/local/bin/crun https://github.com/containers/crun/releases/download/1.6/crun-1.6-linux-amd64
+          sudo chmod +x /usr/local/bin/crun
 
       - name: Build and install Apptainer
         run: |
@@ -314,7 +324,12 @@ jobs:
 
       - name: Fetch deps
         if: env.run_tests
-        run: sudo apt-get -q update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential uidmap squashfs-tools squashfuse fuse-overlayfs fakeroot fuse2fs libseccomp-dev cryptsetup dbus-user-session
+        run: sudo apt-get -q update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential uidmap squashfs-tools squashfuse fuse-overlayfs fakeroot fuse2fs libseccomp-dev cryptsetup dbus-user-session conmon
+
+      - name: Install crun
+        run: |
+          sudo curl -L -o /usr/local/bin/crun https://github.com/containers/crun/releases/download/1.6/crun-1.6-linux-amd64
+          sudo chmod +x /usr/local/bin/crun
 
       - name: Fetch gocryptfs
         run: wget -O gocryptfs.tar.gz https://github.com/rfjakob/gocryptfs/releases/download/v2.3/gocryptfs_v2.3_linux-static_amd64.tar.gz && sudo tar xzvf gocryptfs.tar.gz -C /usr/local/bin gocryptfs
@@ -362,6 +377,7 @@ jobs:
           retention-days: 7
 
   check_pkg_no_buildcfg:
+    if: ${{ github.base_ref != 'oci-action' }}
     name: check_pkg_no_buildcfg
     runs-on: ubuntu-22.04
     steps:

.gitignore

@@ -110,3 +110,12 @@ pkg/library/client/test[0-9]*
 /debian
 
 LICENSE_DEPENDENCIES.csv
+
+# VSCode debugging build targets
+__debug_bin
+*/__debug_bin
+*/*/__debug_bin
+*/*/*/__debug_bin
+*/*/*/*/__debug_bin
+*/*/*/*/*/__debug_bin
+*/*/*/*/*/*/__debug_bin

CHANGELOG.md

@@ -13,13 +13,77 @@ For older changes see the [archived Singularity change log](https://github.com/a
   working directory, though `--pwd` is still supported for compatibility.
 - When building RPM, we will now use `/var/lib/apptainer` (rather than
   `/var/apptainer`) to store local state files.
+- The `apptainer oci` command group now uses `crun`, when available, or otherwise
+  `runc` to manage containers.
+- The `apptainer oci` flags `--sync-socket`, `--empty-process`, and
+  `--timeout` have been removed.
+- `sessiondir maxsize` in `apptainer.conf` now defaults to 64 MiB for new
+  installations. This is an increase from 16 MiB in prior versions.
 
 ### New Features & Functionality
 
 - The `remote status` command will now print the username, realname, and email
   of the logged-in user, if available.
 - New option `--warn-unused-build-args` is provided to output warnings rather than
   fatal errors for any unused variables given in --build-arg or --build-arg-file.
+- A new `--oci` flag for `run/exec/shell` enables the experimental OCI runtime
+  mode. This mode:
+  - Runs OCI container images from an OCI bundle, using `runc` or `crun`.
+  - Supports `docker://`, `docker-archive:`, `docker-daemon:`, `oci:`,
+    `oci-archive:` image sources.
+  - Does not support running Apptainer SIF, SquashFS, or EXT3 images.
+  - Provides an environment similar to Apptainer's native runtime, running
+    with `--compat`.
+  - Supports the following options / flags. Other options are not yet supported:
+    - `--fakeroot` for effective root in the container. Requires subuid/subgid
+      mappings.
+    - Bind mounts via `--bind` or `--mount`. No image mounts.
+    - Additional namespaces requests with `--net`, `--uts`, `--user`.
+    - Container environment variables via `--env`, `--env-file`, and
+      `APPTAINERENV_` host env vars.
+    - `--rocm` to bind ROCm GPU libraries and devices into the container.
+    - `--nv` to bind Nvidia driver / basic CUDA libraries and devices into
+      the container.
+    - `--apply-cgroups`, and the `--cpu*`, `--blkio*`, `--memory*`,
+      `--pids-limit` flags to apply resource limits.
+- Added `--device` flag to "action" commands (`run`/`exec`/`shell`) when run in
+  OCI mode (`--oci`). Currently supports passing one or more (comma-separated)
+  fully-qualified CDI device names, and those devices will then be made
+  available inside the container.
+- Added `--cdi-dirs` flag to override the default search locations for CDI
+  json files, allowing, for example, users who don't have root access on their
+  host machine to nevertheless create CDI mappings (into containers run with
+  `--fakeroot`, for example).
+- OCI mode now supports `--hostname` (requires UTS namespace, therefore this
+  flag will infer `--uts`).
+- OCI mode now supports `--scratch` (shorthand: `-S`) to mount a tmpfs scratch
+  directory in the container.
+- Support `--pwd` in OCI mode.
+- OCI mode now supports `--home`. Supplying a single location (e.g.
+  `--home /myhomedir`) will result in a new tmpfs directory being created at the
+  specified location inside the container, and that dir being set as the
+  in-container user's home dir. Supplying two locations separated by a colon
+  (e.g. `--home /home/user:/myhomedir`) will result in the first location on the
+  host being bind-mounted as the second location in-container, and set as
+  the in-container user's home dir.
+- OCI mode now handles `--dns` and `resolv.conf` on par with native mode: the
+  `--dns` flag can be used to pass a comma-separated list of DNS servers that
+  will be used in the container; if this flag is not used, the container will
+  use the same `resolv.conf` settings as the host.
+- OCI-mode now supports the `--overlay <arg>` flag. `<arg>` can be the path to a
+  writable directory or writable extfs image, in which case changes to the
+  filesystem will persist across runs of the OCI container. Alternatively,
+  `--overlay <arg>:ro` can be used, where `<arg>` is the path to a directory, to
+  a squashfs image, or to an extfs image, to be mounted as a read-only overlay.
+  Multiple overlays can be specified, but all but one must be read-only.
+- OCI-mode now supports the `--workdir <workdir>` option. If this option is
+  specified, `/tmp` and `/var/tmp` will be mapped, respectively, to
+  `<workdir>/tmp` and `<workdir>/var_tmp` on the host, rather than to tmpfs
+  storage. If `--scratch <scratchdir>` is used in conjunction with `--workdir`,
+  scratch directories will be mapped to subdirectories nested under
+  `<workdir>/scratch` on the host, rather than to tmpfs storage.
+- If kernel does not support unprivileged overlays, OCI-mode will attempt to use
+  `fuse-overlayfs` and `fusermount` for overlay mounting and unmounting.
 
 ### New Features & Functionality
 

INSTALL.md

@@ -28,9 +28,16 @@ sudo apt-get install -y \
     fuse-overlayfs \
     fakeroot \
     cryptsetup \
-    curl wget git
+    curl wget git \
+    conmon crun
 ```
 
+_Note_: on Ubuntu 18.04 or Debian 10 leave out `conmon`, `crun`, and
+`fuse-overlayfs` because they are not available, or install them from another
+source. Leaving out the first two will prevent the `--oci` option from working
+and leaving out the third will prevent `--overlay` and `--writable-tmpfs`
+options from working without suid mode.
+
 On CentOS/RHEL:
 
 ```sh
@@ -47,9 +54,12 @@ sudo yum install -y \
     fakeroot \
     /usr/*bin/fuse2fs \
     cryptsetup \
-    wget git
+    wget git \
+    conmon crun
 ```
 
+_Note - use `runc` instead of `crun` on CentOS/RHEL 7._
+
 On SLE/openSUSE
 
 ```sh
@@ -59,9 +69,13 @@ sudo zypper install -y \
   libuuid-devel \
   openssl-devel \
   cryptsetup sysuser-tools \
-  gcc go
+  gcc go \
+  conmon crun
 ```
 
+_Note - `crun` / `runc` can be omitted if you will not use the `apptainer oci`
+commands, or the `--oci` execution mode._
+
 ## Install Go
 
 Apptainer is written in Go, and may require a newer version of Go than is

LICENSE_DEPENDENCIES.md

@@ -17,6 +17,12 @@ The dependencies and their licenses are as follows:
 
 **License URL:** <https://github.com/Netflix/go-expect/blob/master/LICENSE>
 
+## github.com/container-orchestrated-devices/container-device-interface
+
+**License:** Apache-2.0
+
+**License URL:** <https://github.com/container-orchestrated-devices/container-device-interface/blob/master/LICENSE>
+
 ## github.com/containerd/containerd
 
 **License:** Apache-2.0
@@ -35,6 +41,12 @@ The dependencies and their licenses are as follows:
 
 **License URL:** <https://github.com/containernetworking/plugins/blob/master/LICENSE>
 
+## github.com/containers/common
+
+**License:** Apache-2.0
+
+**License URL:** <https://github.com/containers/common/blob/master/LICENSE>
+
 ## github.com/containers/image/v5
 
 **License:** Apache-2.0
@@ -53,11 +65,11 @@ The dependencies and their licenses are as follows:
 
 **License URL:** <https://github.com/containers/ocicrypt/blob/master/LICENSE>
 
-## github.com/containers/storage/pkg
+## github.com/containers/storage
 
 **License:** Apache-2.0
 
-**License URL:** <https://github.com/containers/storage/blob/master/pkg/LICENSE>
+**License URL:** <https://github.com/containers/storage/blob/master/LICENSE>
 
 ## github.com/coreos/go-iptables/iptables
 
@@ -287,6 +299,18 @@ The dependencies and their licenses are as follows:
 
 **License URL:** <https://github.com/opencontainers/runtime-spec/blob/master/specs-go/LICENSE>
 
+## github.com/opencontainers/runtime-tools
+
+**License:** Apache-2.0
+
+**License URL:** <https://github.com/opencontainers/runtime-tools/blob/master/LICENSE>
+
+## github.com/opencontainers/selinux
+
+**License:** Apache-2.0
+
+**License URL:** <https://github.com/opencontainers/selinux/blob/master/LICENSE>
+
 ## github.com/opencontainers/umoci
 
 **License:** Apache-2.0
@@ -521,6 +545,12 @@ The dependencies and their licenses are as follows:
 
 **License URL:** <https://github.com/cyphar/filepath-securejoin/blob/master/LICENSE>
 
+## github.com/fsnotify/fsnotify
+
+**License:** BSD-3-Clause
+
+**License URL:** <https://github.com/fsnotify/fsnotify/blob/master/LICENSE>
+
 ## github.com/gogo/protobuf/proto
 
 **License:** BSD-3-Clause
@@ -623,6 +653,12 @@ The dependencies and their licenses are as follows:
 
 **Project URL:** <https://golang.org/x/exp>
 
+## golang.org/x/mod/semver
+
+**License:** BSD-3-Clause
+
+**Project URL:** <https://golang.org/x/mod/semver>
+
 ## golang.org/x/net
 
 **License:** BSD-3-Clause
@@ -869,6 +905,12 @@ The dependencies and their licenses are as follows:
 
 **License URL:** <https://github.com/rivo/uniseg/blob/master/LICENSE.txt>
 
+## github.com/samber/lo
+
+**License:** MIT
+
+**License URL:** <https://github.com/samber/lo/blob/master/LICENSE>
+
 ## github.com/secure-systems-lab/go-securesystemslib/dsse
 
 **License:** MIT
@@ -917,6 +959,12 @@ The dependencies and their licenses are as follows:
 
 **Project URL:** <https://gopkg.in/yaml.v3>
 
+## sigs.k8s.io/yaml
+
+**License:** MIT
+
+**Project URL:** <https://sigs.k8s.io/yaml>
+
 ## github.com/gosimple/slug
 
 **License:** MPL-2.0