Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a mutable mode for systemd-sysext/confext #986

Closed
pothos opened this issue Mar 24, 2023 · 6 comments
Closed

Support a mutable mode for systemd-sysext/confext #986

pothos opened this issue Mar 24, 2023 · 6 comments

Comments

@pothos
Copy link
Member

pothos commented Mar 24, 2023

Support a mutable mode for systemd-sysext and systemd-confext. For systemd-sysext this mutable mode will help traditional distro users on Debian or Fedora become accustomed to systemd-confext. See systemd/systemd#24864 for the current discussion.
For systemd-confext we could use this mutable mode for our /etc overlay mount, making /usr/share/flatcar/etc a syscfg directory and setting up a symlink to it from /usr/lib/confexts/.
The mount command we use is mount -t overlay overlay -o lowerdir=/sysroot/usr/share/flatcar/etc,upperdir=/sysroot/etc,workdir=/sysroot/.etc-work,redirect_dir=on,metacopy=off,noatime /sysroot/etc (TODO: check implications of redirect_dir enabled vs disabled).

We've noticed that systemd-tmpfiles causes upcopies for C, L, d, Z rules which may be something to fix if the sysext would ship a file that is also covered by a systemd-tmpfiles rules (which could even be indirect if /etc is set up for selinux relabeling with Z).

@pothos pothos added kind/feature A feature request area/sysext sysext roadmap labels Mar 24, 2023
@pothos pothos removed the kind/feature A feature request label Mar 24, 2023
@pothos pothos moved this from 🪵Backlog to 🌱 Upcoming / Focus in Flatcar tactical, release planning, and roadmap May 23, 2023
@pothos pothos changed the title Support a mutable mode for systemd-sysext/syscfg Support a mutable mode for systemd-sysext/confext May 25, 2023
@t-lo
Copy link
Member

t-lo commented Jun 1, 2023

Notes from our discussions with upstream. A note on expectation management: the below is strongly work-in-progress, nothing is committed to. This is a snapshot in time of our discussions with the systemd folks.

To ease migration of existing workloads to (ultimately) immutable /etc, /usr, and /opt as well as to provide an intermediate option that allows operating hybrid systems that support both traditional software deployments as well as sysext/confext, the system may support three modes:

  1. Automatic mode - "merge" will auto-detect how to handle directories based on the presence of /[dir].local directories in the root filesystem.
    1. By default, there is no change in behavior. Overlay mounts are immutable.
    2. If, for any /etc, /usr, or /opt, if a directory /[dir].local exists (e.g. /etc.local for /etc ) then the overlay is mutable and uses the respective /[dir].local as its writable "upperdir". Changes in [dir] will be written to [dir.local].
    3. If /[dir].local is a symlink then the symlink's realpath is used as the writable upperdir. This includes symlinks to the actual mountpoint - if /[dir].local -> /[dir], then /[dir] itself is the writable upperdir and hence will retain user changes even after the confext/sysext is unmerged. One major use case of this scenario is shipping "default" / "vendor" configurations in a sysext which can be customised / overridden by the user.
  2. Configuration file controlled: /etc/systemd/[sysext|confext].conf control the behavior of "merge". This overrides "Automatic mode".
    1. By default, there is no change in behavior. Overlay mounts are immutable.
    2. Via the configuration file (.ini format) the system's behavior can be set to
      1. immutable: default, no change. Set explicitly, it can be used to override [dir.local] auto-detection described above.
      2. mutable: mount target becomes "upperdir", is writable, and will retain user changes. This is the equivalent of the "symlink" use case 1.iii. Note that if the underlying mount point is read-only (e.g. /usr residing on a read-only partition) this option will not apply for that respective directory, i.e. that directory will remain read-only. To enable / retain writes, users may use option 2.ii.c (ephemeral) below, or option 1.ii ([dir].local) above.
      3. ephemeral: mount target is writable but uses a tmpfs as "upperdir", therefore changes are possible as long as the sysext is in use but will not be retained after unmerge.
  3. Controlled by command line options. This overrides the configuration file.
    1. By default, there is no change in behavior. Overlay mounts are immutable.
    2. --immutable the default. Can be used explicitly to override mutable or ephemeral modes configured in the config file.
    3. --mutable similar to 2.ii.b above. The same special case handling for read-only mount target applies.
    4. --ephemeral similar to 2.ii.c above

@t-lo
Copy link
Member

t-lo commented Sep 26, 2023

uapi-group pr here: uapi-group/specifications#78

Please reference this when discussing mutability with upstream systemd (PRs, issues, and otherwise).

@pothos
Copy link
Member Author

pothos commented Nov 6, 2023

(TODO: check implications of redirect_dir enabled vs disabled).

Some distros like Flatcar enable it by default through the kernel option even if the mount option isn't passed. I tried the "undefined" behavior and it's basically that when removing the original folder in the lowerdir, the upper folder won't show the contents of the lowerdir anymore but only those present in the moved folder of the upperdir. That's a sane behavior and I think we should always pass the redirect_dir=on mount option. Not passing it would result in upcopies when a folder is renamed even if it's directly renamed back, and the updates of the lowerdir won't be propagated to the upperdir.

@pothos pothos moved this from 🌱 Upcoming / Focus to ⚒️ In Progress in Flatcar tactical, release planning, and roadmap Nov 28, 2023
@krnowak
Copy link
Member

krnowak commented Mar 26, 2024

This PR brings in patches for mutable overlays into Flatcar: flatcar/scripts#1753. The patches are already upstreamed, should be a part of systemd v256 release.

@pothos
Copy link
Member Author

pothos commented Apr 29, 2024

The migration of /usr/share/flatcar/etc to be a confext should be good to go. Probably through a symlink in /usr/lib/confexts/flatcar-defaults/etc, and a symlink /var/lib/extensions.mutable/etc → /etc, and then calling systemd-confext --mutable=yes --root=/sysroot in bootengine instead of the custom mount command.

A follow-up note on the =import and =ephemeral-import modes when we use a symlink to the original directory for the mutable setting: Currently this is not supported (error is Not importing mutable directory for hierarchy /etc as a lower dir, because it points to the hierarchy itself) but it would be nice to allow users to have a read-only or ephemeral mount while letting the mutable i.e. original directory win against sysexts. The problem with =yes or =ephemeral is that when we convert /usr/share/flatcar/etc to be a confext, that would overlay the original /etc in these modes, losing user changes. When we make =import and =ephemeral-import work we have a sane setting that users could use for the goal of having a read-only /etc or a ephemeral /etc.

@sayanchowdhury
Copy link
Member

Closing as per discussion during the October 2024 community meeting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants