[Error] /etc/localtime failed to bind mount during distrobox-init

[weiren2]

Describe the bug
The command distrobx enter ... failed due to bind mount failure of /etc/localtime. The backing filesystem is ZFS (2.2.0) with native overlay enabled.

To Reproduce

• Create a new distrobox: distrobox create --name ubuntu-22.04 --image ubuntu:22.04 --home /path/to/ubuntu-home
• Enter: distrobox enter ubuntu-22.04

Expected behavior
We should expect the commands finish without error, initialize all the mount points successfully, and enter the distrobox/container.

Logs

• Log of distrobox enter ubuntu-22.04: distrobox-enter.log
• Log from podman: podman.log

Desktop (please complete the following information):

• Are you using podman, docker or lilipod?
  • podman
• Which version or podman, docker or lilipod?
  • podman 4.7.1
• Which version of distrobox?
  • distrobox 1.5.0.2
• Which host distribution?
  • Arch Linux
• How did you install distrobox?
  • From Arch's repo (extra)

Additional context
This issue seems to happen only when the backing filesystem is ZFS. I did not encounter such error when using xfs or btrfs.

As a workaround, I moved /etc/localtime from HOST_MOUNTS_RO to HOST_MOUNTS, which I think avoids being (bind) mounted as read-only. I'm not sure if this is a valid fix, but it shouldn't break any permission work assuming rootless podman/docker.

distrobox/distrobox-init

Lines 1394 to 1441 in 9309e1e

	# We'll also bind mount in READ-ONLY useful directories from the host
	HOST_MOUNTS_RO="
	/etc/localtime
	/var/lib/systemd/coredump
	/var/log/journal"
	for host_mount_ro in ${HOST_MOUNTS_RO}; do
	# Mounting read-only in a user namespace will trigger a check to see if certain
	# "locked" flags are changed. This ensures we explicitly reuse those flags.
	locked_flags="$(get_locked_mount_flags /run/host"${host_mount_ro}")"
	mount_bind /run/host"${host_mount_ro}" "${host_mount_ro}" ro"${locked_flags:+,${locked_flags}}"
	done
	###############################################################################
	###############################################################################
	printf "distrobox: Setting up read-write mounts...\n"
	# On some ostree systems, home is in /var/home, but most of the software expects
	# it to be in /home. In the hosts systems this is fixed by using a symlink.
	# Do something similar here with a bind mount.
	if [ -e "/var/home/${container_user_name}" ]; then
	if ! mount_bind "/run/host/var/home/${container_user_name}" "/home/${container_user_name}"; then
	printf "Warning: Cannot bind mount %s to /run/host%s\n" "/var/home" "/home"
	fi
	fi
	# We'll also bind mount READ-WRITE useful mountpoints to pass external drives and libvirt from
	# the host to the container
	HOST_MOUNTS="
	/etc/host.conf
	/etc/machine-id
	/media
	/mnt
	/run/libvirt
	/run/media
	/run/netconfig/
	/run/systemd/journal
	/run/systemd/resolve/
	/run/systemd/seats
	/run/systemd/sessions
	/run/systemd/users
	/run/udev
	/var/lib/libvirt
	/var/mnt"
	for host_mount in ${HOST_MOUNTS}; do
	if ! mount_bind /run/host"${host_mount}" "${host_mount}"; then
	printf "Warning: Cannot bind mount %s to /run/host%s\n" "${host_mount}" "${host_mount}"
	fi
	done

And it seemed working (at least the commands finished without errors/warnings and I can continue working inside container).

Let me know if you need any further testing.

[rrahl0]

This bug is fixed in the main version. So you would need to wait for a new release, or the git version.

[weiren2]

This bug is fixed in the main version. So you would need to wait for a new release, or the git version.

The same error still occurred when I tested with the latest git version just now.

@rrahl0 Which bug or commit are you referring to (that is supposed to fix the issue)? I can help look into the issue to see if the fix can be applied.

[rrahl0]

@weiren2 pretty sure it was 24ff0c0. at least on my fedora/tumbleweed installs it works fine

[weiren2]

The error I'm getting is: (also in the complete log)

+ mount --rbind -o ro /run/host/usr/share/zoneinfo/America/Chicago /etc/localtime
mount: /etc/localtime: filesystem was mounted, but any subsequent operation failed: Unknown error 5005.
+ printf Warning: failed to bind mount %s to %s\n /run/host/usr/share/zoneinfo/America/Chicago /etc/localtime
+ return 1
+ [ 1 -ne 0 ]
+ printf Error: An error occurred\n

My guess is that it may be related to zfs being used as the underlying filesystem, which includes additional mount option like nodev (and it cannot be removed with rootless). By moving /etc/localtime from the read-only mount points (HOST_MOUNTS_RO) to the read-write mount points (HOST_MOUNTS), I was able to finishing running the distrobox-init script without errors on my particular system.

@89luca89 Is there a specific reason that /etc/localtime needs to be mounted as read-only?

FYI I found this PR (containers/toolbox#1340) in the toolbox project that fixed a similar issue.

[weiren2]

I did some further debugging. It turns out that the issue is related to missing mount flags when bind mounting as ro. This function get_locked_mount_flags does not check for mount flags if the mount point is a file, but /etc/localtime is indeed a file (and it can have flags like nodev).

I've created a PR #1041 to address this issue.

This issue does not happen when using BTRFS or XFS as the underlying root file system, since there is no additional flags like nodev when using kernel-native filesystems for root. But with ZFS as root file system, the mount point ends up having nodev by default.

On a side note, the workaround (changing to read-write mount) is just a red herring. Changing the mount mode to rw works because there is no remount. [Source: containerd/containerd/pull/6913.]