wiki: Ubuntu 16.04 Root on ZFS /var/tmp and encrypted swap issues

[pdf]

1. I had repeated failures on boot mounting /var/tmp after following the wiki article (section 3.3). It appears that systemd can race zfs mounting for writes to /var/tmp. The only way I could consistently have /var/tmp as a ZFS subvol on boot was to use the -o overlay=true option. I'm not sure this is ideal though, since it will lead to inconsistent persisting of the files there.

2. The second issue I had - that is pretty painful to recover from - was related to encrypted swap (section 7.2b). In the article, section 6.6 on creating a home directory provides an optional path (6.6b) for encrypting the user's homedir, which specifies installing ecryptfs-utils. If the user elects to instead follow 6.6a for unencrypted homedir, but chooses to follow 7.2b for encrypted swap, they will run into problems as they're missing the required cryptsetup binaries. The result is that systemctl start systemd-cryptsetup@cryptswap1.service hangs indefinitely, and if the system is rebooted before the tools are installed, boot will hang indefinitely as this service is a dependency for the device initialisation target, which has no timeout, and must succeed before networking/service/tty startup, even if there's no swap device configured in /etc/fstab. The only recourse to fix the problem then is to reboot to recovery and install the required dependencies (this is probably a bug that should be fixed upstream - the service should fail on missing tools - but the article still needs updating to be functional).

TLDR 2. Section 7.2b should specify apt install cryptsetup as first step.

ping @rlaager

[rlaager]

On the first item... I ran into that too. The issue is with services that use systemd's PrivateTmp feature. I'm not sure what the right solution is (besides maybe better systemd mount integration?). I need to look into this a little more. If I can't find anything better, then regretably, I may need to just remove /var/tmp. I'll try to finish this up one way or another in the next few days.

On the second item... that's an easy fix. I've made that change. Thanks!

[pdf]

Yeah, it's unfortunate to have to remove /var/tmp because we can't disable auto-snapshots on it, for example.

What's interesting is that all services that use PrivateTmp already specify After=... -.mount ... tmp.mount var-tmp.mount ... and RequiresMountsFor=/tmp /var/tmp, but the way ZFS mounts are currently handled, the mount targets may not be available then.

I've just had a browse through the issue tracker, and it seems that improvements to systemd integration really are required, related issues for posterity: #4898 #4943 #3836 #4178 #3768 #4784 zfsonlinux/pkg-zfs#205

[rlaager]

@pdf How are you seeing that After specification? I'm only getting a RequiresMountsFor.
systemctl show -p After -p RequiresMountsFor systemd-timesyncd.service

[pdf]

@rlaager I believe After will only be populated if you have active mount units for those mountpoints, though there may also be some distro-specific patches involved depending on which systemd version you're using.

[rlaager]

I believe I found a way to work-around this, until we have a systemd generator (#4898). If we set the mountpoint to legacy and list the filesystem in /etc/fstab, then systemd is aware of it. When I fixed /var/tmp, I seemed to always lose the /var/log race (which happened intermittently before). I fixed that similarly. This is now step 4.9 in the instructions.

Can you give that a try?

[pdf]

This appears to be an adequate work-around in the short-term.

Is there any chance though of file handles being held open on either mountpoint at the time mountpoint=legacy is set? This would result in failed unmount, and hence failure to set the prop. Might be safer to set mountpoint=legacy and manually mount at initial dataset creation (3.3).

[rightaditya]

I have no idea why this worked, but I was able to workaround this without using a legacy mountpoint by setting subvol to be rpool/log and rpool/var-tmp; i.e., I don't create a subvolume for /var and put the subvolumes for /var/log and /var/tmp directly under rpool. I then set the mountpoints as appropriate to the desired paths.