Disappearing snapshots

[TyberiusPrime]

Hello,

I have a reoccurring occurrence where a snapshot that was accessed just now suddenly has no files in .zfs/snapshot/xyz. It was being accessed via samba - mostly an sqlite file being reopened often (websystem...) inside the snapshot.

The data is not gone though.

• rebooting restores the snapshot
• I believed I once was able to restore it by unmounting/mounting the zfs in question
• but now I think it might have been a timeout thing - because the snapshot once reappeared
  after about 10 minutes (300 second timeout?) - and because I haven't been able
  to unmount the zfs in question during later occurrences.
• toggling snapdir = visible and back to hidden does not help.

I can not mount the snapshot somewhere else with mount -t zfs - it claims
it is already mounted.
This suggests to me that it's not the unmount behavior that's causing the disappearance.

I'm on ubuntu 14.04 with the current 8~trusty package from zfs-native/stable.
Scrub is clean, RAM is ecc.

I've made a lengthy zfs-discuss thread where I kept posting 'life',
but I think the above description distills the relevant information:
https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/aAJsQoSY3BQ

Best regards
Florian

[behlendorf]

@TyberiusPrime it sounds as if the snapshot is being automatically unmounted and then for some reason it's not being automatically remounted when it's needed again.

Normally when you traverse in to a snapshot directory it will be auto-mounted where it is needed. You can actually see this if you cat /proc/mounts and look for the mount point. Periodically ZFS will attempt to unmount the snapshot, every 300 seconds by default. If the mount point is busy this will fail harmlessly, if it's not the mount will succeed. Any subsequent lookup in that directory should trigger the auto-mount again so if it's needed it will be available.

It sounds as if you might have found a case where this isn't always working properly. If you're looking for a possible workaround until the root cause can be determined I can suggest two things.

1. Increase the zfs_expire_snapshot value to prevent the timeout from occurring. Leaving the snapshot mounted indefinitely won't be a problem unless you need to concurrently access 1000's of snapshots this way.

2. Somehow keep an open file handle in the snapshot. This should prevent the unmount from succeeding until that file handle is closed.

[TyberiusPrime]

The mount is not in /proc/mounts - other snapshots are.
I can not remount it with mount -t zfs - 'is already mounted' - I can mount other unmounted snapshots.

Seems it's stuck somewhere partial into the unmounting.

Sorry, where do I need to stick zfs_expire_snapshot? /etc/default/zfs? Every other variable there is upper case.

[TyberiusPrime]

Now this is weird. I had it happen again, this time in a zfs where there was some write traffic. Snapshot disappeared. I ended the processes writing to the ZFS - and the snapshot reappeared.

Trying to keep an open file in the snapshot now.

[behlendorf]

@TyberiusPrime It's an module option, add it to /etc/modprobe.d/zfs.

$ modinfo zfs | grep expire
parm:           zfs_expire_snapshot:Seconds to expire .zfs/snapshot (int)

[TyberiusPrime]

It occured again today - timeout is set to 3000 seconds, package is 8~trusty from the ubuntu stable ppa.

[Mic92]

I have the same issue, but rebooting did not fix my problem.
I saw the following line in my dmesg:

Unable to automount zroot/root@zfs-backup-56 at /.zfs/snapshot/zfs-backup-56: 512

After digging in the zfsonlinux code, I found the source, where the error was thrown:

//module/zfs/zfs_ctldir.c:849
argv[2] = kmem_asprintf(SET_MOUNT_CMD, full_name, full_path);                                                                                                                                                   
error = call_usermodehelper(argv[0], argv, envp, UMH_WAIT_PROC);                                                                                                                                                
strfree(argv[2]);                                                                                                                                                                                               
if (error && !(error & MOUNT_BUSY << 8)) {                                                                                                                                                                      
    printk("ZFS: Unable to automount %s at %s: %d\n",                                                                                                                                                           
        full_name, full_path, error);                                                                                                                                                                           
    error = SET_ERROR(EISDIR);                                                                                                                                                                                  
    goto error;                                                                                                                                                                                                 
} 

where argv is a shell issuing a mount command for example:

sh -c exec 0</dev/null      1>/dev/null      2>/dev/null; mount -t zfs -n 'zroot/root@zfs-auto-snap_hourly-2014-12-26-0900' '/.zfs/snapshot/zfs-auto-snap_hourly-2014-12-26-0900'

After running the command manually:

$ mount -t zfs -n 'zroot/root@zfs-auto-snap_hourly-2014-12-26-0900' '/.zfs/snapshot/zfs-auto-snap_hourly-2014-12-26-0900'

it mounts without problems, but:

cd zfs-auto-snap_hourly-2014-12-26-0900
-bash: cd: zfs-auto-snap_hourly-2014-12-26-0900: Input/output error

this is probably because zfs tries to mount the snapshot twice.
If I however choose a different mountpoint, it works:

$ mount -t zfs -n 'zroot/root@zfs-auto-snap_hourly-2014-12-26-0900' '/mnt/snapshot/zfs-auto-snap_hourly-2014-12-26-0900'

To debug this problem further, I replaced /bin/mount with my own version:

#!/bin/bash

echo "$@" >> /tmp/mount.log
exec /bin/mount.org "$@" 2>&1 | tee -a /tmp/mount.log

Where I got the following error output, if I tried to access a snapshot directory:

-t zfs -n zroot/root@zfs-auto-snap_hourly-2014-12-26-1300 /.zfs/snapshot/zfs-auto-snap_hourly-2014-12-26-1300 (deleted)
filesystem 'zroot/root@zfs-auto-snap_hourly-2014-12-26-1300' cannot be mounted at '/.zfs/snapshot/zfs-auto-snap_hourly-2014-12-26-1300 (deleted)' due to canonicalization error 2.

After tracking this bug further I came to the following snippet:

    // cmd/mount_zfs/mount_zfs.c:430
    /* canonicalize the mount point */                                                                                                                                                                              
    if (realpath(argv[1], mntpoint) == NULL) {                                                                                                                                                                      
        (void) fprintf(stderr, gettext("filesystem '%s' cannot be "                                                                                                                                                 
            "mounted at '%s' due to canonicalization error %d.\n"),                                                                                                                                                 
            dataset, argv[1], errno);                                                                                                                                                                               
        return (MOUNT_SYSERR);                                                                                                                                                                                      
    }  

It fails to execute realpath, where error 2 means "No such file or directory".

After replacing mount with a more advanced mount wrapper:

#!/usr/bin/python
import sys
import subprocess

args = []
for arg in sys.argv:
  if " (deleted)" in arg:
    arg = arg.replace(" (deleted)", "")
    os.mkdir(arg)
    args.append(arg)
  else:
    args.append(arg)
args[0] = "/bin/mount.org"

log = open("/tmp/mount.log", "w+", 1)
proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

for l in proc.stdout:
  encoded = l.decode("utf-8")
  sys.stdout.write(encoded)
  log.write(encoded)

for l in proc.stderr:
  encoded = l.decode("utf-8")
  sys.stdout.write(encoded)
  log.write(encoded)

proc.communicate()

... mount fails with the following error:

filesystem 'zroot/root@zfs-auto-snap_daily-2014-12-22-2302' cannot be mounted at '/.zfs/snapshot/zfs-auto-snap_daily-2014-12-22-2302' due to canonicalization error 5.

error 5 is EIO (I/O error)

[behlendorf]

@Mic92 Thanks for digging in to this. Just one question, it wasn't clear to me from the testing you did if you verified that the mount point directory exists before manually issuing the mount command. It should, these directories are dynamically generated based on the list of snapshots.

[behlendorf]

I should add that this might be fixed when yshui/zfs@3ab9d33578e0ad26c31bb7c6a55a795e3b5e1af5 is merged. This fixes a regression which would cause one of the .zfs/snapshot directories to be missing.

[Mic92]

@behlendorf It exists. From my understanding https://github.com/yshui/zfs/commit/3ab9d33578e0ad26c31bb7c6a55a795e3b5e1af5 should only affect only one snapshot, but in my case it affects all snapshots. My questions is, why the second argument ends with a (deleted) suffix.

[behlendorf]

@Mic92 (deleted) will be automatically appended to a directory entry (dentry) which isn't hashed in the dcache. I think you're definitely on the right track here, the key will be figuring out how and why the dentry is getting unhashed.

[mehuser]

I've tried merging yshui@3ab9d33 but the problem still persists.

[behlendorf]

@maci0 what kernel version are you using? This may only impact 3.18 kernels.

[Mic92]

@behlendorf Yes 3.18 breaks this feature.

[behlendorf]

I've proposed a patch in issue #3344 to address this.