Make README's recursive dataset snapshot pruning warning clearer

[danboid]

I have a ZFS LTSP server which I want to recursively replicate /home and its child datasets (each user has their own dataset within /home) to a remote ZFS server then have the snapshots thinned out on both sides but this section of the znapzend readme has me doubting whether znapzend would work for me or not:

"Note that while recursive configurations are well supported to set up backup and retention policies for a whole dataset subtree under the dataset to which you have applied explicit configuration, at this time pruning of such trees ("I want every dataset under var except var/tmp") is not supported. You probably do not want to enable ZnapZend against the root datasets of your pools due to that, but would have to be more fine-grained in your setup. This is consistent with (and due to) usage of recursive ZFS snapshots, where the command is targeted at one dataset and impacts it and all its children, allowing to get a consistent point-in-time set of snapshots across multiple datasets."

Does this mean that znapzend doesn't support pruning recursive datasets at all or does it mean that znapzend doesn't support pruning recursive datasets with exceptions? In my case I want to replicate every dataset under /home without any exceptions so should that work?

The source server is currently running zfs-auto-snapshot for snapshotting. Would I be required to remove all existing snapshots on the pool before I configure znapzend or might it work OK still if I leave the existing snapshots in place and just uninstall z-a-s?

Thanks

[danboid]

If znapzend works in the way I think it does, then I think you should change that first sentence to something like:

"Note that whilst znapzend supports recursive replication of datasets, it does not support pruning of recursive datasets that contain exceptions ie when any child datasets are flagged to not be snapshotted."

Is that how it works or not?

[oetiker]

As far as I understand your use case, the znapzend will do exactly what you expect ... note that pruning ≠ thinning!

What some people want is, recursive backups but still be able to have some control over the thinning rules applied per dataset, or exclude a dataset completely ... both are not supported at the moment.

if you would like to suggest alternative language, please PR!

[oetiker]

that and a copy of mbuffer

[jimklimov]

I think mbuffer was optional? highly recommended though.

As for recursive snapshots, IIRC the implementation last I looked (and fiddled) was to make a recursive snapshot from the given tree root (dataset where you have the "retention policy" configured as its zfs properties), where zfs tool more or less atomically makes snapshots with same name and timestamp; then remove the unwanted snapshots from sub-tree roots where respective config properties say you don't want it tracked, and then replicates stuff offsite if configured so (with a recursive send or a series of singular sends).

Occasionally this breaks (e.g. some error and script aborts, or system reboots) between the recursive snapshot and scripted clean-up - leaving large-ish read-only snaps eating your disk space until you discover them and clean away manually. Think a swap dataset which eventually gets all replaced so uses up twice the space (live and snap data).

Znapzend can coexist with snapshots made by other tools (e.g. z-a-s, or OS upgrades done via ABEs, or manually, or backups of databases coupled with live freeze/thaw support, or some of my scripts for automation of rsync or git checkouts - any things also make snaps of stable data sets; mkdir .zfs/snapshots/snapname support helps do this even over NFS). Auto-cleanup of each such tool may be puzzled, especially if given a goal to keep some threshold of disk space available, and each assumes it is its own sandbox (hopefully coupled with naming patterns to constrain it). The Others' contemporary snapshots would inherit the block references and little space would get actually freed.

As a non-default (IIRC) mode the znapzend replication can be asked to send all snapshots, not just ones that match its pattern, so you get a complete history of the original dataset tree. It only cleans away its own, and nowadays has a way to track on source pool which snapshots were last replicated to this or that destination - so a prolonged loss of connection does not make those pools with auto-cleanup involved unreconcilable.

[danboid]

Thanks for sharing those useful tips Jim!

I'll hopefully get znapzend fully configured and working tonight. Currently, it is only creating snapshots in the source pool but it isn't replicating snapshots to the remote server/pool via ssh. systemctl status znapzend is giving lots of errors like:

znapzend[261187]:  +-->   ERROR: cannot send snapshots to

and when I run znapzend manually I see lots of permissions errors.

I was previously mirroring this pool to the remote pool using syncoid but sanoid failed to prune/trim/thin (or whatever the correct term is) snapshots so I ended up with hundreds of thousands of snapshots in both pools after a year or so. I deleted all of the snapshots on both pools and I'm starting over with my snapshots with znapzend.

I've checked all of the zfs permissions that znapzend requires but it turns out I'd already given all of the ones required except for userprop, which I have since added to the ZFS permissions for my zfsbackup user. It seems this is the only zfs permission that znapzend requires that syncoid unpriv does not.

As it stands it looks like I'm probably just going to destroy all of the datasets on the target pool as I imagine that will fix any permissions issues but if you know of anything else that could be preventing znapzend replicating snapshots to my remote pool then let me know. Passwordless ssh login is configured and working fine for my zfsbackup user.

I think I'm correct in saying znapzend only requires that you install it on the source machine right? The target machines just require a ssh server and the standard ZFS tools right?

[jimklimov]

Ugh, deleting literally all snapshots could be a bad idea. The nearest common denominator for znapzend to replicate from is zero, e.g. nuke (or rename away) the target dataset and send anew...

[jimklimov]

if you know of anything else that could be preventing znapzend replicating snapshots

You can run it with --debug, at least in single-shot (--run-once) CLI mode, there will be walls of text including zfs commands it prepares and executes.

Just in case: did you configure its rules for actual replication of the snapshots to a different destination? (in znapzendzetup or zfs get all <dataset> you would see them as dst_0 or some such)

I think I'm correct in saying znapzend only requires that you install it on the source machine right? The target machines just require a ssh server and the standard ZFS tools right?

yes