snapraid-btrfs-runner - cleanup guidance?

[fmoledina]

Hello,

I'm revisiting snapraid-btrfs as I'm adapting the snapraid-runner tool to work with it. My tool is aptly named snapraid-btrfs-runner. With this tool I implement a basic automated sync + scrub workflow using snapraid-btrfs.

I'd like to implement a cleanup yes/no option in the configuration that's exposed to users, and if the sync and scrub are successful, run a snapraid-btrfs cleanup. Is it as simple as that or are there any special considerations that I should account for depending on how users want to specify --cleanup algorithms or --use-snapshot-all? Thanks.

[automorphism88]

Hi,

As I understand your question, yes, it should indeed be as simple as just running snapraid-btrfs cleanup. That's what I do myself, and it's how the command is designed to be used.

Specifying a cleanup algorithm with the --cleanup option will only affect the snapper cleanup command, which is not run by snapraid-btrfs cleanup. Instead, snapraid-btrfs cleanup uses snapper rm to delete any snapshots created by snapraid-btrfs which are older than the one which a sync was last successfully completed with (or older than whatever snapshot you specify with --use-snapshot or --use-snapshot-all, but the default for the cleanup command is sync, and I don't think it's terribly useful to change that default, but it's allowed because there's not really a reason not to allow it). Any snapshots not created by snapraid-btrfs (i.e., without the snapraid-btrfs snapper userdata key defined) will always be ignored. (The cleanup-all command does the same thing, but using snapshot 0 rather than sync, so that it deletes all snapshots with the userdata key defined, and it does ignore the --use-snapshot and --use-snapshot-all options.) The --cleanup option really only exists for the sake of completeness, since I wanted to pass through all the options snapper offers when creating snapshots, and I don't personally use it myself.

The behavior above should be explained by the output of snapraid-btrfs --help, so if that was less than clear, I'd welcome any suggestions you might have for improving the documentation (though I'd understand if you just wanted to double check). I'd also welcome any suggestions for how the script's functionality might be improved to better work with your tool.

A few caveats:

• If you're using snapraid-btrfs pool, make sure to run it in between the sync and the cleanup. Otherwise you'll end up with broken symlinks that point to deleted snapshots.
• Use caution when automating snapraid-btrfs, and keep an eye on it to make sure it's doing what you expect. I wrote snapraid-btrfs primarily for my own use, and test it on my own system. It should be considered experimental software, and as stated in the license, there is no warranty. Bug reports are, of course, welcome.
• The above describes the current behavior of the cleanup command. Given the experimental nature of snapraid-btrfs, it's recommended to use the latest version. The most recent version, released less than 24 hours ago, comes with improvements for both the scrub and cleanup commands. I'd been thinking it would be nice to add a changelog to better document those improvements, but hearing about your tool makes that seem like a higher priority. In the meantime, you can look at how the output of snapraid-btrfs --help has changed, since that's kept up to date with any changes in the script's behavior.

Thanks for opening this issue. It's interesting to hear how you're making use of snapraid-btrfs. I'm going to convert this issue into a discussion, but please feel free to follow up if you have any other questions.

[fmoledina]

Thank you for the prompt and extremely detailed response. I'll concede that the output of snapraid-btrfs --help is very detailed, and as I take time to get more familiar with Snapper, I better understand how snapraid-btrfs exposes the cleanup options of the underlying tool. My goal with snapraid-btrfs-runner is to develop a tool with the minimal viable set of options to for automated daily runs. If a user identifies a one-time requirement to do something differently, they can use snapraid-btrfs directly to acheive that outcome.

Within that context, I'm debating which snapraid-btrfs options to expose in the configuration. At the moment, I'm brainstorming use cases for automated use of the --use-snapshot-all option in the config, and if it is available, it would apply to all of the automated operations in the script, including diff, sync, scrub, and the soon-to-be-added pool and cleanup commands.

If you're using snapraid-btrfs pool, make sure to run it in between the sync and the cleanup. Otherwise you'll end up with broken symlinks that point to deleted snapshots.

This is a good point. I don't use the pooling features of snapraid or snapraid-btrfs, opting instead to user MergerFS. However, this should be a configurable option and run as you suggest between the sync and cleanup.

Instead, snapraid-btrfs cleanup uses snapper rm to delete any snapshots created by snapraid-btrfs which are older than the one which a sync was last successfully completed with (or older than whatever snapshot you specify with --use-snapshot or --use-snapshot-all, but the default for the cleanup command is sync, and I don't think it's terribly useful to change that default, but it's allowed because there's not really a reason not to allow it)

From the above, one could run snapraid-btrfs --use-snapshot-all <option> cleanup and it would cleanup based on that selection? Is it possible for someone to run a dangerous version of that command using the new option? I'm consdering allowing any valid argument for --use-snapshot-all aside from menu and new. If someone would like to use --use-snapshot-all new, it's likely that they expect to use it only for diff and sync, and not necessarily for the remaining commands. As new is the default behavior for diff and sync, the user would be achieving their goal without inadvertently impacting scrub, pool, and cleanup. In this case, snapraid-btrfs-runner would error out with a note to that effect.

I also now think that I should expose --snapper-configs and --snapper-configs-file underlying options in the config as well, in case the default config behavior is insufficient.

Thanks again for the detailed response and I look forward to working with you for continued development. I too am building snapraid-btrfs-runner for my own needs, as the upstream snapraid-runner is the best automated snapraid tool that I've come across. I had waded into this effort early last year (in the now archived https://github.com/fmoledina/snapraid-btrfs-runner-py2.7 project), but it was short-lived as I switched away from using Snapraid altogether for a while. I'm glad to be back on it. The combination of Snapraid and BTRFS is very powerful and for me personally, I like that using snapraid-btrfs allows me to continue writing to the pool while the daily automated sync and scrub run takes place.

[automorphism88]

Thank you for the prompt and extremely detailed response. I'll concede that the output of snapraid-btrfs --help is very detailed, and as I take time to get more familiar with Snapper, I better understand how snapraid-btrfs exposes the cleanup options of the underlying tool. My goal with snapraid-btrfs-runner is to develop a tool with the minimal viable set of options to for automated daily runs. If a user identifies a one-time requirement to do something differently, they can use snapraid-btrfs directly to acheive that outcome.

I see. Yeah, snapraid-btrfs uses snapper for everything related to snapshot management, so it's a good idea to be familiar with how snapper works. But the snapper cleanup algorithms aren't really designed for the snapraid-btrfs use case, and perhaps I shouldn't have added that option at all. Snapper is a SUSE project, and it's primarily designed with their btrfs root filesystem setup in mind, where the package manager creates pre and post snapshots with the "number" cleanup algorithm, and you can use GRUB to boot into a readonly snapshot if your system breaks.

Within that context, I'm debating which snapraid-btrfs options to expose in the configuration. At the moment, I'm brainstorming use cases for automated use of the --use-snapshot-all option in the config, and if it is available, it would apply to all of the automated operations in the script, including diff, sync, scrub, and the soon-to-be-added pool and cleanup commands.

What you're describing with using the same --use-snapshot-all option for all commands doesn't really make sense. There's a reason the commands have different defaults. When you sync, you want to create a new snapshot, so that you're updating the parity with whatever changes have been made since the last sync. But when you run, e.g., scrub, you want to use the snapshot that matches your parity. The default snapshot choices are supposed to be reasonable, and have been made more so over time. For instance, the "scrub" target was something I recently added, and made the default for commands like scrub that used to use "sync" as their default snapshot.

The cases that come to mind where you'd want to use --use-snapshot or --use-snapshot-all are more related to dealing with exceptional circumstances than running routine syncs or scrubs, and even then, they're less needed than they once were, as the defaults have gotten better. For instance, if you replace a drive, and recover the files, then want to run a scrub or check, now that the "scrub" target exists, it'll find and use a post-fix snapshot, so it's only needed if, say, you were able to copy the files off of a failing drive instead of recovering it with a fix, and the new drive doesn't have any snapshots on it. In the past, that kind of situation would cause snapraid-btrfs to exit with an error, and you'd have to run it again with --use-snapshot foo=0 to use the live filesystem, but now it will actually give an interactive prompt if it can't find a matching snapshot, and ask you to choose which snapshot to use (after displaying the output of snapper ls for that config, with the option to choose 0 for the live filesystem or to create a new snapshot), unless you specify the -q/--quiet option, in which case it will still exit with an error (which is intended to allow snapraid-btrfs to be run automatically in a context where an interactive prompt won't work, but I could see creating a separate option to suppress that interactive prompt without the other effects of the --quiet option, if it would be used by applications like yours).

This is a good point. I don't use the pooling features of snapraid or snapraid-btrfs, opting instead to user MergerFS. However, this should be a configurable option and run as you suggest between the sync and cleanup.

I actually use both. MergerFS is read/write and points to the live filesystem, while the pool directory has symlinks to readonly snapshots.

From the above, one could run snapraid-btrfs --use-snapshot-all <option> cleanup and it would cleanup based on that selection? Is it possible for someone to run a dangerous version of that command using the new option? I'm consdering allowing any valid argument for --use-snapshot-all aside from menu and new. If someone would like to use --use-snapshot-all new, it's likely that they expect to use it only for diff and sync, and not necessarily for the remaining commands. As new is the default behavior for diff and sync, the user would be achieving their goal without inadvertently impacting scrub, pool, and cleanup. In this case, snapraid-btrfs-runner would error out with a note to that effect.

Yeah, I suppose running snapraid-btrfs --use-snapshot-all=new cleanup wouldn't be a great idea, but it's no worse than running snapraid-btrfs cleanup-all, which is equivalent to snapraid-btrfs --use-snapshot-all=0 cleanup. It would be easy to force --use-snapshot-all=sync with cleanup the same way cleanup-all forces --use-snapshot-all=0, but I don't really see a reason to. But from the standpoint of doing routine maintenance, there shouldn't be a need to ever use anything other than the default "sync" target with cleanup. The default of "sync" is conservative (it could potentially free up more space by using "res" or "scrub"), and if there's no synced snapshot for a config, it will just skip that config (and print a warning to stderr).

I also now think that I should expose --snapper-configs and --snapper-configs-file underlying options in the config as well, in case the default config behavior is insufficient.

I don't see any real reason not to expose these options to users, but they also shouldn't be necessary. They were just added as a backup, in case a new version of snapper breaks the way snapraid-btrfs looks for snapshots again. It used to run snapraid list-configs and compare its output to snapraid.conf, but then snapper was changed so that only root is allowed to run snapper list-configs. So I changed it to look in /etc/snapper/configs instead (which I had previously considered, but thought running snapper list-configs was more robust), and added those options just in case. It just looks at the filenames in that directory, so you don't need read permission for the files themselves (since it runs snapper -c foo get-config to find the SUBVOLUME and see if it matches snapraid.conf), though you do need read permission for the /etc/snapper/configs directory itself, and to have permission for the relevant snapper configs in ALLOW_USERS or ALLOW_GROUPS (any configs you don't have permission for will be ignored, so you don't need permissions for all snapper configs on the system, and if you use the --verbose option it will tell you they're being skipped).

Thanks again for the detailed response and I look forward to working with you for continued development. I too am building snapraid-btrfs-runner for my own needs, as the upstream snapraid-runner is the best automated snapraid tool that I've come across. I had waded into this effort early last year (in the now archived https://github.com/fmoledina/snapraid-btrfs-runner-py2.7 project), but it was short-lived as I switched away from using Snapraid altogether for a while. I'm glad to be back on it. The combination of Snapraid and BTRFS is very powerful and for me personally, I like that using snapraid-btrfs allows me to continue writing to the pool while the daily automated sync and scrub run takes place.

Yeah, that's exactly why I wrote snapraid-btrfs! I just did it for my own use, but it's cool to see other people using it too.

[fmoledina]

What you're describing with using the same --use-snapshot-all option for all commands doesn't really make sense. There's a reason the commands have different defaults. When you sync, you want to create a new snapshot, so that you're updating the parity with whatever changes have been made since the last sync. But when you run, e.g., scrub, you want to use the snapshot that matches your parity. The default snapshot choices are supposed to be reasonable, and have been made more so over time. For instance, the "scrub" target was something I recently added, and made the default for commands like scrub that used to use "sync" as their default snapshot.

I completely agree with this, and the defaults are sane enough for my use case. After having this discussion, I think snapraid-btrfs-runner shouldn't expose --use-snapshot-all at all, and if the user needs it for exceptional circumstances, they can use snapraid-btrfs directly.

I don't see any real reason not to expose these options to users, but they also shouldn't be necessary. They were just added as a backup, in case a new version of snapper breaks the way snapraid-btrfs looks for snapshots again.

Understood, and again on my system, the sane default is perfectly adequate for my use case. I'll focus on implementing snapraid-btrfs pool and snapraid-btrfs cleanup first, and then follow up on the additional snapper config items.

Thank you again for the discussion!

[fmoledina]

Hello @automorphism88, I've now implemented the pool and cleanup functions in between sync and scrub. I've also included the --snapper-configs and --snapper-configs-file options. Thanks for your input!