Add backup and restore instructions #81

rocodes · 2021-05-21T00:15:31Z

Status WIP/Ready for initial review
Description Document backup and restore process
Resolves #46
Related freedomofpress/securedrop-workstation#120

The approach taken in this PR is to suggest backing up the vault vm and the configuration files from dom0, and nothing else. (See comms in linked issues above).

I hope to improve on this approach eventually, because it's still a little unwieldy. Open questions include:

Re-downloading and verifying the rpm and then shuttling it into dom0 seems like a real pain, but relying on a (potentially very old) rpm + the updater also doesn't seem great. Currently, to be conservative, I recommend users re-download/verify/shuttle the RPM if it's not the current version. Open to feedback.
Currently, if users have configured custom RPC policies (such as tags that grant extended clipboard access), they have to manually re-add those after restoring the backup. Rationale: overwriting /etc/qubes-rpc in dom0 is risky, shouldn't be undertaken lightly, and has potential for breakage or bad consequences (for example, if VM names change and a hardcoded rpc policy is in place, or if a VM is granted access in an RPC policy but isn't restored in the backup, then if a future VM is created with that name, it could have unexpected permissions grandfathered in...yknow, stuff to be cautious/paranoid about). Again, tried to be conservative here, open to feedback.

I think it will make sense to have a much more streamlined backup/restore, but this is a start. (Future ideas: maybe a "bundle backup configs" button in the GUI, maybe an easier way to grab/verify the RPM, and maybe a different consensus on whether or not we backup and restore more of the system, such as templates, so as to avoid long wall times/manual intervention during restore?)

conorsch

Very nice! Left extremely verbose comments throughout. Basically my feedback boils down to

let's ignore most of what's in dom0
the fresh install pre-restore should be as pristine as possible, including latest version

Note that I haven't actually run through these instructions to verify the steps functionally. After you've had a chance to review the comments, please ping me again for a deeper review!

conorsch · 2021-06-03T23:33:15Z

docs/admin/backup.rst

+To perform backups, you will need:
+
+ - an encrypted storage medium
+ - a secure place to store backup credentials    


Yes to early checklists! Maybe more detail is warranted here. Instead of "an encrypted storage medium", how about "a LUKS-encrypted external hard drive, with at least 50GB free space"? In a few places, the docs mention that maybe veracrypt was used, but because the backup use case is explicitly Qubes to Qubes, it's OK to strongly recommend use of LUKS IMO.

Similiarly, consider

a secure place to store backup credentials (such as a password manager on your primary laptop)

conorsch · 2021-06-03T23:47:49Z

docs/admin/backup.rst

+  .. code-block:: sh
+
+    mkdir ~/backup/
+    cp -r /usr/share/securedrop-workstation-dom0-config/ ~/backup/


🚨 We only want two (2) files from the config dir:

/usr/share/securedrop-workstation-dom0-config/config.json

/usr/share/securedrop-workstation-dom0-config/sd-journalist.sec

The whole dir looks like this (taken from my machine):

[user@dom0 ~]$ tree /usr/share/securedrop-workstation-dom0-config/ /usr/share/securedrop-workstation-dom0-config/ ├── config.json ├── config.json.example ├── scripts │ ├── clean-salt │ ├── configure-environment │ ├── destroy-vm │ ├── lint-all │ ├── provision-all │ ├── sdw-admin.py │ └── validate_config.py └── sd-journalist.sec 1 directory, 10 files

All those files under scripts/ are provisioned as part of the RPM. During the restore docs below, the order of operations (as currently written in this draft) is:

install prod rpm (thereby laying out the most recent scripts/ files)

restore from back up, cp the files back, recursively (thereby clobbering the newest scripts/ files with an older version from the backup)

Ideally we'd be able to cp /* within that dir and trust everything we get. The fact that scripts are in the "share" directory at all is definitely nonstandard, so that deserves its own issue in the SDW repo to adjust the packaging and provisioning logic accordingly. But, for the sake of this PR, explictly copying just those two files strikes me as the right approach.

Thanks for catching this - addressed in c462517

conorsch · 2021-06-03T23:53:44Z

docs/admin/backup.rst

+
+If you have customized :doc:`clipboard access <managing_clipboard>` 
+to SecureDrop Workstation, or have made any other customizations 
+involving RPC policies, back up related configuration files:   


Clipboard whitelisting happens via tagging, not editing the RPC policies: https://workstation.securedrop.org/en/stable/admin/managing_clipboard.html

Sure, it's possible an admin customized the RPC policies, but I'm not convinced we want to encourage or even support that. If an admin is knowledgeable enough about Qubes even to attempt customizing RPC policies, I think we can trust them to continue to manage independently.

Also, the backup-and-restore procedures as written do not handle tags, so after the reinstall, an admin will need to rerun the relevant tags command. Even if they restore the files inside /etc/qubes-rpc/policy/, the clipboard settings will not be restored! That's OK, IMO, but let's not promise otherwise.

In my experience, tags are restored when a qube is restored. (I remembered saying this a while back, but I was doubting myself, so I just tried tagging and restoring a test qube and indeed the custom tags were still present).

Backing up the RPC policy directories (with the idea that admins could audit them if need be, but not restore them directly) was some sort-of-futureproofing in case we are dealing with customized export workflows that involve customized qubes. But I can remove it and we can cross that bridge if/when we come to it.

I've removed all of this in c462517 and left a note that advanced users may need to back up additional dom0 components. I think for now, that's a reasonable tradeoff information-wise. If we have orgs using a lot of customized qubes and/or modifying RPC policies to accommodate their own workflows down the line, we can address this in more depth (or maybe we will develop a system that avoids the need for them doing this).

conorsch · 2021-06-03T23:54:58Z

docs/admin/backup.rst

+   mkdir -p ~/backup/etc/qubes
+   mkdir ~/backup/etc/qubes-rpc
+   cp -a /etc/qubes/* ~/backup/etc/qubes/
+   cp -a /etc/qubes-rpc/* ~/backup/etc/qubes-rpc


Closely related to the above: I don't think we should include the any of the /etc/qubes/ files at all. The install scripts will configure all the SDW requirements fresh, which is good, and the remainder are managed by Qubes upstream. This step can be omitted entirely. (To other interested reviewers, the recommendation is present in https://www.qubes-os.org/doc/backup-restore/, but I still don't think it's appropriate for SDW.)

removed in c462517

conorsch · 2021-06-03T23:57:53Z

docs/admin/backup.rst

+
+ - ``dom0``
+ - ``vault``
+ - any customized VMs that you wish to preserve.


Backing up both dom0 and vault feels somewhat redundant to me. The motivation to back up vault is presumably for the pw manager (https://workstation.securedrop.org/en/stable/admin/install.html#copy-securedrop-login-credentials)—makes sense. For dom0, we really only need the config.json and sd-journalist.sec (see above). Does it make sense to copy those two files into vault prior to the backup?

Anecdotally, I've stopped including dom0 in my (personal) backups, since restoring it often had surprising consequences. Hardly actionable info, I know, but if we can minimize what we must restore, I think we'll eliminate the potential for problems down the road. Open to suggestions on approach here.

I agree with you that I don't personally restore dom0, but here are the things that I wonder about w.r.t whether or not to back up dom0:

ease of restore workflow for a user that has configured copy-paste between VMs or other custom RPC policies, possibly with our assistance (very common eg with passwords and the vault vm), and doesn't know how to get back t the state they were in pre- backup

if just doing a simple hardware migration for some reason, and a user already has the latest RPM in dom0, it seems rude to make them go through the whole 'download, verify, shuttle to dom0 again' procedure.

I'm in favour of rethinking where those files are stored. I think we can potentially expect that they will live on in perpetuity wherever they are copied to, so we have to take that into account, esp with sd-journalist.sec.

user that has configured copy-paste between VMs or other custom RPC policies, possibly with our assistance (very common eg with passwords and the vault vm)

Backing up dom0 won't help in that case. In order to support retaining clipboard access settings, we'd have to either 1) backup the sd-* VMs (which would include tag info for each VM); or 2) write a script to dump the tags from the SDW VMs, and then encourage the admin to re-apply the tags after restoring. Neither seem worthwhile to my eye, although at least mentioning the clipboard access tags in the backup/restore docs certainly seems reasonable.

if just doing a simple hardware migration for some reason, and a user already has the latest RPM in dom0

That strikes me as a lot of conditionals. In order to reduce headaches for admins, I think our efforts would be best spent in trying to avoid the shuttle-to-dom0 workflow entirely, e.g. via --enablerepo=qubes-templates-community. That'll take a lot of work in coordination with Qubes upstream, so definitely a long-term goal. I defer to your judgment on what's most helpful to admins at present!

Resolved in c462517, where I no longer suggest backing up dom0.

conorsch · 2021-06-04T00:02:56Z

docs/admin/backup.rst

+
+.. note::
+ If you backed up a number of custom VMs, during the installer, you may opt
+ to un-select "create default application qubes (personal, work, untrusted, vault)".


To reduce the number of "ifs", maybe we can just say explicitly here don't install those default VMs. We don't care about personal, work, or untrusted—and vault will be coming from the backup.

oh, I remember why I had the "if" and this workflow here -- if they clean install and then need to re-download the RPM, without the default application qubes, they don't actually have a network-connected VM with which to grab the RPM, so our instructions are technically incomplete.

(fixed in c462517)

conorsch · 2021-06-04T00:03:48Z

docs/admin/backup.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Ensure your storage medium is plugged in, attached to a VM, and mounted.
+(LUKS-encrypted storage media may be attached to ``sd-usb``, or you may use a custom 


s/sd-usb/sd-devices/

conorsch · 2021-06-04T00:05:53Z

docs/admin/backup.rst

+  .. code-block:: sh
+
+   cd $DOM0_BACKUP/backup
+   cp -r securedrop-workstation-dom0-config/ /usr/share/


🚨 See note in backup section about the potentially harmful recursive copy here—we only want two (2) files. It's most important to specify them explicitly by fullpath in the backup section, but I'd still do the same in the restore section, to guard against surprises.

thanks for this flag-- fixed in c462517

conorsch · 2021-06-04T00:09:00Z

docs/admin/backup.rst

+ version of the rpm, in which case you can skip the above steps and move
+ straight to installing the rpm. 
+ Use ``rpm -qi securedrop-workstation.rpm`` to compare the version with the
+ latest version at ``https://yum.securedrop.org/workstation/dom0/f25/``    


In the interest of reducing ifs—which I find cause me to stop reading and think hard about whether the condition applies to my use case—let's just recommend installing from scratch again. It feels like an edge case for the back up and restore to happen on the same version of securedrop-workstation-dom0-config. Even in the case of a minor version bump, we want to make sure the new system uses the latest and greatest.

docs/admin/backup.rst

rocodes · 2021-06-08T22:37:22Z

@conorsch, thanks for your review! I've addressed your comments and now the backup/restore workflow involves:

backup: copy sd-journalist.sec and config.json to the vault vm; back up vault vm
restore: reinstall qubes, restore vault, re-create work VM, re-download RPM, copy in and verify config files from vault vm, run apply.

conorsch · 2021-06-10T23:15:54Z

docs/admin/backup.rst

+
+  .. code-block:: sh
+
+    sudo cp tmp{sd-journalist.sec,config.json} /usr/share/securedrop-workstation-dom0-config/


Missing leading slash on tmp, should be /tmp/.

conorsch · 2021-06-10T23:17:11Z

Thanks for the copious edits, @rocodes! Looks great to my eye. Before merge, I'd like to see @creviera take a pass through it so it's got her formal approval. No further concerns from me.

sssoleileraaa · 2021-06-22T18:49:09Z

on my radar: I can review today or tomorrow

sssoleileraaa

lgtm, thanks ro!

sssoleileraaa

looks great! 🚀

rocodes requested review from conorsch and sssoleileraaa May 21, 2021 00:15

eloquence assigned conorsch Jun 3, 2021

conorsch reviewed Jun 4, 2021

View reviewed changes

rocodes force-pushed the 46-backup-restore branch from 05b8dfd to c462517 Compare June 8, 2021 22:34

rocodes requested a review from conorsch June 9, 2021 16:45

conorsch reviewed Jun 10, 2021

View reviewed changes

rocodes force-pushed the 46-backup-restore branch from c462517 to 83e9c31 Compare June 22, 2021 19:12

sssoleileraaa mentioned this pull request Jun 23, 2021

Install diceware in the SDW templates for admin use freedomofpress/securedrop-workstation#712

Closed

sssoleileraaa previously approved these changes Jun 23, 2021

View reviewed changes

rocodes dismissed sssoleileraaa’s stale review via f813de4 June 24, 2021 20:15

rocodes force-pushed the 46-backup-restore branch from 83e9c31 to f813de4 Compare June 24, 2021 20:15

rocodes added 2 commits June 28, 2021 19:16

Add preliminary backup and restore instructions

bf537f3

Avoid backing up dom0; incorporate review feedback.

3a320ec

rocodes force-pushed the 46-backup-restore branch from 030a1e0 to 3a320ec Compare June 28, 2021 23:16

sssoleileraaa approved these changes Jun 28, 2021

View reviewed changes

sssoleileraaa merged commit 311a0c9 into main Jun 28, 2021

sssoleileraaa deleted the 46-backup-restore branch June 28, 2021 23:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add backup and restore instructions #81

Add backup and restore instructions #81

rocodes commented May 21, 2021

conorsch left a comment

conorsch Jun 3, 2021

conorsch Jun 4, 2021

conorsch Jun 3, 2021

rocodes Jun 8, 2021 •

edited

Loading

conorsch Jun 3, 2021

rocodes Jun 7, 2021

rocodes Jun 8, 2021 •

edited

Loading

conorsch Jun 3, 2021

rocodes Jun 8, 2021 •

edited

Loading

conorsch Jun 3, 2021

rocodes Jun 4, 2021 •

edited

Loading

conorsch Jun 4, 2021

rocodes Jun 8, 2021 •

edited

Loading

conorsch Jun 4, 2021

rocodes Jun 8, 2021

rocodes Jun 8, 2021

conorsch Jun 4, 2021

conorsch Jun 4, 2021

rocodes Jun 8, 2021 •

edited

Loading

conorsch Jun 4, 2021

rocodes commented Jun 8, 2021

conorsch Jun 10, 2021

conorsch commented Jun 10, 2021

sssoleileraaa commented Jun 22, 2021

sssoleileraaa left a comment

sssoleileraaa left a comment


		.. code-block:: sh

		sudo cp tmp{sd-journalist.sec,config.json} /usr/share/securedrop-workstation-dom0-config/

Add backup and restore instructions #81

Add backup and restore instructions #81

Conversation

rocodes commented May 21, 2021

conorsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rocodes Jun 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rocodes Jun 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rocodes Jun 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rocodes Jun 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rocodes Jun 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rocodes Jun 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rocodes commented Jun 8, 2021

Choose a reason for hiding this comment

conorsch commented Jun 10, 2021

sssoleileraaa commented Jun 22, 2021

sssoleileraaa left a comment

Choose a reason for hiding this comment

sssoleileraaa left a comment

Choose a reason for hiding this comment

rocodes Jun 8, 2021 •

edited

Loading

rocodes Jun 8, 2021 •

edited

Loading

rocodes Jun 8, 2021 •

edited

Loading

rocodes Jun 4, 2021 •

edited

Loading

rocodes Jun 8, 2021 •

edited

Loading

rocodes Jun 8, 2021 •

edited

Loading