Improve Backup/Restore/Migration docs, add v3 instructions #133

rocodes · 2021-01-13T17:06:31Z

Include backup and restore instructions for v3 onion services. Add hardware migration instructions for v3 onion services and for cases where onion credentials may not be restored.

Status

Work in progress

Description of Changes

Description: Update Restore/Migration docs to include steps for v3 services. Slight edits to condense existing Backup/Restore copy. WIP: would appreciate review.
Fixes Issue Add docs on v3 onion service restore-from-backup story #36

Testing

manual review

Release

none

Checklist (Optional)

Doc linting (make docs-lint) passed locally
Doc link linting (make docs-linkcheck) passed
You have previewed (make docs) docs at http://localhost:8000

docs/backup_and_restore.rst

eloquence · 2021-01-14T00:13:56Z

(Taking a first pass.)

docs/admin.rst

docs/backup_and_restore.rst

eloquence · 2021-01-14T02:23:14Z

Thanks @rocodes for filling in some very important detail for these instructions! Left some comments based on my initial review; the structural question may be easier to talk through synchronously esp. with @zenmonkeykstop to figure out what we want to address in the docs revision now, vs. defer to later.

rocodes · 2021-01-14T15:59:18Z

@eloquence these comments are great, thank you. It was a little challenging figuring out how to clarify the steps here. First round of revisions coming later today.

zenmonkeykstop

I think the migration instructions could be simpler. Either way I'd recommend restructuring them to treat the v2, v3, v2+3 scenarios separately.

docs/backup_and_restore.rst

zenmonkeykstop · 2021-01-20T23:11:15Z

docs/backup_and_restore.rst

+  - Restoring the backup to the new installation;
+  - Repairing Tor credentials for the new installation. 
+
+Detailed Migration Instructions


These instructions are a bit complex and branching - it might make sense to split them into 3 sections as follows:

v3-only - this should be simple:

copy tor_v3_keys.json to the fresh clone of securedrop,

then proceed with a fresh install

and then restore from the backup

v2+v3 - actually, this will be the same as v3-only, as who cares about the v2 ssh setup, it's not used. You can copy app's *ths files as below if you want the ssh app-legacy and ssh mon-legacy commands to work

v2-only - this one is probably the most complex - you will need to

copy app's *ths files from old install, do a fresh install,

restore from backup (which will fail with an error that says to run tailsconfig)

copy app's *ths files into place,

run tailsconfig,

then run restore again.

The trick here is that you don't have to rebuild mon's config for v2, you can just set up a new ssh config. Also for the configs with a v3 element, all you should need is the tor_v3_keys.json file.

(Also I may be missing steps here, it's been a while!)

Ooh also you'll need to copy keys from securedrop.old and you might as well copy site-specific while you're in there.

Or you could skip the new clone and just move files you don't want out of the way....

So, what's the scenario in which we want to support them preserving v2 as part of a migration/reinstall? Should we just make it explicit that they should migrate to v3 while they're there? These docs didn't use to exist at all, if we add them now, it feels like we might as well make them v3-only.

Hypothetically? Someone may have a downed v2 instance with an old backup available, and they may want to restore it before migrating. Or they have some other reason unique to them for going right up to the wire on v2. But adding a "Data-only migration to v3" option might not be a bad idea. something like...

If you want to restore the data from a backup on a fresh instance with new onion service addresses:

complete the fresh install first

copy the backup file FILENAME to install_files/ansible base

run ./securedrop-admin restore --preserve-tor-config FILENAME

...would do the trick. (this is v2/v3 only but if we disable new v2 installs next release that's a strong hint.)

So the "data-only migration to v3" section is in here, more opaquely referred to as "Restoring Data Without Restoring Tor Configuration", and is for the case of instances with a v2 backup file who want to recover data (ahem, if certain deadlines are missed).

I think there's an argument for skipping the v2-only hardware migration instructions, and telling users to contact Support. As soon as we stop allowing v2 onion services on fresh installs, the v2 migration instructions won't work. However, since they are written up I'm including them for now anyone says otherwise, and we can remove them in a few weeks' time.

@rocodes:

However, since they are written up I'm including them for now

Sounds reasonable; perhaps we can just add a blurb at the top of the section, e.g.:

If you are migrating or restoring an installation using v2 onion services, please be aware that support for them will be removed from SecureDrop starting in March 2021. We encourage you to migrate to v3 onion services as part of any migration. To do so, reinstall SecureDrop with v3 onion services enabled and restore your data without overwriting the generated Tor configuration [link to section]. The instructions for users of v2 onion services below will be removed in a future release.

Added in note on depreciation in 254b0d9, and also split out migration instructions by onion services version.

rocodes · 2021-01-22T00:17:59Z

I've addressed @zenmonkeykstop's minor edits (for typos etc) in 254b0d9, and also taken a stab at the restructuring/splitting into sections via onion service. His workflow is slightly different than the one I used (and I think his is better), but I haven't tested each path yet so I still consider this WIP.

eloquence · 2021-01-22T02:05:19Z

Thanks @rocodes. I've taken another visual spin through, and it looks great to me.

@rocodes and @zenmonkeykstop: What level of additional technical review would you recommend for these changes? My main concern is that stepping through a full migration, let alone all 3, will take significant time. Perhaps we should decouple this specific docs update from the 1.7.0 release timetable?

rocodes · 2021-01-22T12:57:30Z

I would like to step through all the migration paths on Monday before asking for anyone else's review, and I think that's doable. I think it's reasonable to decouple this from 1.7.0 and land it even a day or two after release day.

rocodes · 2021-01-26T02:56:05Z

@zenmonkeykstop: on the v3 scenario, I'm not having success with the workflow you suggested. Specifically, after the restore, I still need to reboot the app server and repair ssh creds, subbing in the original app-ssh.auth_private, similar to the v2 workflow. tor_v3_keys.json alone doesn't seem to do the trick for me. Let's connect tomorrow to make sure I'm not missing anything.

rocodes · 2021-01-26T16:43:30Z

Here are the v3 steps that worked for me @zenmonkeykstop, could use your eye before I write it up.

copy tor_v3_keys.json, pubkey files, and site-specific to the fresh clone of securedrop, and preserve app-ssh.auth_private file elsewhere (for now)
proceed with a fresh install, run tailsconfig, then run ssh app and ssh mon
copy in the backup file, restore from the backup, which will work but then time out
reboot the app server (or restart tor via console)
copy in app-ssh.auth_private file, rerun tailsconfig, then ssh app
run install, then run tailsconfig again.

… main

Add hardware migration instructions

…ersion (WIP)

zenmonkeykstop · 2021-02-23T22:05:23Z

docs/backup_and_restore.rst


 .. tip:: Although it varies, the average throughput of an onion service is
-         about 300 kB/s, or roughly 2 hours for 2GB. Plan your backup and
+         about 3 Mbps, or roughly 90 minutes for 2GB. Plan your backup and
         restore accordingly.

 You can use the following command to determine the volume of submissions
 currently on the *Application Server*: log in over SSH and run
 ``sudo du -sh /var/lib/securedrop/store``.


This could be made a one-liner from the Admin WS terminal, eg:
...
To check the disk space occupied by your submissions, open a Terminal on the Admin Workstation via Applications > System Tools > Terminal and run the command:

ssh app sudo du -sh /var/lib/securedrop/store

...
(Also might be good to put in its own code block for readability/copypastiness.)

fixed in 5c29677 (& all such code-blocks also)

zenmonkeykstop · 2021-02-23T23:02:41Z

docs/backup_and_restore.rst

-          approximate size of the backup file (since the majority of the backup
-          archive is the stored submissions), and Tails' **Disks** utility to
-          see how much free space you have on your persistent volume.
+Run ``git describe --exact-match`` to verify that you have a tagged SecureDrop


This step may no longer be necessary, as there's a change on develop securedrop to enforce a version check (thanks @eloquence ).

I would suggest conservatism until after April 30, given that out-of-date workstations will not have this change when it's needed, and so the check should still be performed manually during migrations. Agree that the 0.12.0 reference should go though.

zenmonkeykstop · 2021-02-23T23:09:49Z

docs/backup_and_restore.rst

-          archive is the stored submissions), and Tails' **Disks** utility to
-          see how much free space you have on your persistent volume.
+Run ``git describe --exact-match`` to verify that you have a tagged SecureDrop
+release, 0.12.0 or newer, checked out. 


And IMO this should just say "the latest version" regardless. No reason not to recommend to people to keep their sticks updated.

zenmonkeykstop · 2021-02-23T23:20:52Z

docs/backup_and_restore.rst

+The instructions differ depending on which onion services version 
+your instance is using.
+
+* :ref:`Instances using v3 onion services <migrate_v3>`


Unclear whether this refers to the instance that was backed up, or the target instance.

fixed in 5c29677, and addressed all other comments in same commit

zenmonkeykstop · 2021-02-23T23:21:55Z

docs/backup_and_restore.rst

+''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
+#. :ref:`Back up the existing installation <backing_up>`.
+
+#. **Preserve files and credentials**: Create a folder to


You could simplify this by just renaming the existing securedrop to securedrop.old without copying anything. Then you can reference it directly in the instructions below.

zenmonkeykstop · 2021-02-23T23:34:51Z

docs/backup_and_restore.rst

+
+#. **Reinstall SecureDrop on new hardware:** Re-clone the SecureDrop repository.
+
+   Copy the ``.asc`` files and the ``tor_v3_keys.json`` file into


To reduce the chance of user error, the commands should be explicit and preferably code-blocked.

zenmonkeykstop · 2021-02-23T23:37:03Z

docs/backup_and_restore.rst

+
+       ./securedrop-admin restore sd-backup-<your_backup_file>.tar.gz
+
+#. **Delete temporary files**: Remove any temporary files or directories


Could be an optional step as they may wanna keep securedrop.old around for safety's sake.

zenmonkeykstop · 2021-02-24T14:03:29Z

docs/backup_and_restore.rst

+   The ``app_ssh_url`` portion is your SSH service .onion address for the
+   *Application Server*.
+
+#. **Fix Tor Credentials:**


This doesn't affect tor credentials, but SSH host keys.

zenmonkeykstop · 2021-02-24T16:08:55Z

docs/backup_and_restore.rst

+   -  ``tor_v3_keys.json``
+   - All ``.asc`` files (these correspond to your *Submission Public Key*, your OSSEC alerts public key, and (if configured) your journalist alerts public key
+   - The file ``~/Persistent/securedrop/install_files/ansible-base/group-vars/all/site-specific``
+   - (optional) ``app-ssh-aths`` and ``mon-ssh-aths``, if you wish to reinstate the ``app-ssh-legacy`` and ``mon-ssh-legacy`` commands


mon isn't affected by the restore, so the monitor server after the restore will still have its address from the fresh install. This means there's some divergence between how to handle v2 and v3 in this case. You can just retain the fresh install mon-ssh-aths in the v2 case, but you would need to update the original tor_v3_keys.json, adding in the mon-ssh keypair from the fresh install's tor_v3_keys.json, before running either tailsconfig (which will update the auth_private file) or install (which will update the client auth files on the server).

zenmonkeykstop · 2021-02-24T16:14:20Z

docs/backup_and_restore.rst

+#. **Delete temporary files**: Remove any temporary files or directories
+   you created as you performed this migration, including the 
+   ``securedrop.old`` directory.
+


There's one other restore/migrate case to consider based on changes in freedomofpress/securedrop#5677 - namely the v2+v3 -> v3 one (previously didn't work but now unblocked)

See my comment- I have addressed this in a note in front of the v2+v3 instructions, indicating that migrating to Ubuntu 20.04 --> follow the v3-only instructions.

- write up v3 steps - pull shared instructions into `includes` file - rebase onto latest main

rocodes · 2021-02-25T14:39:33Z

@zenmonkeykstop: I think I've addressed all your comments. I also stepped through the v3 instructions again last night.

For v3 services, there is no mucking around with tor_v3_keys.json if you copy it into place before running the clean install, as detailed in these instructions.
I've separated the common instructions into an includes. Honestly, the only thing that differs about the v2 and v3 instructions is the "hey, if you want, you can also save the old app-ssh-aths file so your legacy commands work." That will be irrelevant real soon anyways, so we will be able to cut this document in half pretty soon.
To address the v2v3 xenial -> v3 focal scenario, I've made a note on the v2/v3 instructions that, when restoring onto Focal, the correct path is just to follow the v3 steps. Does that work for you? I've also written up Focal migration instructions (separate PR, tk later today) that indicate which migration documents to follow (basically, the "v3 docs" or the "preserve-tor-config" docs).

zenmonkeykstop · 2021-03-05T22:13:41Z

^^ one thing to note about patched-in changes below - the migration instructions include a tag verification/checkout step, which currently refers to 1.7.1. It should be 1.8.0, this is updated by update_version.sh which will be run prior to pushing to stable.

rocodes · 2021-03-08T19:06:02Z

Closing in favour of #168

rocodes requested review from eloquence and zenmonkeykstop January 13, 2021 17:06

rocodes changed the title ~~Clarify backup and restore preconditions~~ Improve Backup/Restore/Migration docs, add v3 instructions Jan 13, 2021

rocodes force-pushed the docs-v3-restore branch from 16c9515 to b7cfbf6 Compare January 13, 2021 17:09

rocodes commented Jan 13, 2021

View reviewed changes

docs/backup_and_restore.rst Show resolved Hide resolved

eloquence self-assigned this Jan 14, 2021