Document how to run staging env in Qubes

[conorsch]

Status

Ready for review.

Description of Changes

Fixes freedomofpress/securedrop-workstation#105.

Implements a staging environment for use in Qubes, so that devs working on the workstation code do not need to run VMs on a separate laptop in order to contact an SD server for development. The work is the result of collaboration with @joshuathayer. Submitting the PR as a "docs-only" PR because it doesn't change any files used for managing production SecureDrop instances:

$ git diff develop --name-status
A       docs/development/qubes_staging.rst
M       docs/index.rst
M       docs/servers.rst
A       molecule/qubes-staging/create.yml
A       molecule/qubes-staging/destroy.yml
A       molecule/qubes-staging/molecule.yml
A       molecule/qubes-staging/playbook.yml
A       molecule/qubes-staging/qubes-vars.yml
A       molecule/qubes-staging/ssh_config.j2

There are a few unimplemented features of the staging environment:

Config tests are not wired up

We already have the full config test suite running in CI on every (non-docs) PR. That's sufficient for preventing regressions. Happy to add config tests to the staging environment to the work presented here, but it should wait for the Vagrant -> Molecule conversion (#3208), in which can ideally axe the testinfra/test.py wrapper script.

Machines need a nudge to boot after a reboot has triggered

When Ansible requests the VM reboot, it powers off, but Qubes won't restart it. You must run qvm-start sd-app sd-mon in order to boot the VMs again, and you must run that command within the "wait_for" window, which is five minutes. Technically it's possible for us to detect that we're in Qubes and make the qvm-start calls dynamically—the Qubes Admin API makes this possible—but doing so would affect Ansible logic that runs against prod instances, so I'd prefer to follow up in a discrete PR targeting that improvement at a later date, if warranted.

Halting is not a first-class feature

Molecule assumes we want to destroy a VM after testing it. By default, there's no corresponding action in Molecule to vagrant halt <vm>. In the future, we can certainly implement one, with a bit of additional scripting, but I'd prefer to rely on developer feedback before attempting a convenient solution. Developers working in Qubes will need to run qvm-shutdown sd-app sd-mon in order to preserve the VMs for later use. The Molecule logic is only required the first time (or if server-side code changes are required); thereafter, the qvm- actions will be sufficient to get the Source and Journalist Interfaces up and running.

No automatic integration of HidServAuth tokens

After provisioning the staging environment, the sd-dev VM will have the HidServAuth tokens in the usual place: install_files/ansible-base/*-aths. These will need to be copied manually as config.json for use with development on the SecureDrop Workstation. Given that the VM provisioning and token porting tasks are each one-time, I consider that an acceptable trade-off, especially because we've reduced the number of required laptops from two to one with these changes.

Testing

Qubes machine required for testing.

• Follow the documentation to create a "staging" VM setup in Qubes.
• Run molecule test -s qubes-staging.
• When the machines go down for a reboot, run qvm-start sd-app sd-mon in dom0.
• Confirm playbook finishes without error
• Access the Source Interface over Tor and confirm it loads.
• Run qvm-prefs sys-whonix and confirm it fails with "Service call error: request refused".

If anything is unclear in the docs, get loud. If anything doesn't work with the staging provisioning process, please post error messaging to aid in debugging.

Worthy of special attention are the Qubes RPC grants, which enable VM management from within sd-dev, rather than from dom0. Using the Qubes Admin API makes for a much slicker developer workflow, but we should scrutinize any grants we provide, here or hereafter. To the best of my ability, I've restricted the permissions grant management rights of only the staging instances. For further reading:

• https://www.qubes-os.org/news/2017/06/27/qubes-admin-api/
• https://www.qubes-os.org/doc/admin-api/

Deployment

None, dev env only.

Checklist

If you made changes to the server application code:

• Linting (make ci-lint) and tests (make -C securedrop test) pass in the development container

If you made changes to securedrop-admin:

• Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

• Configuration tests pass

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

If you made changes to documentation:

• Doc linting (make docs-lint) passed locally

[conorsch]

On a few recent runs of this setup, I noticed a provisioning error related to a missing AppArmor profile for the tor abstractions. None of the diff here indicates that's caused by these changes, so I'm inviting review immediately, to see if the problem occurs for anyone else.

[codecov-io]

Codecov Report

Merging #3624 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #3624   +/-   ##
========================================
  Coverage    85.04%   85.04%           
========================================
  Files           37       37           
  Lines         2367     2367           
  Branches       260      260           
========================================
  Hits          2013     2013           
  Misses         290      290           
  Partials        64       64

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 997db39...0d02344. Read the comment docs.

[emkll]

I went through these (very clear) docs and successfully built a SecureDrop staging environment in Qubes. The Qubes API policy works as intended.
Because the playbook updates all packages and given the incremental effort in using template-based VMs, I think standalone VMs are the right choice in this situation. I've encountered two errors on playbook runs during the course of my testing, both occuring when I use the molecule scenarios:

• An issue with reboot_if_first_install (more comments inline, but does not impact the run, just fails at the end)
• An error at the OSSEC step where the public key is copied over to the mon server as part of the Add the OSSEC GPG public key to the OSSEC manager keyring task:

Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc:1, err: chown: changing ownership of '/tmp/ansible-tmp-15325777.69-4139702993444/': Operation not permitted

Workaround (or fix) is to install the acl package on the mon server and running the converge scenario once again.

[emkll]

Perhaps we refer to the (now merged) Qubes dev env docs introduced in #3623 to avoid duplication

[conorsch]

refer to the (now merged) Qubes dev env docs

Absolutely, will rebase and cross-link.

[emkll]

While it is the default option, it may be useful to specify on which disk to install the operating system, perhaps by stating configure LVM and use Virtual disk 1 (xvda 20.0GB Xen Virtual Block device)

[conorsch]

Can do; @joshuathayer had originally explicitly documented the target volume, and the declaration got lost in one of the follow-up concision edits.

[emkll]

There also appears to be an issue at the end of the install playbook run where it errors out at the end with the following error:

Unable to retrieve file contents from /home/user/securedrop/molecule/qubes-staging/tasks/reboot_if_first_install.yml

This happens at the very end, and the install is still completely functional.

[conorsch]

Thanks, @msheiny wrote a different reboot handler for the AWS scenario. Will take a closer look and see if we can modify the original to work everywhere, otherwise I'll implement a scenario-specific handler.

[emkll]

Is the prepare step necessary? The molecule create does call the prepare action after creating the VMs, and warns that there is no prepare playbook found (molecule/qubes-staging/prepare.yml). Running molecule prepare explicitly returns a Skipping, instances already prepared message.

[conorsch]

Good catch, @emkll, the prepare step is indeed not necessary. I had the prepare action in there as a stopgap to do some of the config that we later moved to the base VM setup docs. Will revise!

[conorsch]

Workaround (or fix) is to install the acl package on the mon server and running the converge scenario once again.

Great catch on this one. We don't need that package in prod or other VMs, so I'm loathe to install via the prod config logic simply to satisfy the Qubes VMs. After a bit of digging, that error relates to privilege escalation via Ansible, and one of the recommended resolutions is:

Use pipelining. When pipelining is enabled, Ansible doesn’t save the module to a temporary file on the client. Instead it pipes the module to the remote python interpreter’s stdin. Pipelining does not work for python modules involving file transfer (for example: copy, fetch, template), or for non-python modules.

That's precisely what we do with our Ansible config, in the ansible.cfg file, but notably the Qubes scenario wasn't including that config file, so the pipelining option was off, triggering the error. I'll add a reference to that same config file to the scenario, so that we're testing with the same connection for Ansible that we use elsewhere. That should obviate the need to install acl, as well (although that's also a recommended resolution in the docs).

[conorsch]

@emkll Nearly all comments addressed, please re-review. The sole comment unaddressed is:

An issue with reboot_if_first_install (more comments inline, but does not impact the run, just fails at the end)

Took at look at this, and turns out we encountered this problem in the Molecule scenario for AWS/CI, as well, in which we added a var to detect whether we're on AWS, then overrode the reboot logic with a custom implementation specific to that scenario. I'd prefer not to do the same for the Qubes staging setup.

The next best bet is to convert the included task to a role, preserving conditional logic. This will resolve the failure to find the file that you reported. I've tested this locally and it works well-ish, since the additional reboot requires developer intervention in the form of another qvm-start sd-app sd-mon, but as mentioned above, that's an acceptable shortcoming for now. However, that logic change will affect the playbooks, meaning it's inappropriate for a docs- PR, as I've submitted.

Willing to PR into this PR, if that is a reasonable workflow for you during review. Otherwise, I suggest merging as-is (with the documentation fixes already in place), and I can follow-up with a separate, tiny PR that will include a staging CI run to validate the tweak to logic doesn't break the non-Qubes config. I slightly prefer the latter option, since that would allow me to clean up the duplicated reboot logic in the molecule/aws/ scenario, as well, but that's a stretch goal, so I'm willing to forgo it to keep the wheels turning.

For clarity, see proposed patch on the task to role conversion described above in this gist: https://gist.github.com/conorsch/49830560f8fce0835ab4b477e8387f56

[emkll]

Thanks for the quick fixes @conorsch ! I confirm that changing the ANSIBLE_CONFIG env variable fixes the issue with the key copy.
As for the reboot error, I agree, we shouldn't block merge on this, given the scope of this PR ( and the scenario actually works). I have opened a ticket to track the outstanding issue (#3629), as to better scope QA to those changes (which will likely require careful qa for both the qubes but also other scenarios).