Adds upgrade testing boxes for 0.11.1

[conorsch]

Status

Ready for review.

Description of Changes

Checks the final box on, and therefore closes, #4060.

Standard procedure to update the base boxes used for the "upgrade" Molecule
scenario. Used the recently updated dev docs on maintenance of these
boxes to validate the documented procedure.

Changes proposed in this pull request:

• New base images for the "upgrade" scenario posted to S3
• New corresponding metadata for base images

It's worth pointing out that #4080 changes the storage paths for the debs, and as such, perhaps slight updates to the upgrade/ scenario logic will be required after merge of #4080. This PR is likely to get in first, so would prefer to follow up separately, to keep diffs small.

Testing

• Run rm -rf build/* && make build-debs to ensure you have clean debs (especially important given pending review of Stores Trusty & Xenial deb packages side by side #4080)
• make upgrade-start ; first run will take a while to fetch the images, then future runs will be snappy
• make upgrade-test-local ; confirm no errors
• Manually verify that the Source Interface shows 0.12.0~rc1

Deployment

No, dev env only.

Checklist

If you made changes to the server application code:

• Linting (make ci-lint) and tests (make -C securedrop test) pass in the development container

If you made changes to securedrop-admin:

• Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

• Configuration tests pass

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

If you made changes to documentation:

• Doc linting (make docs-lint) passed locally

[emkll]

Thanks @conorsch this looks good:

• Make sure you have local debs in the build/ dir, else run make build-debs
• make upgrade-start ; first run will take a while to fetch the images, then future runs will be snappy
• Go to source interface URL and observe SecureDrop version 0.11.1
• make upgrade-test-local ; confirm no errors
• Manually verify that the Source Interface shows 0.12.0~rc1

I am however getting a pip related error:

 "Preparing to unpack .../securedrop-ossec-agent_3.0.0+0.12.0~rc1_amd64.deb ...", "Unpacking securedrop-ossec-agent (3.0.0+0.12.0~rc1) over (3.0.0+0.11.1) ...", "Setting up securedrop-keyring (0.1.2+0.12.0~rc1) ...", "Setting up securedrop-config (0.1.2+0.12.0~rc1) ...", "ssh stop/waiting", "ssh start/running, process 3775", "Setting up securedrop-app-code (0.12.0~rc1+xenial) ...", "Ignoring indexes: https://pypi.python.org/simple/", "Requirement already up-to-date: alembic==0.9.9 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 7))", "Could not find any downloads that satisfy the requirement argon2-cffi==18.1.0 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))", "Downloading/unpacking argon2-cffi==18.1.0 (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))", "Cleaning up...", "No distributions at all found for argon2-cffi==18.1.0 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))", "Storing debug log for failure in /root/.pip/pip.log", "Setting up securedrop-ossec-agent (3.0.0+0.12.0~rc1) ...", "Killing ossec-logcollector .. ", "Killing ossec-syscheckd .. ", "Killing ossec-agentd .. ", "ossec-execd not running ..", "OSSEC HIDS v3.0.0 Stopped", "Starting OSSEC HIDS v3.0.0 (by Trend Micro Inc.)...", "Started ossec-execd...", "Started ossec-agentd...", "Started ossec-logcollector...", "Started ossec-syscheckd...", "Completed.", "Setting up securedrop-app-code (0.12.0~rc1+xenial) ...", "Ignoring indexes: https://pypi.python.org/simple/", "Requirement already up-to-date: alembic==0.9.9 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 7))", "Could not find any downloads that satisfy the requirement argon2-cffi==18.1.0 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))", "Downloading/unpacking argon2-cffi==18.1.0 (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))", "Cleaning up...", "No distributions at all found for argon2-cffi==18.1.0 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))", "Storing debug log for failure in /root/.pip/pip.log", "Reading package lists...", "Building dependency tree...", "Reading state information...", "Reading extended state information...", "Initializing package states..."]}

securedrop-app-code is not fully installed:

vagrant@app-staging:~$ sudo apt install securedrop-app-code
Reading package lists... Done
Building dependency tree       
Reading state information... Done
securedrop-app-code is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Setting up securedrop-app-code (0.12.0~rc1+xenial) ...
+ set -o pipefail
+ case "$1" in
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/lib/securedrop/
+ chmod 0700 /var/lib/securedrop/
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/lib/securedrop/tmp
+ chmod 0700 /var/lib/securedrop/tmp
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/lib/securedrop/store
+ chmod 0700 /var/lib/securedrop/store
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/lib/securedrop/keys
+ chmod 0700 /var/lib/securedrop/keys
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/lib/securedrop//keys/private-keys-v1.d
+ chmod 0700 /var/lib/securedrop//keys/private-keys-v1.d
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/lib/securedrop//keys/openpgp-revocs.d
+ chmod 0700 /var/lib/securedrop//keys/openpgp-revocs.d
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/lib/securedrop/backups
+ chmod 0700 /var/lib/securedrop/backups
+ for dir in '/var/lib/securedrop/{,tmp,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop
+ mkdir -p /var/www/securedrop
+ chmod 0700 /var/www/securedrop
+ chown -R www-data:www-data /var/lib/securedrop /var/www/securedrop
+ pip install --no-index --find-links=/var/securedrop/wheelhouse --upgrade -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt
Ignoring indexes: https://pypi.python.org/simple/
Requirement already up-to-date: alembic==0.9.9 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 7))
Could not find any downloads that satisfy the requirement argon2-cffi==18.1.0 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))
Downloading/unpacking argon2-cffi==18.1.0 (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))
Cleaning up...
No distributions at all found for argon2-cffi==18.1.0 in /usr/local/lib/python2.7/dist-packages (from -r /var/www/securedrop/requirements/securedrop-app-code-requirements.txt (line 8))
Storing debug log for failure in /home/vagrant/.pip/pip.log
dpkg: error processing package securedrop-app-code (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 securedrop-app-code
E: Sub-process /usr/bin/dpkg returned an error code (1)

It looks like it's there?

~/src/securedrop$ dpkg-deb --contents build/securedrop-app-code-0.12.0~rc1-amd64.deb | grep argon
-rw-r--r-- root/root     87843 2019-01-31 15:24 ./var/securedrop/wheelhouse/argon2_cffi-18.1.0-cp27-none-linux_x86_64.whl

I've built the debs twice and get the same error. Did you see this in your local testing? Given that this seems unrelated to this PR, it could maybe be worth merging as-is and opening a follow-up?

[conorsch]

Thanks for clear report, @emkll, will try to reproduce locally.

[conorsch]

will try to reproduce locally

I was able to reproduce, but only if I have xenial-specific debs left over in my build/ dir, related to testing/development on #4080. At a glance, the specific subdir logic shouldn't pollute the local debs testing flow, but evidently it did: the error output you shared includes Setting up securedrop-app-code (0.12.0~rc1+xenial), which won't be created from this branch.

Accordingly, I've updated the test plan to include rm -rf build/*, but before you do that, can you confirm that the inappropriate xenial-deb was present on your system within build/, not within build/<subdir>/? That'll help us evaluate whether we want to aggressively purge that dir when building (I'd prefer not to, but comments welcome).

To be clear, if I run rm -rf build/* && make build-debs before kicking off the upgrade scenario on this branch, then all steps, up to and including make upgrade-test-local, pass without error.

[kushaldas]

• Run rm -rf build/* && make build-debs to ensure you have clean debs (especially important given pending review of Stores Trusty & Xenial deb packages side by side #4080)
• make upgrade-start ; first run will take a while to fetch the images, then future runs will be snappy
• make upgrade-test-local ; confirm no errors
• Manually verify that the Source Interface shows 0.12.0~rc1

Approved. 👍

[emkll]

Must have been user error, these new boxes are working well for me locally, thanks @conorsch