Upgrade boxes for 1.8.0 (Xenial)

[conorsch]

Status

Ready for review

Description of Changes

Closes #5794

Creates upgrade boxes for 1.8.0, Xenial-only. Refs #5794. We should create Focal boxes too, but that's tracked in #5512 .

Testing

• Check out this branch
• make build-debs
• make upgrade-start (will take a while as it needs to fetch the boxes)
• source interface shows SecureDrop version 1.8.0
• make upgrade-test-local completes without error
• source interface shows SecureDrop version 1.9.0~rc1 (requires other tasks on release epic to be resolved)

Deployment

None, dev-only.

[emkll]

rebased on lastest develop after merging https://github.com/freedomofpress/securedrop/pull/5866/files

[zenmonkeykstop]

On Debian-10, make upgrade-start hangs for me when booting upgrade-app-staging. It looks like the boot process hangs when switching display dtivers to cirrus. Manually stopping the VM, setting the driver to qxl, and restarting the VM gets it working but the rest of the upgrade scenario doesn't run.

[zenmonkeykstop]

Steps required to get these boxes working on Debian 10:

• installed latest vagrant (2.2.14) via .deb package (not Debian repo version)
• installed libvirt-dev in addition to regular livbvirt reqs
• ran CONFIGURE_ARGS="with-libvirt-include=/usr/include/libvirt with-libvirt-lib=/usr/lib" vagrant plugin install vagrant-libvirt (extra args needed, otherwise build fails)
• installed mutate plugin, mutated bento/ubuntu-16.04 box, proceeded with make build-debs && make upgrade-start
• after a couple of false starts, make upgrade-start proceeds, fails with 'rep_origin' is undefined message. (Looks like a new variable was introduced for apt repo origin (probably because unattended-upgrades needs it?)
• added rep_origin = SecureDrop to upgrade scenario overrides, make upgrade-start succeeds! 1.8.0 confirmed!
• make upgrade-test-local fails with a disk i/o error - ran it again, success! 1.9.0~rc1 confirmed.

[zenmonkeykstop]

Ran through upgrade scenario once more - this time submitting via the SI before running make upgrade-test-local to give theDB migration something to chew on. Failed with:

    TASK [Perform safe upgrade] ****************************************************
[WARNING]: The value True (type bool) in a string field was converted to 'True'
(type string). If this does not look like what you expect, quote the entire
value to ensure it does not change.
    changed: [mon-staging]
fatal: [app-staging]: FAILED! => {"changed": false, "msg": "'/usr/bin/apt-get upgrade --with-new-pkgs ' failed: E: Sub-process /usr/bin/dpkg returned an error code (1)\n", "rc": 100, "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nCalculating upgrade...\nThe following packages will be upgraded:\n  linux-libc-dev securedrop-app-code securedrop-config securedrop-keyring\n  securedrop-ossec-agent\n5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 11.4 MB of archives.\nAfter this operation, 104 kB of additional disk space will be used.\nGet:1 https://apt.freedom.press xenial/main amd64 securedrop-config all 0.1.4+1.9.0~rc1+xenial [2744 B]\nGet:2 https://apt.freedom.press xenial/main amd64 securedrop-keyring amd64 0.1.4+1.9.0~rc1+xenial [5842 B]\nGet:3 https://apt.freedom.press xenial/main amd64 securedrop-app-code amd64 1.9.0~rc1+xenial [10.6 MB]\nGet:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-libc-dev amd64 4.4.0-206.238 [836 kB]\nGet:5 https://apt.freedom.press xenial/main amd64 securedrop-ossec-agent amd64 3.6.0+1.9.0~rc1+xenial [4606 B]\nPreconfiguring packages ...\nFetched 11.4 MB in 0s (15.7 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 49082 files and directories currently installed.)\r\nPreparing to unpack .../linux-libc-dev_4.4.0-206.238_amd64.deb ...\r\nUnpacking linux-libc-dev:amd64 (4.4.0-206.238) over (4.4.0-204.236) ...\r\nPreparing to unpack .../securedrop-config_0.1.4+1.9.0~rc1+xenial_all.deb ...\r\nUnpacking securedrop-config (0.1.4+1.9.0~rc1+xenial) over (0.1.4+1.8.0+xenial) ...\r\nPreparing to unpack .../securedrop-keyring_0.1.4+1.9.0~rc1+xenial_amd64.deb ...\r\nUnpacking securedrop-keyring (0.1.4+1.9.0~rc1+xenial) over (0.1.4+1.8.0+xenial) ...\r\nPreparing to unpack .../securedrop-app-code_1.9.0~rc1+xenial_amd64.deb ...\r\nUnpacking securedrop-app-code (1.9.0~rc1+xenial) over (1.8.0+xenial) ...\r\nPreparing to unpack .../securedrop-ossec-agent_3.6.0+1.9.0~rc1+xenial_amd64.deb ...\r\nUnpacking securedrop-ossec-agent (3.6.0+1.9.0~rc1+xenial) over (3.6.0+1.8.0+xenial) ...\r\nSetting up linux-libc-dev:amd64 (4.4.0-206.238) ...\r\nSetting up securedrop-config (0.1.4+1.9.0~rc1+xenial) ...\r\n+ manage_tor_repo_config\r\n+ rm -f /etc/apt/sources.list.d/deb_torproject_org_torproject_org.list\r\n+ rm -f /etc/apt/sources.list.d/tor_apt_freedom_press.list\r\n+ apt_security_list=/etc/apt/security.list\r\n+ [ -f /etc/apt/security.list ]\r\n+ sed -i /deb\\.torproject\\.org\\/torproject\\.org/d /etc/apt/security.list\r\n+ sed -i /tor-apt\\.freedom\\.press/d /etc/apt/security.list\r\n+ remove_2fa_tty_req\r\n+ auth_file=/etc/pam.d/common-auth\r\n+ sed -i /^auth\\ required\\ pam_google.*/d /etc/pam.d/common-auth\r\n+ grep -qF PasswordAuthentication no /etc/ssh/sshd_config\r\n+ sed -i /^UsePAM\\ /s/\\ .*/\\ no/ /etc/ssh/sshd_config\r\n+ sed -i /^ChallengeResponseAuthentication\\ /s/\\ .*/\\ no/ /etc/ssh/sshd_config\r\n+ service ssh restart\r\n+ update_release_prompt\r\n+ set -e\r\n+ upgrade_config=/etc/update-manager/release-upgrades\r\n+ sed -i s/Prompt=.*/Prompt=never/ /etc/update-manager/release-upgrades\r\n+ revert_update_release_available_script\r\n+ [ -f /usr/lib/ubuntu-release-upgrader/check-new-release ]\r\n+ sed -i s|Visit https://securedrop\\.org/xenial-upgrade for more information|Run 'do-release-upgrade' to upgrade to it.| /usr/lib/ubuntu-release-upgrader/check-new-release\r\n+ [ -f /var/lib/ubuntu-release-upgrader/release-upgrade-available ]\r\n+ sed -i s|Visit https://securedrop\\.org/xenial-upgrade for more information|Run 'do-release-upgrade' to upgrade to it.| /var/lib/ubuntu-release-upgrader/release-upgrade-available\r\n+ [ -f /etc/cron-apt/action.d/1-remove ]\r\n+ allow_apt_user_in_iptables\r\n+ rules_v4=/etc/network/iptables/rules_v4\r\n+ [ -f /etc/network/iptables/rules_v4 ]\r\n+ perl -npi -e s/^.*--uid-owner root.*apt updates.*$/-A OUTPUT -p tcp --match multiport --dports 80,8080,443 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT -m comment --comment \"apt updates\"/ /etc/network/iptables/rules_v4\r\n+ perl -npi -e s/^(.*--dport 53) -m owner --uid-owner root(.*)$/$1$2/ /etc/network/iptables/rules_v4\r\n+ exit 0\r\nSetting up securedrop-keyring (0.1.4+1.9.0~rc1+xenial) ...\r\nSetting up securedrop-app-code (1.9.0~rc1+xenial) ...\r\nInstalling new version of config file /var/www/securedrop/static/i/logo.png ...\r\n+ set -o pipefail\r\n+ SDVE=/opt/venvs/securedrop-app-code\r\n+ SDBIN=/opt/venvs/securedrop-app-code/bin\r\n+ case \"$1\" in\r\n+ set_paxctld_config\r\n+ paxctld_config=/etc/paxctld.conf\r\n+ '[' -f /etc/paxctld.conf ']'\r\n+ grep -q '^/usr/sbin/apache2' /etc/paxctld.conf\r\n+ systemctl enable paxctld\r\nSynchronizing state of paxctld.service with SysV init with /lib/systemd/systemd-sysv-install...\r\nExecuting /lib/systemd/systemd-sysv-install enable paxctld\r\n+ systemctl start paxctld\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop/\r\n+ chmod 0700 /var/lib/securedrop/\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop/tmp\r\n+ chmod 0700 /var/lib/securedrop/tmp\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop/shredder\r\n+ chmod 0700 /var/lib/securedrop/shredder\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop/store\r\n+ chmod 0700 /var/lib/securedrop/store\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop/keys\r\n+ chmod 0700 /var/lib/securedrop/keys\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop//keys/private-keys-v1.d\r\n+ chmod 0700 /var/lib/securedrop//keys/private-keys-v1.d\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop//keys/openpgp-revocs.d\r\n+ chmod 0700 /var/lib/securedrop//keys/openpgp-revocs.d\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/lib/securedrop/backups\r\n+ chmod 0700 /var/lib/securedrop/backups\r\n+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop\r\n+ mkdir -p /var/www/securedrop\r\n+ chmod 0700 /var/www/securedrop\r\n+ '[' -e /var/lib/securedrop/keys/gpg-agent.conf ']'\r\n+ grep -qE '^allow-loopback-pinentry$' /var/lib/securedrop/keys/gpg-agent.conf\r\n+ '[' '!' -d /var/lib/securedrop/keys/private-keys-v1.d ']'\r\n+ chown -R www-data:www-data /var/lib/securedrop /var/www/securedrop\r\n+ chown -R www-data:www-data /var/www/securedrop\r\n+ chown www-data:www-data /var/www/journalist.wsgi\r\n+ chown www-data:www-data /var/www/source.wsgi\r\n+ a2dissite 000-default\r\nSite 000-default already disabled\r\n+ a2dissite default-ssl\r\nSite default-ssl already disabled\r\n+ service apache2 stop\r\n+ /opt/venvs/securedrop-app-code/bin/mod_wsgi-express module-config\r\n+ a2enmod wsgi\r\nModule wsgi already enabled\r\n+ '[' -e /etc/apparmor.d/disable/usr.sbin.apache2 ']'\r\n+ aa-enforce /etc/apparmor.d/usr.sbin.tor\r\nSetting /etc/apparmor.d/usr.sbin.tor to enforce mode.\r\n+ aa-enforce /etc/apparmor.d/usr.sbin.apache2\r\nSetting /etc/apparmor.d/usr.sbin.apache2 to enforce mode.\r\n+ adjust_wsgi_configuration\r\n+ journalist_conf=/etc/apache2/sites-available/journalist.conf\r\n+ test -f /etc/apache2/sites-available/journalist.conf\r\n+ grep -qP '^WSGIPassAuthorization' /etc/apache2/sites-available/journalist.conf\r\n+ grep -qP '^WSGIScriptAlias / /var/www/journalist.wsgi$' /etc/apache2/sites-available/journalist.conf\r\n+ grep -qP '^WSGIProcessGroup journalist' /etc/apache2/sites-available/journalist.conf\r\n+ remove_bytecode\r\n+ find /opt/venvs/securedrop-app-code -name '*.py[co]' -delete\r\n+ service apache2 restart\r\n+ rm -fr /var/www/securedrop/static/gen/source.js\r\n+ '[' -n 1.8.0+xenial ']'\r\n+ '[' 1.8.0+xenial = 0.3 ']'\r\n+ chmod u+w /var/www/securedrop/static/i/logo.png\r\n+ database_migration\r\n+ database_dir=/var/lib/securedrop\r\n+ database_file=/var/lib/securedrop/db.sqlite\r\n+ '[' '!' -e /var/lib/securedrop/db.sqlite ']'\r\n+ cd /var/www/securedrop/\r\n++ /opt/venvs/securedrop-app-code/bin/alembic current\r\n/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.\r\n  from cryptography.exceptions import AlreadyFinalized\r\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\r\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\r\n+ '[' -z '' ']'\r\n++ sqlite3 /var/lib/securedrop/db.sqlite .tables\r\n+ sqlite_tables='alembic_version           replies                   seen_replies            \r\ninstance_config           revoked_tokens            source_stars            \r\njournalist_login_attempt  seen_files                sources                 \r\njournalists               seen_messages             submissions             '\r\n+ echo 'alembic_version           replies                   seen_replies            \r\ninstance_config           revoked_tokens            source_stars            \r\njournalist_login_attempt  seen_files                sources                 \r\n+ grep -q journalists\r\njournalists               seen_messages             submissions             '\r\n+ /opt/venvs/securedrop-app-code/bin/alembic stamp 15ac9509fc68\r\n/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.\r\n  from cryptography.exceptions import AlreadyFinalized\r\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\r\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\r\nINFO  [alembic.runtime.migration] Running stamp_revision  -> 15ac9509fc68\r\n++ date +%Y-%m-%d-%H-%M-%S\r\n+ db_backup=/var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite\r\n+ grep -q '(head)'\r\n+ /opt/venvs/securedrop-app-code/bin/alembic current\r\n/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.\r\n  from cryptography.exceptions import AlreadyFinalized\r\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\r\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\r\n+ cp /var/lib/securedrop/db.sqlite /var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite\r\n+ /opt/venvs/securedrop-app-code/bin/alembic upgrade head\r\n/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.\r\n  from cryptography.exceptions import AlreadyFinalized\r\nINFO  [alembic.runtime.migration] Context impl SQLiteImpl.\r\nINFO  [alembic.runtime.migration] Will assume non-transactional DDL.\r\nINFO  [alembic.runtime.migration] Running upgrade 15ac9509fc68 -> faac8092c123, enable security pragmas\r\nINFO  [alembic.runtime.migration] Running upgrade faac8092c123 -> 3d91d6948753, Create source UUID column\r\nTraceback (most recent call last):\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1244, in _execute_context\r\n    cursor, statement, parameters, context\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/default.py\", line 552, in do_execute\r\n    cursor.execute(statement, parameters)\r\nsqlite3.OperationalError: duplicate column name: uuid\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File \"/opt/venvs/securedrop-app-code/bin/alembic\", line 11, in <module>\r\n    load_entry_point('alembic==0.9.9', 'console_scripts', 'alembic')()\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/config.py\", line 486, in main\r\n    CommandLine(prog=prog).main(argv=argv)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/config.py\", line 480, in main\r\n    self.run_cmd(cfg, options)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/config.py\", line 463, in run_cmd\r\n    **dict((k, getattr(options, k, None)) for k in kwarg)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/command.py\", line 254, in upgrade\r\n    script.run_env()\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/script/base.py\", line 427, in run_env\r\n    util.load_python_file(self.dir, 'env.py')\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/util/pyfiles.py\", line 81, in load_python_file\r\n    module = load_module_py(module_id, path)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/util/compat.py\", line 83, in load_module_py\r\n    spec.loader.exec_module(module)\r\n  File \"<frozen importlib._bootstrap_external>\", line 665, in exec_module\r\n  File \"<frozen importlib._bootstrap>\", line 222, in _call_with_frames_removed\r\n  File \"alembic/env.py\", line 82, in <module>\r\n    run_migrations_online()\r\n  File \"alembic/env.py\", line 76, in run_migrations_online\r\n    context.run_migrations()\r\n  File \"<string>\", line 8, in run_migrations\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/runtime/environment.py\", line 836, in run_migrations\r\n    self.get_context().run_migrations(**kw)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/runtime/migration.py\", line 330, in run_migrations\r\n    step.migration_fn(**kw)\r\n  File \"/var/www/securedrop/alembic/versions/3d91d6948753_create_source_uuid_column.py\", line 27, in upgrade\r\n    op.add_column(\"sources_tmp\", sa.Column(\"uuid\", sa.String(length=36)))\r\n  File \"<string>\", line 8, in add_column\r\n  File \"<string>\", line 3, in add_column\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/operations/ops.py\", line 1565, in add_column\r\n    return operations.invoke(op)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/operations/base.py\", line 319, in invoke\r\n    return fn(self, operation)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/operations/toimpl.py\", line 123, in add_column\r\n    schema=schema\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/ddl/impl.py\", line 172, in add_column\r\n    self._exec(base.AddColumn(table_name, column, schema=schema))\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/ddl/impl.py\", line 118, in _exec\r\n    return conn.execute(construct, *multiparams, **params)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 988, in execute\r\n    return meth(self, multiparams, params)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/sql/ddl.py\", line 72, in _execute_on_connection\r\n    return connection._execute_ddl(self, multiparams, params)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1050, in _execute_ddl\r\n    compiled,\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1248, in _execute_context\r\n    e, statement, parameters, cursor, context\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1466, in _handle_dbapi_exception\r\n    util.raise_from_cause(sqlalchemy_exception, exc_info)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/util/compat.py\", line 383, in raise_from_cause\r\n    reraise(type(exception), exception, tb=exc_tb, cause=cause)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/util/compat.py\", line 128, in reraise\r\n    raise value.with_traceback(tb)\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1244, in _execute_context\r\n    cursor, statement, parameters, context\r\n  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/default.py\", line 552, in do_execute\r\n    cursor.execute(statement, parameters)\r\nsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: uuid\r\n[SQL: ALTER TABLE sources_tmp ADD COLUMN uuid VARCHAR(36)]\r\n(Background on this error at: http://sqlalche.me/e/e3q8)\r\n+ echo 'Error migrating database: Backup saved to /var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite'\r\nError migrating database: Backup saved to /var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite\r\n+ return 1\r\ndpkg: error processing package securedrop-app-code (--configure):\r\n subprocess installed post-installation script returned error exit status 1\r\nSetting up securedrop-ossec-agent (3.6.0+1.9.0~rc1+xenial) ...\r\nErrors were encountered while processing:\r\n securedrop-app-code\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Calculating upgrade...", "The following packages will be upgraded:", "  linux-libc-dev securedrop-app-code securedrop-config securedrop-keyring", "  securedrop-ossec-agent", "5 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.", "Need to get 11.4 MB of archives.", "After this operation, 104 kB of additional disk space will be used.", "Get:1 https://apt.freedom.press xenial/main amd64 securedrop-config all 0.1.4+1.9.0~rc1+xenial [2744 B]", "Get:2 https://apt.freedom.press xenial/main amd64 securedrop-keyring amd64 0.1.4+1.9.0~rc1+xenial [5842 B]", "Get:3 https://apt.freedom.press xenial/main amd64 securedrop-app-code amd64 1.9.0~rc1+xenial [10.6 MB]", "Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 linux-libc-dev amd64 4.4.0-206.238 [836 kB]", "Get:5 https://apt.freedom.press xenial/main amd64 securedrop-ossec-agent amd64 3.6.0+1.9.0~rc1+xenial [4606 B]", "Preconfiguring packages ...", "Fetched 11.4 MB in 0s (15.7 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 49082 files and directories currently installed.)", "Preparing to unpack .../linux-libc-dev_4.4.0-206.238_amd64.deb ...", "Unpacking linux-libc-dev:amd64 (4.4.0-206.238) over (4.4.0-204.236) ...", "Preparing to unpack .../securedrop-config_0.1.4+1.9.0~rc1+xenial_all.deb ...", "Unpacking securedrop-config (0.1.4+1.9.0~rc1+xenial) over (0.1.4+1.8.0+xenial) ...", "Preparing to unpack .../securedrop-keyring_0.1.4+1.9.0~rc1+xenial_amd64.deb ...", "Unpacking securedrop-keyring (0.1.4+1.9.0~rc1+xenial) over (0.1.4+1.8.0+xenial) ...", "Preparing to unpack .../securedrop-app-code_1.9.0~rc1+xenial_amd64.deb ...", "Unpacking securedrop-app-code (1.9.0~rc1+xenial) over (1.8.0+xenial) ...", "Preparing to unpack .../securedrop-ossec-agent_3.6.0+1.9.0~rc1+xenial_amd64.deb ...", "Unpacking securedrop-ossec-agent (3.6.0+1.9.0~rc1+xenial) over (3.6.0+1.8.0+xenial) ...", "Setting up linux-libc-dev:amd64 (4.4.0-206.238) ...", "Setting up securedrop-config (0.1.4+1.9.0~rc1+xenial) ...", "+ manage_tor_repo_config", "+ rm -f /etc/apt/sources.list.d/deb_torproject_org_torproject_org.list", "+ rm -f /etc/apt/sources.list.d/tor_apt_freedom_press.list", "+ apt_security_list=/etc/apt/security.list", "+ [ -f /etc/apt/security.list ]", "+ sed -i /deb\\.torproject\\.org\\/torproject\\.org/d /etc/apt/security.list", "+ sed -i /tor-apt\\.freedom\\.press/d /etc/apt/security.list", "+ remove_2fa_tty_req", "+ auth_file=/etc/pam.d/common-auth", "+ sed -i /^auth\\ required\\ pam_google.*/d /etc/pam.d/common-auth", "+ grep -qF PasswordAuthentication no /etc/ssh/sshd_config", "+ sed -i /^UsePAM\\ /s/\\ .*/\\ no/ /etc/ssh/sshd_config", "+ sed -i /^ChallengeResponseAuthentication\\ /s/\\ .*/\\ no/ /etc/ssh/sshd_config", "+ service ssh restart", "+ update_release_prompt", "+ set -e", "+ upgrade_config=/etc/update-manager/release-upgrades", "+ sed -i s/Prompt=.*/Prompt=never/ /etc/update-manager/release-upgrades", "+ revert_update_release_available_script", "+ [ -f /usr/lib/ubuntu-release-upgrader/check-new-release ]", "+ sed -i s|Visit https://securedrop\\.org/xenial-upgrade for more information|Run 'do-release-upgrade' to upgrade to it.| /usr/lib/ubuntu-release-upgrader/check-new-release", "+ [ -f /var/lib/ubuntu-release-upgrader/release-upgrade-available ]", "+ sed -i s|Visit https://securedrop\\.org/xenial-upgrade for more information|Run 'do-release-upgrade' to upgrade to it.| /var/lib/ubuntu-release-upgrader/release-upgrade-available", "+ [ -f /etc/cron-apt/action.d/1-remove ]", "+ allow_apt_user_in_iptables", "+ rules_v4=/etc/network/iptables/rules_v4", "+ [ -f /etc/network/iptables/rules_v4 ]", "+ perl -npi -e s/^.*--uid-owner root.*apt updates.*$/-A OUTPUT -p tcp --match multiport --dports 80,8080,443 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT -m comment --comment \"apt updates\"/ /etc/network/iptables/rules_v4", "+ perl -npi -e s/^(.*--dport 53) -m owner --uid-owner root(.*)$/$1$2/ /etc/network/iptables/rules_v4", "+ exit 0", "Setting up securedrop-keyring (0.1.4+1.9.0~rc1+xenial) ...", "Setting up securedrop-app-code (1.9.0~rc1+xenial) ...", "Installing new version of config file /var/www/securedrop/static/i/logo.png ...", "+ set -o pipefail", "+ SDVE=/opt/venvs/securedrop-app-code", "+ SDBIN=/opt/venvs/securedrop-app-code/bin", "+ case \"$1\" in", "+ set_paxctld_config", "+ paxctld_config=/etc/paxctld.conf", "+ '[' -f /etc/paxctld.conf ']'", "+ grep -q '^/usr/sbin/apache2' /etc/paxctld.conf", "+ systemctl enable paxctld", "Synchronizing state of paxctld.service with SysV init with /lib/systemd/systemd-sysv-install...", "Executing /lib/systemd/systemd-sysv-install enable paxctld", "+ systemctl start paxctld", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop/", "+ chmod 0700 /var/lib/securedrop/", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop/tmp", "+ chmod 0700 /var/lib/securedrop/tmp", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop/shredder", "+ chmod 0700 /var/lib/securedrop/shredder", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop/store", "+ chmod 0700 /var/lib/securedrop/store", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop/keys", "+ chmod 0700 /var/lib/securedrop/keys", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop//keys/private-keys-v1.d", "+ chmod 0700 /var/lib/securedrop//keys/private-keys-v1.d", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop//keys/openpgp-revocs.d", "+ chmod 0700 /var/lib/securedrop//keys/openpgp-revocs.d", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/lib/securedrop/backups", "+ chmod 0700 /var/lib/securedrop/backups", "+ for dir in '/var/lib/securedrop/{,tmp,shredder,store,keys,/keys/private-keys-v1.d,/keys/openpgp-revocs.d,backups}' /var/www/securedrop", "+ mkdir -p /var/www/securedrop", "+ chmod 0700 /var/www/securedrop", "+ '[' -e /var/lib/securedrop/keys/gpg-agent.conf ']'", "+ grep -qE '^allow-loopback-pinentry$' /var/lib/securedrop/keys/gpg-agent.conf", "+ '[' '!' -d /var/lib/securedrop/keys/private-keys-v1.d ']'", "+ chown -R www-data:www-data /var/lib/securedrop /var/www/securedrop", "+ chown -R www-data:www-data /var/www/securedrop", "+ chown www-data:www-data /var/www/journalist.wsgi", "+ chown www-data:www-data /var/www/source.wsgi", "+ a2dissite 000-default", "Site 000-default already disabled", "+ a2dissite default-ssl", "Site default-ssl already disabled", "+ service apache2 stop", "+ /opt/venvs/securedrop-app-code/bin/mod_wsgi-express module-config", "+ a2enmod wsgi", "Module wsgi already enabled", "+ '[' -e /etc/apparmor.d/disable/usr.sbin.apache2 ']'", "+ aa-enforce /etc/apparmor.d/usr.sbin.tor", "Setting /etc/apparmor.d/usr.sbin.tor to enforce mode.", "+ aa-enforce /etc/apparmor.d/usr.sbin.apache2", "Setting /etc/apparmor.d/usr.sbin.apache2 to enforce mode.", "+ adjust_wsgi_configuration", "+ journalist_conf=/etc/apache2/sites-available/journalist.conf", "+ test -f /etc/apache2/sites-available/journalist.conf", "+ grep -qP '^WSGIPassAuthorization' /etc/apache2/sites-available/journalist.conf", "+ grep -qP '^WSGIScriptAlias / /var/www/journalist.wsgi$' /etc/apache2/sites-available/journalist.conf", "+ grep -qP '^WSGIProcessGroup journalist' /etc/apache2/sites-available/journalist.conf", "+ remove_bytecode", "+ find /opt/venvs/securedrop-app-code -name '*.py[co]' -delete", "+ service apache2 restart", "+ rm -fr /var/www/securedrop/static/gen/source.js", "+ '[' -n 1.8.0+xenial ']'", "+ '[' 1.8.0+xenial = 0.3 ']'", "+ chmod u+w /var/www/securedrop/static/i/logo.png", "+ database_migration", "+ database_dir=/var/lib/securedrop", "+ database_file=/var/lib/securedrop/db.sqlite", "+ '[' '!' -e /var/lib/securedrop/db.sqlite ']'", "+ cd /var/www/securedrop/", "++ /opt/venvs/securedrop-app-code/bin/alembic current", "/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.", "  from cryptography.exceptions import AlreadyFinalized", "INFO  [alembic.runtime.migration] Context impl SQLiteImpl.", "INFO  [alembic.runtime.migration] Will assume non-transactional DDL.", "+ '[' -z '' ']'", "++ sqlite3 /var/lib/securedrop/db.sqlite .tables", "+ sqlite_tables='alembic_version           replies                   seen_replies            ", "instance_config           revoked_tokens            source_stars            ", "journalist_login_attempt  seen_files                sources                 ", "journalists               seen_messages             submissions             '", "+ echo 'alembic_version           replies                   seen_replies            ", "instance_config           revoked_tokens            source_stars            ", "journalist_login_attempt  seen_files                sources                 ", "+ grep -q journalists", "journalists               seen_messages             submissions             '", "+ /opt/venvs/securedrop-app-code/bin/alembic stamp 15ac9509fc68", "/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.", "  from cryptography.exceptions import AlreadyFinalized", "INFO  [alembic.runtime.migration] Context impl SQLiteImpl.", "INFO  [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO  [alembic.runtime.migration] Running stamp_revision  -> 15ac9509fc68", "++ date +%Y-%m-%d-%H-%M-%S", "+ db_backup=/var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite", "+ grep -q '(head)'", "+ /opt/venvs/securedrop-app-code/bin/alembic current", "/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.", "  from cryptography.exceptions import AlreadyFinalized", "INFO  [alembic.runtime.migration] Context impl SQLiteImpl.", "INFO  [alembic.runtime.migration] Will assume non-transactional DDL.", "+ cp /var/lib/securedrop/db.sqlite /var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite", "+ /opt/venvs/securedrop-app-code/bin/alembic upgrade head", "/var/www/securedrop/secure_tempfile.py:10: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.", "  from cryptography.exceptions import AlreadyFinalized", "INFO  [alembic.runtime.migration] Context impl SQLiteImpl.", "INFO  [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO  [alembic.runtime.migration] Running upgrade 15ac9509fc68 -> faac8092c123, enable security pragmas", "INFO  [alembic.runtime.migration] Running upgrade faac8092c123 -> 3d91d6948753, Create source UUID column", "Traceback (most recent call last):", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1244, in _execute_context", "    cursor, statement, parameters, context", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/default.py\", line 552, in do_execute", "    cursor.execute(statement, parameters)", "sqlite3.OperationalError: duplicate column name: uuid", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"/opt/venvs/securedrop-app-code/bin/alembic\", line 11, in <module>", "    load_entry_point('alembic==0.9.9', 'console_scripts', 'alembic')()", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/config.py\", line 486, in main", "    CommandLine(prog=prog).main(argv=argv)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/config.py\", line 480, in main", "    self.run_cmd(cfg, options)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/config.py\", line 463, in run_cmd", "    **dict((k, getattr(options, k, None)) for k in kwarg)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/command.py\", line 254, in upgrade", "    script.run_env()", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/script/base.py\", line 427, in run_env", "    util.load_python_file(self.dir, 'env.py')", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/util/pyfiles.py\", line 81, in load_python_file", "    module = load_module_py(module_id, path)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/util/compat.py\", line 83, in load_module_py", "    spec.loader.exec_module(module)", "  File \"<frozen importlib._bootstrap_external>\", line 665, in exec_module", "  File \"<frozen importlib._bootstrap>\", line 222, in _call_with_frames_removed", "  File \"alembic/env.py\", line 82, in <module>", "    run_migrations_online()", "  File \"alembic/env.py\", line 76, in run_migrations_online", "    context.run_migrations()", "  File \"<string>\", line 8, in run_migrations", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/runtime/environment.py\", line 836, in run_migrations", "    self.get_context().run_migrations(**kw)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/runtime/migration.py\", line 330, in run_migrations", "    step.migration_fn(**kw)", "  File \"/var/www/securedrop/alembic/versions/3d91d6948753_create_source_uuid_column.py\", line 27, in upgrade", "    op.add_column(\"sources_tmp\", sa.Column(\"uuid\", sa.String(length=36)))", "  File \"<string>\", line 8, in add_column", "  File \"<string>\", line 3, in add_column", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/operations/ops.py\", line 1565, in add_column", "    return operations.invoke(op)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/operations/base.py\", line 319, in invoke", "    return fn(self, operation)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/operations/toimpl.py\", line 123, in add_column", "    schema=schema", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/ddl/impl.py\", line 172, in add_column", "    self._exec(base.AddColumn(table_name, column, schema=schema))", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/alembic/ddl/impl.py\", line 118, in _exec", "    return conn.execute(construct, *multiparams, **params)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 988, in execute", "    return meth(self, multiparams, params)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/sql/ddl.py\", line 72, in _execute_on_connection", "    return connection._execute_ddl(self, multiparams, params)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1050, in _execute_ddl", "    compiled,", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1248, in _execute_context", "    e, statement, parameters, cursor, context", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1466, in _handle_dbapi_exception", "    util.raise_from_cause(sqlalchemy_exception, exc_info)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/util/compat.py\", line 383, in raise_from_cause", "    reraise(type(exception), exception, tb=exc_tb, cause=cause)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/util/compat.py\", line 128, in reraise", "    raise value.with_traceback(tb)", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/base.py\", line 1244, in _execute_context", "    cursor, statement, parameters, context", "  File \"/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/sqlalchemy/engine/default.py\", line 552, in do_execute", "    cursor.execute(statement, parameters)", "sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) duplicate column name: uuid", "[SQL: ALTER TABLE sources_tmp ADD COLUMN uuid VARCHAR(36)]", "(Background on this error at: http://sqlalche.me/e/e3q8)", "+ echo 'Error migrating database: Backup saved to /var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite'", "Error migrating database: Backup saved to /var/lib/securedrop/backups/2021-03-23-16-33-02-db.sqlite", "+ return 1", "dpkg: error processing package securedrop-app-code (--configure):", " subprocess installed post-installation script returned error exit status 1", "Setting up securedrop-ossec-agent (3.6.0+1.9.0~rc1+xenial) ...", "Errors were encountered while processing:", " securedrop-app-code"]}

[conorsch]

Had some trouble running through the test plan myself. It looks like s3 is returning an HTTP 416 on the mon box, although if I use curl it fetches fine, with the sha256 checksum added here. At this point I think I'd prefer to rebuild and reupload the boxes and see if that helps, but I can't point to a more specific failure. Haven't investigated the GUI driver problems that @zenmonkeykstop reported, I'm not sure those are problems on my system currently. Will return for a closer look here.

[zenmonkeykstop]

That 416 is a range not supported error from the server hosting the boxes. Nuke the contents of ~.vagrant/tmp/ and you should be good to restart. I think the gui stuff was a red herring and the boxes were stalling for some other reason. On latest Vagrant they complete the boot and are accessible via SSH, tho the gui is still frozen.

[conorsch]

Pushed up a commit cleaning up the 'upgrade' scenario a bit, addressing a few of the frustrations I had debugging. Mostly it's an eye toward #5512, which will largely be a copy/paste job from the Xenial upgrade logic. These changes don't solve the core problem with this PR, which is that the upgrade boxes as advertised don't work reliably.

I believe the problem is that I intentionally fell back to an older version of the box #5794 (comment). In retrospect, I should have modified molecule/vagrant-packager/prepare.yml to sideload the updater package and avoid the problem of #5781 popping up again. I've got a build running locally now to see if that works any better.

[conorsch]

I've got a build running locally now to see if that works any better.

Sadly, doesn't resolve. I'm still seeing the box hang on first boot, during initialization of the display driver, same as @zenmonkeykstop originally reported. I've pushed up a commit with new checksums anyway, since the prepare logic is going to be required in the future, or by other builders. Make sure you remove any 1.8.0 boxes you may have locally to use the latest version.

At this point, I suggest we step back from 1.8.0 and confirm that 1.7.1 upgrade boxes work well from the develop branch, to determine where the breakage lies. @rmol since I'm out tomorrow, perhaps you can lend a hand debugging here?

[rmol]

@conorsch Yeah, of course.

[rmol]

I didn't get through either the plain old process on develop -- kept getting an error about mon being unreachable -- or via the process @zenmonkeykstop described.

I'll pick it up again tomorrow.

[conorsch]

I tested on develop just now, falling back to 1.7.1, and was able to create the boxes just fine. I'll open an issue to track, since @rmol reports different results.

[conorsch]

Was not able to get the new boxes working on my end. Tacked on a commit forcing virtio video drivers, which lets the VM boot past the cirrus stage where it was getting stuck before, but vagrant still marks the machine as in "pending" state indefinitely. I'm able to open a console to it, log in, and make network calls.

At this point, I suggest we stop debugging the Xenial 1.8.0 boxes and work on writing a fresh upgrade scenario that supports Focal.

[conorsch]

At this point, I suggest we stop debugging the Xenial 1.8.0 boxes and work on writing a fresh upgrade scenario that supports Focal.

Closing, but keeping the branch around since some of the work will be relevant for Focal. Will report progress over in #5512.