Stores Trusty & Xenial deb packages side by side #4080

conorsch · 2019-01-28T20:15:08Z

Status

Work in progress.

Description of Changes

Fixes #3961. Fixes #4051.

Changes proposed in this pull request:

Removes tor-apt repo ; tor debs are now stored in a single, consolidated apt repo
Adds customization of securedrop-app-code package for discrete dependencies under Trusty & Xenial
Builds Xenial debs inside Xenial container
Deb packages are now stored in subdirs: build/trusty/ and build/xenial/.

Testing

Given the magnitude of changes here, we must confirm that both staging and prod environments are fully functioning, on both Trusty and Xenial.

Staging environment

make build-debs && make staging passes without error
All infra tests pass molecule verify -s {libvirt,virtualbox}-staging
make build-debs-xenial && make staging-xenial passes without error
All infra tests pass molecule verify -s {libvirt,virtualbox}-staging-xenial

** Note that due to #3916 some PaX-related tests might fail, and there should be a note referring to that issue in the output.

Production environment

vagrant box add bento/ubuntu-16.04
# Ensure you have vagrant mutate installed
vagrant plugin install vagrant-mutate
vagrant mutate bento/ubuntu-16.04 libvirt

If testing xenial, you must change the base box in the Vagrantfile: replace bento/ubuntu-14.04 with bento/ubuntu-16.04 for app-prod and mon-prod

vagrant up --no-provision /prod/

In Tails, update the vars to use the apt-test server. Specifically, that entails:

In install_files/ansible-base/roles/install-fpf-repo/defaults/main.yml, set apt_repo_url: https://apt-test.freedom.press
cp install_files/ansible-base/roles/install-fpf-repo/files/{apt-test-signing-key,fpf-signing-key}.pub

Then, proceed with testing. Confirm the following in your report:

Clean installation (prod vms) on Xenial using apt-test.freedom.press completes without error (Basic server testing is successful)
Clean installation (prod vms) on Trusty using apt-test.freedom.press completes without error (Basic server testing is successful)

Deployment

At present, the "xenial" apt channel is only served up from non-prod URLs. We'll update the prod config to do the same as part of the 0.12.0 release.

Developers can now retain platform-specific deb packages built from the develop branch for faster iteration on testing the Xenial migration.

Visual review

Ensure the logic and changes introduced in this pr are sound

Checklist

If you made changes to `securedrop-admin`:

Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

Configuration tests pass

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

If you made changes to documentation:

Doc linting (make docs-lint) passed locally

emkll · 2019-01-30T16:42:07Z

Clean install on Trusty looks good with current apt setup:

vagrant@app-prod:~$ apt-cache policy tor
tor:
  Installed: 0.3.4.9-1~trusty+1
  Candidate: 0.3.4.9-1~trusty+1
  Version table:
 *** 0.3.4.9-1~trusty+1 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.3.2.10-1~trusty+1 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
     0.2.4.27-1ubuntu0.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages
     0.2.4.20-1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
vagrant@app-prod:~$ apt-cache policy securedrop-app-code
securedrop-app-code:
  Installed: 0.12.0~rc1+trusty
  Candidate: 0.12.0~rc1+trusty
  Version table:
 *** 0.12.0~rc1+trusty 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.11.1 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
     0.11.0 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
     0.10.0 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
     0.9.1 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
     0.9.0 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
vagrant@app-prod:~$ apt-cache policy ossec-agent   
ossec-agent:
  Installed: 3.0.0
  Candidate: 3.0.0
  Version table:
 *** 3.0.0 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.2 0
        500 https://apt-test-xenial.freedom.press/ trusty/main amd64 Packages

Clean install on xenial looks good:

vagrant@app-prod:~$ apt-cache policy tor
tor:
  Installed: 0.3.5.7-1~xenial+1
  Candidate: 0.3.5.7-1~xenial+1
  Version table:
 *** 0.3.5.7-1~xenial+1 500
        500 https://apt-test-xenial.freedom.press xenial/main amd64 Packages
        100 /var/lib/dpkg/status
     0.2.9.14-1ubuntu1~16.04.3 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
     0.2.7.6-1ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
vagrant@app-prod:~$ apt-cache policy ossec-client
N: Unable to locate package ossec-client
vagrant@app-prod:~$ apt-cache policy ossec-client^C
vagrant@app-prod:~$ apt-cache policy securedrop-app-code
securedrop-app-code:
  Installed: 0.12.0~rc1+xenial
  Candidate: 0.12.0~rc1+xenial
  Version table:
 *** 0.12.0~rc1+xenial 500
        500 https://apt-test-xenial.freedom.press xenial/main amd64 Packages
        100 /var/lib/dpkg/status
vagrant@app-prod:~$ apt-cache policy ossec-agent
ossec-agent:
  Installed: 3.0.0
  Candidate: 3.0.0
  Version table:
 *** 3.0.0 500
        500 https://apt-test-xenial.freedom.press xenial/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.2 500
        500 https://apt-test-xenial.freedom.press xenial/main amd64 Packages

Renames the "builder" scenario to "builder-trusty", and adds a "builder-xenial" scenario. Creates env var consumption for determining platform build. Adds tests for the "postinst" scripts, to ensure they're landing inside the deb package. (During rapid development, they were initially missed.) Also updates the target deb filepath for securedrop-app-code, since the new build logic slightly changes the parent directory name.

Mirror trusty strategy to not test debs in staging

The top-level `debian/` dir is used for source packages, and via `debian/rules` will generated `DEBIAN/` dirs for binary packages. Declaring the deb package compatibility of "5" here, which is the lowest compatibility version supported by debhelper. We were previously using inferred compat of "1", given that it was undefined. Moves changelog for securedrop-app-code The `dpkg-buildpackage` logic assumes that the package changelog exists at a specific location. The filepath doesn't appear to be overrideable via a CLI flag, so moving it. Requires corresponding update in the update_version logic.

Updates the Ansible-based build logic to use `dpkg-buildpackage`, so that we can use substvars for Depends, to provide different values for Trusty/Xenial dependencies. As such, removes the "sed" implementation to substitute the depends in-line, which was always a short-term solution anyway. The fetch logic was updated to find the deb in a different location. Include `debhelper` in the Dockerfiles used for building. `sudo` also required, simply because the Ansible logic uses it to run builds, and it wasn't in the Xenial image by default. Adding to both for clarity, since it's genuinely a dependency for both. Installs python-requests for OSSEC URL fetching The dependency has long been present, but in the Trusty container image used for building debs, `python-requests` was already installed. That's not true in the Xenial image, so we'll add it in the build role logic where it belongs.

Now using the source package layout. The DEBIAN/ dir will be automatically generated for each distro at build time.

Since version strings and deb filenames are generated via changelog strings provided by `dch` in update_version.sh, let's handle the split these changelogs and ensure each case is handled in update_version.sh This will ensure we can handle trusty or xenial-specific changes in the changelog The Xenial changelog only includes versions starting from 0.12.0, since versions prior to that did not support Xenial.

Trusty debs with now be under build/trusy, xenial debs under build/xenial Updates the test logic a bit, as well. Update securedrop-app-code deb filename for split build logic Deb file names are relevant to only staging logic, updated the staging logic to override vars for securedrop_app_code_deb

The dpkg-buildpackage logic creates a slightly different parent directory for the debs; the test vars must be updated accordingly to find the package. Additionally, the Jinja conditional logic determining Trusty/Xenial inside the app deb build logic was always returning True, meaning it always named the file "+xenial", which isn't what we want.

Use upstream images that contain a fix to the apt vulnerability, CVE-2019-3462, which was recently patched in develop (before the Xenial builder image existed). The `tzdata` dependency is required for building OSSEC packages. Not storing `python-requests` in the image file, but rather pulling in via Ansible logic, because including it generated a pyssl error. We already declare the dependency in declared in the OSSEC Ansible logic for building, so it's not strictly necessary in the container image, but it would be nice to speed up builds somewhat.

Ossec's install.sh returns 0 if the process fails, and resulting deb package do not contain compiled binaries. Let's test to ensure all expected binaries are in the deb file.

When developer systems are under heavy load, the multi-container build scenarios can fail, due to Ansible failing to gather facts from each container within 10s (the default timeout for gathering facts). Rather than edit ansible.cfg, which would affect prod SD servers, let's override the timeout exclusively for the build scenarios, via env var.

Touches up the package build logic with explicit mention of the Xenial targets. Also removes several of the known problems with Xenial, given that we've addressed them as part of recent sprints. Leaving in the config test verification mention, since we have a separate ticket tracking that (3964). Have not made mention of prod envs for Xenial, since we have more work to do on that front yet.

Tor 0.3.5.x series now supports authenticated v3 hidden services. Since *stealth* hidden services are not supported in v3, the current torrc configuration returns an error under 0.3.5.x series. Tor no longer serves 0.3.4.x series debs, so we should migrate to the 0.3.5.x series soon.

Now properly supports both Trusty & Xenial, based on the platform being configured. This allows the same Ansible logic to be applied to either platform, and it'll do the right thing. Includes configuration test updates, to remove hardcoding of "trusty" throughout, and instead substitute the result of `lsb_release -sc` via testinfra's system_info module.

As part of the 0.12.0 release, we're consolidating the separate tor-apt repo into part of the main apt.freedom.press repository. As such, we must clean up references to the old tor-apt repository on disk. Accordingly, increments the version of the `securedrop-config` package. Includes updates to relevant Ansible logic, for backwards compatibility, as well as config tests, which aim to ensure that the old tor-specific apt repos are not configured anywhere on the system.

conorsch · 2019-01-31T01:46:31Z

Ready for review! Looking for input from @kushaldas in particular. See the test plan above. Given the large amount of testing (staging & prod under both Trusty & Xenial), consider breaking up the testing work with e.g. @zenmonkeykstop. Perhaps @kushaldas can take the staging environments, and @zenmonkeykstop the prod.

The Molecule scenarios were still forcing trusty repos on xenial hosts, due to a hardcoded override. Now that we've added full Xenial support, we can remove the override. Caught this in the config tests.

kushaldas

Staging environment

make build-debs && make staging passes without error
All infra tests pass molecule verify -s {libvirt,virtualbox}-staging
make build-debs-xenial && make staging-xenial passes without error
All infra tests pass molecule verify -s {libvirt,virtualbox}-staging-xenial fails with the following errors

   =================================== FAILURES ===================================
    __________ test_pax_flags[ansible://app-staging-/usr/sbin/grub-probe] __________
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://app-staging-/usr/sbin/grub-mkdevicemap] _______
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://app-staging-/usr/bin/grub-script-check] _______
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    __________ test_pax_flags[ansible://mon-staging-/usr/sbin/grub-probe] __________
    [gw0] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://mon-staging-/usr/sbin/grub-mkdevicemap] _______
    [gw1] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    _______ test_pax_flags[ansible://mon-staging-/usr/bin/grub-script-check] _______
    [gw0] linux2 -- Python 2.7.13 /home/kdas/code/sd/bin/python2
    [XPASS(strict)] PaX flags unset at install time, see issue #3916
    === 6 failed, 447 passed, 13 skipped, 2 xfailed, 2 xpassed in 211.42 seconds ===

zenmonkeykstop

Clean installation (prod vms) on Xenial using apt-test.freedom.press completes without error
(Basic server testing is successful) - aa-status reports that haveged process is running unconfined
Clean installation (prod vms) on Trusty using apt-test.freedom.press completes without error
(Basic server testing is successful)

conorsch · 2019-02-01T01:22:36Z

@kushaldas Those failures look an awful lot like #3916 to me—mind posting a comment over there? It may be that the problem doesn't occur on Xenial boxes, in which case we can remove the strict setting on those tests. I suspect the ultimate solution is still to reorder the role includes, otherwise tests passing is dependent on the build time of the box, which certainly isn't what we want.

@zenmonkeykstop Thanks for flagging the havaged profile; see 017ba1f, which came in as part of #3833. We could shoehorn addressing that into #3964, but we should probably open a dedicated issue to evaluate AppArmor policies across the board for Xenial. What do you think?

kushaldas

I did not find any other issues in the staging environment or in the building process. I did a visual walk through of the diff a few times. This looks good to me.

conorsch · 2019-02-02T01:18:14Z

Thanks for re-review, @kushaldas!

Due to a typo that slipped in as part of #4080, the Xenial deb packages weren't being built in CI at all, causing the whole nightly run to fail. Fixes #4111.

This was overlooked in the migration away from tor-apt.freedom.press in #4080. It's harmless but unnecessary and potentially confusing.

This was overlooked in the migration away from tor-apt.freedom.press in freedomofpress#4080. It's harmless but unnecessary and potentially confusing.

This was overlooked in the migration away from tor-apt.freedom.press in #4080. It's harmless but unnecessary and potentially confusing.

conorsch added the PR: Pending additional work label Jan 28, 2019

conorsch requested review from kushaldas and emkll January 28, 2019 20:15

conorsch requested review from msheiny and redshiftzero as code owners January 28, 2019 20:15

emkll mentioned this pull request Jan 29, 2019

Upgrade Tor to 0.3.5.7 (or 0.3.4.10) #4031

Closed

emkll force-pushed the 3961-separate-subdir-per-distro-for-deb-pkg-builds branch from 4e7ae79 to 5c488d6 Compare January 30, 2019 14:56

Conor Schaefer and others added 16 commits January 30, 2019 16:16

Fix build-debs-xenial makefile target

f5073f2

Mirror trusty strategy to not test debs in staging

Copy files for securedrop-app-code package during build

d0a7929

Converts DEBIAN/ -> debian/ for securedrop-app-code

3a8ec14

Now using the source package layout. The DEBIAN/ dir will be automatically generated for each distro at build time.

Fix test for wheel in Xenial and Trusty

1a01f99

Add regression test for ossec packages

b1656e0

Ossec's install.sh returns 0 if the process fails, and resulting deb package do not contain compiled binaries. Let's test to ensure all expected binaries are in the deb file.

conorsch force-pushed the 3961-separate-subdir-per-distro-for-deb-pkg-builds branch from b6fd4af to c1c0d5f Compare January 31, 2019 00:18

conorsch force-pushed the 3961-separate-subdir-per-distro-for-deb-pkg-builds branch from c1c0d5f to 680dd37 Compare January 31, 2019 01:32

conorsch removed the PR: Pending additional work label Jan 31, 2019

conorsch requested a review from zenmonkeykstop January 31, 2019 01:45

Removes trusty override from xenial staging

ced3d89

The Molecule scenarios were still forcing trusty repos on xenial hosts, due to a hardcoded override. Now that we've added full Xenial support, we can remove the override. Caught this in the config tests.

kushaldas suggested changes Jan 31, 2019

View reviewed changes

emkll mentioned this pull request Jan 31, 2019

Explicitly declare onion services as v2 for existing installs #4092

Merged

8 tasks

conorsch mentioned this pull request Jan 31, 2019

Adds upgrade testing boxes for 0.11.1 #4093

Merged

9 tasks

zenmonkeykstop requested changes Jan 31, 2019

View reviewed changes

zenmonkeykstop mentioned this pull request Feb 1, 2019

[xenial] Verify Trusty backup -> Xenial recovery story #3960

Closed

kushaldas mentioned this pull request Feb 1, 2019

PaX flags are unset during install time #3916

Closed

kushaldas approved these changes Feb 1, 2019

View reviewed changes

conorsch merged commit e7b1004 into develop Feb 2, 2019

conorsch pushed a commit that referenced this pull request Feb 8, 2019

Fixes CI for building Xenial debs

bd60297

Due to a typo that slipped in as part of #4080, the Xenial deb packages weren't being built in CI at all, causing the whole nightly run to fail. Fixes #4111.

conorsch mentioned this pull request Feb 8, 2019

[xenial] File uploads fail on source interface on Xenial, with SecureTemporaryFile-related error #4110

Closed

This was referenced Feb 12, 2019

[xenial] Updated release-upgrades Prompt for Trusty, Xenial as appropriate #4116

Merged

Upgrade scenario not working #4125

Closed

conorsch mentioned this pull request Feb 13, 2019

fix update version script #4129

Merged

This was referenced Feb 15, 2019

CI is failing for PRs to 0.12.0 branch #4144

Closed

[0.12.0] AppArmor deny messages for lsb-release on upgrade #4161

Closed

conorsch mentioned this pull request Feb 21, 2019

Re-enables unattended upgrades to AppArmor profiles #4167

Merged

13 tasks

heartsucker deleted the 3961-separate-subdir-per-distro-for-deb-pkg-builds branch February 27, 2019 10:38

eloquence added a commit that referenced this pull request Jul 31, 2019

Remove unused tor_apt_repo_url override

f2ff4b0

This was overlooked in the migration away from tor-apt.freedom.press in #4080. It's harmless but unnecessary and potentially confusing.

eloquence mentioned this pull request Jul 31, 2019

Remove unused tor_apt_repo_url override #4655

Merged

kushaldas pushed a commit that referenced this pull request Sep 25, 2019

Remove unused tor_apt_repo_url override

f4c2a2d

This was overlooked in the migration away from tor-apt.freedom.press in #4080. It's harmless but unnecessary and potentially confusing.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stores Trusty & Xenial deb packages side by side #4080

Stores Trusty & Xenial deb packages side by side #4080

conorsch commented Jan 28, 2019 •

edited by emkll

Loading

emkll commented Jan 30, 2019 •

edited

Loading

conorsch commented Jan 31, 2019

kushaldas left a comment

zenmonkeykstop left a comment

conorsch commented Feb 1, 2019

kushaldas left a comment

conorsch commented Feb 2, 2019

Stores Trusty & Xenial deb packages side by side #4080

Stores Trusty & Xenial deb packages side by side #4080

Conversation

conorsch commented Jan 28, 2019 • edited by emkll Loading

Status

Description of Changes

Testing

Staging environment

Production environment

Deployment

Visual review

Checklist

If you made changes to securedrop-admin:

If you made changes to the system configuration:

If you made non-trivial code changes:

If you made changes to documentation:

emkll commented Jan 30, 2019 • edited Loading

conorsch commented Jan 31, 2019

kushaldas left a comment

Choose a reason for hiding this comment

Staging environment

zenmonkeykstop left a comment

Choose a reason for hiding this comment

conorsch commented Feb 1, 2019

kushaldas left a comment

Choose a reason for hiding this comment

conorsch commented Feb 2, 2019

conorsch commented Jan 28, 2019 •

edited by emkll

Loading

If you made changes to `securedrop-admin`:

emkll commented Jan 30, 2019 •

edited

Loading