Tor functional test auto generation of instance_information.json #3705

msheiny · 2018-08-10T19:08:50Z

Status

Ready for review

Description of Changes

I specifically only tested this against the upgrade scenario but really any staging scenario should work that includes the app-test role (where the logic resides).

Changes proposed in this pull request:

Adds an tor ansible fact gathering script to yank out tor details on the application server
Adds logic to the app-test role to pull back those details into a local file for functional testing in-take

Testing

How should the reviewer test this PR?

molecule converge -s upgrade (you must be on a kvm box, and if you havent run this in a while... its going to take 30min+ depending on your connection the first run. It'll get stuck at the vagrant stage while pulling 3GB+ of boxes.)
molecule side-effect -s upgrade (will upgrade instances to whatever debs you have in build plus build the instance_information.json locally)
cd securedrop && ./bin/dev-shell ./bin/run-test --capture=no -v tests/functional

while that is going ^^ since we have a current failing test scenario the best way to confirm things are working quickly is to log into the app server:

molecule login -s upgrade -h app
and then tail the apache logs to make sure connections are happening
tail -f /var/log/apache2/journalist-access.log (as root)

Deployment

Any special considerations for deployment? Consider both:

Only affects CI and local dev-env.

Checklist

If you made changes to the server application code:

Linting (make ci-lint) and tests (make -C securedrop test) pass in the development container

If you made non-trivial code changes:

I have written a test plan and validated it for this PR

This is really for functional testing in staging environments, but the script doesnt hurt to be installed in prod. It doesn't elevate permissions or expose any sensitive details - you need to run as root in order to gain useful data.

This commit adds logic to the test runner so that a functional test json config gets dumped for usage in the tbb selenium test tools against staging.

Without this change, the upgrade scenario would utilize the app-test logic from the old repo. Which is not what we want. This is of course kind of "hacky". I welcome the opportunity to improve this with feedback :)

I'm not sure we are ready to shove this on prod instances AND I was running into weird old/new role logic under the molecule scenario. Ideally the fact should be in both roles but I dunno.. this seemed like an easier short-term fix.

This directory no longer exists with modern molecule versions and the ephemeral directory is now something like /tmp/molecule/${git_dir}/${scenario_dir}

msheiny · 2018-08-10T19:09:03Z

ahhh shoot one sec... gotta retarget this

zenmonkeykstop · 2018-08-14T04:46:27Z

(tl;dr: LGTM)

Tested as follows on TBBGenTestConfig branch:

make build-debs
molecule converge -s upgrade

Logged into app-staging with molecule login -s upgrade -h app-staging, set up test db:

sudo -u www-data bash
   cd /var/www/securedrop/
   ./manage.py reset
   ./create-dev-data.py # edited script to just set up 'journalist' admin account

Logged out of app-staging, created instance_information.json and ran tests as follows:

molecule side-effect -s upgrade
cd securedrop && ./bin/dev-shell  ./bin/run-test -v tests/functional > results.txt

Reviewed tests/functional/instance_information.json and test output in results.txt. Correct info in JSON file, and tests ran, with some unrelated exisiting failures (mostly timeouts).

Repeated testing but using Vagrant staging environment instead (make build-debs && vagrant up /staging). tests/functional/instance_information.json was created with correct details. Functional tests ran with some unrelated failures.

It might be good to change the default 'sleep_time' value to something longer than 10sec - 30sec seems to be a more reasonable value, at least for me. Otherwise this looks good to go.

redshiftzero · 2018-08-15T17:45:06Z

So the changes in #3697 mean that the application tests are failing to run in the staging CI job:

TASK [Run application tests] ***************************************************
    Friday 10 August 2018  19:28:17 +0000 (0:00:00.037)       0:01:08.223 *********
    fatal: [app-staging]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 127, "stderr": "Shared connection to 52.36.194.59 closed.\r\n", "stdout": "/home/sdrop/.ansible/tmp/ansible-tmp-1533929297.62-93522333058246/app-tests.sh: line 13: pytest: command not found\r\n", "stdout_lines": ["/home/sdrop/.ansible/tmp/ansible-tmp-1533929297.62-93522333058246/app-tests.sh: line 13: pytest: command not found"]}
    ...ignoring

We want (let me know if you disagree):

Get functional tests running in an external test container ([functional testing] Support tor-browser-selenium tests against app-staging in CI #3661)
Application unit tests running in the staging environment

In addition, the testinfra staging test failures look to be legitimate due to the removal of testing related pip deps from the application server (not introduced in this PR). I think it makes sense to merge this and then address the unit tests in staging / testinfra fix in a followup PR.

redshiftzero · 2018-08-15T18:03:23Z

OK upon discussion, let's remove the application unit test run from staging since it's done in the ci/circleci: tests job. When we have grsecurity kernels running in the staging environment (i.e. when we're using nested virt to spin up the staging boxes) then it's worth adding the application unit test run back in (noting that the reason why the test run is slow is primarily the functional and pages-layout tests, which we would not run). Until then I'm going to merge this so we can continue on

msheiny added 6 commits August 9, 2018 15:12

Logic to dump app tor onion data to func config

e8b09c2

This commit adds logic to the test runner so that a functional test json config gets dumped for usage in the tbb selenium test tools against staging.

Wire-up current app-test role to our upgrade scenario

8166cf0

Without this change, the upgrade scenario would utilize the app-test logic from the old repo. Which is not what we want. This is of course kind of "hacky". I welcome the opportunity to improve this with feedback :)

Move tor fact logic from tor role --> app-test role

784f4b8

I'm not sure we are ready to shove this on prod instances AND I was running into weird old/new role logic under the molecule scenario. Ideally the fact should be in both roles but I dunno.. this seemed like an easier short-term fix.

ansible spacing' and tag nits

ddaaea4

Remove legacy reference to .molecule/roles dir

953b446

This directory no longer exists with modern molecule versions and the ephemeral directory is now something like /tmp/molecule/${git_dir}/${scenario_dir}

msheiny requested review from conorsch, heartsucker and redshiftzero as code owners August 10, 2018 19:08

msheiny changed the base branch from develop to tbb-0.9.0 August 10, 2018 19:09

msheiny added the functional_tests label Aug 13, 2018

redshiftzero approved these changes Aug 15, 2018

View reviewed changes

redshiftzero merged commit 885ccfa into tbb-0.9.0 Aug 15, 2018

redshiftzero deleted the TBBGenTestConfig branch August 15, 2018 18:04

eloquence mentioned this pull request Aug 16, 2018

[functional testing] Extract tor details from staging to instance_information.json #3621

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tor functional test auto generation of instance_information.json #3705

Tor functional test auto generation of instance_information.json #3705

msheiny commented Aug 10, 2018

msheiny commented Aug 10, 2018 •

edited

Loading

zenmonkeykstop commented Aug 14, 2018

redshiftzero commented Aug 15, 2018

redshiftzero commented Aug 15, 2018

Tor functional test auto generation of instance_information.json #3705

Tor functional test auto generation of instance_information.json #3705

Conversation

msheiny commented Aug 10, 2018

Status

Description of Changes

Testing

Deployment

Checklist

If you made changes to the server application code:

If you made non-trivial code changes:

msheiny commented Aug 10, 2018 • edited Loading

zenmonkeykstop commented Aug 14, 2018

redshiftzero commented Aug 15, 2018

redshiftzero commented Aug 15, 2018

msheiny commented Aug 10, 2018 •

edited

Loading