Skip to content
This repository has been archived by the owner on Jan 7, 2024. It is now read-only.

Latest commit

 

History

History
220 lines (147 loc) · 9.5 KB

README.md

File metadata and controls

220 lines (147 loc) · 9.5 KB
Raw

Python SDK for SecureDrop

CircleCI

This SDK provides a convenient Python interface to the SecureDrop Journalist Interface API. The development of the SDK was primarily motivated by the creation of the SecureDrop Workstation based on Qubes OS.

The SDK is currently used by the SecureDrop Client that is a component of the SecureDrop Workstation. When used in Qubes OS, the SDK uses the securedrop-proxy service, as the VM which runs the client does not have network access by design.

Development

Quick Start

virtualenv --python=python3 .venv
source .venv/bin/activate
pip install --require-hashes -r dev-requirements.txt
make test

We cover all the API calls supported by the SecureDrop Journalist Interface API.

To install the SDK into your virtualenv for testing purposes:

pip uninstall securedrop-sdk
pip install git+https://github.com/freedomofpress/securedrop-sdk@my_branch#egg=securedrop-sdk

Running tests and checks

When tests are run, they replay recorded API request and response data instead of making actual API calls to a server. This is why tests can pass even when there is no server running. If you want to add new tests that make API calls or if the server ever changes its API, then you'll need to record new request and response data by following the steps outlined in Generating new cassettes.

Note: We have a CI test that does not use the recorded API request and response data in order to make sure we are testing the latest changes to the SDK against the latest server API (see test-against-latest-api in https://github.com/freedomofpress/securedrop-sdk/blob/main/.circleci/config.yml).

To run all tests and checks, run:

make check

To run all tests, run:

make test

To run all tests that make API calls over HTTP, run:

make test TESTS=tests/test_api.py

To run all tests that make API calls over qrexec, run:

make test TESTS=tests/test_apiproxy.py

To run a single test, specify file name, class name, and test name, e.g.:

make test TESTS=tests/test_api.py::TestAPI::test_get_sources

Generating new cassettes

We use vcrpy to record and replay API calls made over HTTP and a decorator called @dastollervey_datasaver to record and replay API calls made over qrexec. Each request made from a test and its response from the server is stored in a "cassette" in the data directory. Tests replay these cassettes instead of making actual API calls to a server.

If the server changes its API, then you'll see the following warning indicating that a request failed to be found in an existing cassette and that you'll need to regenerate cassettes:

Can't overwrite existing cassette ('<path-to-cassette-for-a-functional-test>') in your current record mode ('once').

The steps to generate new cassettes are split into two sections: Generating cassettes for API calls over HTTP and Generating cassettes for API calls over qrexec.

Generating cassettes for API calls over HTTP

  1. Start the server in a docker container by running:

    NUM_SOURCES=5 make dev

  2. [Skip if adding a new test] Delete the cassettes you wish to regenerate or just delete all yaml files by running:

    rm data/*.yml

  3. Generate new cassettes that make API calls over HTTP by running:

    make test TESTS=tests/test_api.py

Note: Some tests alter source and conversation data on the server so you may need to restart the server in between test runs.

Generating cassettes for API calls over qrexec

In order to generate cassettes for tests that make API calls over qrexec, you'll need to run the server and proxy on a separate VM. If this is the first time you are generating cassettes, then you'll need to first follow the steps outlined in the Setup for qrexec communication section.

Once your VMs are set up, follow these steps:

  1. Start the server in a docker container on a separate VM (called dev-server if you followed the instructions from Setup for qrexec communication) by running:

    NUM_SOURCES=5 make dev

  2. [Skip if adding a new test] Delete the cassettes you wish to regenerate or just delete all json files by running:

    rm data/*.json

  3. Comment out the @dastollervey_datasaver decorator above the test you want to generate a new cassette for or just generate all new cassettes by commenting out the decorator above all methods in the test_apiproxy.py::TestAPIProxy class.

  4. Make qrexec calls to the server and collect real response data:

    make test TESTS=tests/test_apiproxy.py

  5. Uncomment the @dastollervey_datasaver decorator wherever you commented it out.

  6. Record new cassettes from the response data collected in step 4:

    make test TESTS=tests/test_apiproxy.py

Note: If you get a 403 error it's becuase the test is trying to reuse an old TOTP code, so wait for 60 seconds and try again. Some tests alter source and conversation data on the server so you should restart the server in between test runs.

Note: Remember that file download checks don't read actual file path in the APIProxy tests as it requires QubesOS setup. You can manually uncomment those lines to execute them on QubesOS setup.

Setting up qrexec communication for SDK test VMs

If this is the first time you are generating new cassettes that make API calls over qrexec, then you'll need to set up a new VM for running the server and proxy following these steps:

  1. Create a new template cloned from the debian-10 template called dev-server-template.

  2. Install the lastest proxy package:

    wget https://apt.freedom.press/pool/main/s/securedrop-proxy/<latest-package>.deb
dpkg -i <latest-package>.deb

  3. Create /etc/sd-proxy.yaml with the following contents (assuming the VM you'll be running the SDK functional tests from is called dev-client):

    host: 127.0.0.1
scheme: http
port: 8081
target_vm: dev-client
dev: False

  4. Install Docker.

  5. Create a new Qubes VM called dev-server from the dev-server-template VM you just created.

  6. Clone securedrop on dev-server and run the server in a Docker container:

    git clone https://github.com/freedomofpress/securedrop
virtualenv .venv --python=python3
source .venv/bin/activate
pip install -r securedrop/requirements/python3/develop-requirements.txt
NUM_SOURCES=5 make dev

  7. Open a terminal in dev-client and create /etc/sd-sdk.conf with the following contents:

[proxy]
name=dev-server
  1. Update /etc/qubes-rpc/policy/securedrop.Proxy in dom0 by adding the following line to the top of the file:
dev-client dev-server allow

  1. Verify qrexec communication between VMs is set up properly. a. Make sure the server is running in a docker container in the dev-server VM that you just created. b. Comment out the @dastollervey_datasaver decorator above the test_apiproxy.py::TestAPIProxy::setUp method so that test_api_auth makes an actual API call over qrexec. c. Run test_api_auth:

     ```bash
 make test TESTS=tests/test_apiproxy.py::TestAPIProxy::test_api_auth
 ```

    Note: If the test fails, run journalctl -f in dom0 before trying again to see if access from dev-client to dev-server is being denied. A successful log looks like this:

     ```
 Aug 28 15:45:13 dom0 qrexec[1474]: securedrop.Proxy: dev-client -> dev-server: allowed to dev-server
 ```

Releasing

To make a release, you should:

  1. Create a branch named release/$new_version_number
  2. Update CHANGELOG.md and setup.py
  3. Commit the changes.
  4. Create a PR and get the PR reviewed and merged into master.
  5. git tag $new_version_number and push the new tag.
  6. Checkout the new tag locally.
  7. Push the new release source tarball to the PSF's PyPI following this documentation. Do not upload the wheel (by deleting it from your dist/ directory prior to upload).
  8. If you want to publish the new SDK release to the FPF PyPI mirror, Hop over to the the securedrop-debian-packaging repo and follow the build-a-package instructions to push the package up to our PyPI mirror: https://pypi.org/simple

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

License

The Python SecureDrop SDK is licensed in the GPLv3. See LICENSE for more details.