docs: Add test & debugging doc

test-engineering/contract-tests/README.md

@@ -1,4 +1,4 @@
-# contile-contract-tests
+# Contile Contract Tests
 
 This directory contains the automated contract test suite for the Mozilla Tile 
 Service (MTS). Passing contract tests are a prerequisite for moving to the next 
@@ -14,13 +14,14 @@ the individual components of the suite.
 **Test Scenario: success_tiles_cached_for_identical_proxy_params** 
 ![Sequence diagram of the integration tests][sequence_diagram]
 
-To run the contract tests locally, execute the following from the repository root:
+### client
 
-```text
-docker compose \ 
--f test-engineering\contract-tests\docker-compose.yml \ 
-up --abort-on-container-exit --build
-```
+The `client` directory contains a Python-based test framework for the
+contract tests. The HTTP client used in the framework requests tiles from the
+MTS and performs checks against the responses. The framework implements response
+models for the MTS API.
+
+For more details see the client [README][client_readme]
 
 ### partner
 
@@ -39,26 +40,73 @@ MTS.
 
 For more details see the partner [README][partner_readme]
 
-### client
-
-The `client` directory contains a Python-based test framework for the
-contract tests. The HTTP client used in the framework requests tiles from the
-MTS and performs checks against the responses. The framework implements response
-models for the MTS API.
-
-For more details see the client [README][client_readme]
-
 ### volumes
 
 The `volumes` directory contains subdirectories which will be mounted as
 volumes into the Docker containers used in the contract test suite:
 
-- the `volumes/partner` directory contains a YML file which defines every
-response that the API returns keyed by form-factor and then os-family
-- the `volumes/contile` directory contains files that need to be provided to a
-MTS Docker container such as a partner settings file
 - the `volumes/client` directory contains a YML file which defines every test
 scenario that the contract test suite will run
+- the `volumes/contile` directory contains files that need to be provided to a
+MTS Docker container such as a partner settings file
+- the `volumes/partner` directory contains a YML file which defines every
+response that the API returns keyed by form-factor and then os-family
+
+
+## Local Execution
+
+To run the contract tests locally, execute the following from the repository root:
+
+**Build Contile Docker Image**
+```shell
+docker build -t app:build .
+```
+
+**Build Contract Test Docker Images & Execute Tests**
+```shell
+docker compose \
+    -f test-engineering\contract-tests\docker-compose.yml \
+    up --abort-on-container-exit --build
+```
+
+## Debugging
+
+The contract testing system is optimized to run within a set of related docker images.
+
+### client
+
+See the `Debugging` section of the client [README][client_readme] 
+
+### Contile
+
+To run the contile service, and it's dependant partner service locally, execute the 
+following from the contract-tests root:
+
+```shell
+docker-compose run -p 8000:8000 contile
+```
+
+Contile runs, by default, on `http://localhost:8000/`.
+However, it may be necessary to run the tests outside of docker, in order to debug 
+functions or manually verify expected results.
+
+**Fetching Tiles**
+
+Contile will attempt to look up your IP address if you access it using Firefox. 
+The easiest way to get a test response back would be to craft a curl request. 
+For example (presuming that Contile is running on `http://localhost:8000/v1/tiles`):
+
+```sh
+curl -v \
+    -H "UserAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:10.0) Gecko/20100101 Firefox/91.0'" \
+    -H "X-Forwarded-For: '89.160.20.115'" \
+    "http://localhost:8000/v1/tiles"
+```
+
+### partner
+
+See the `Local Execution` and `Debugging` sections of the partner [README][partner_readme] 
+
 
 [client_readme]: ./client/README.md
 [contract-test-repo]: https://github.com/mozilla-services/contile-integration-tests

test-engineering/contract-tests/client/README.md

@@ -9,13 +9,14 @@ checks.
 
 The framework implements response models for the MTS and partner APIs.
 
-For more details on contract test design, refer to the contile-contract-tests
+For more details on contract test design, refer to the Contile Contract Tests
 [README][contract_tests_readme].
 
-## Scenarios
+## Overview
 
-The client is instructed on request and response check actions via steps recorded in a 
-scenario file. A scenario is defined by a name, description and steps.
+The client is instructed on request and response check actions via scenarios, 
+recorded in the `scenarios.yml` file. A scenario is defined by a name, a description 
+and steps.
 
 ### Steps
 
@@ -112,4 +113,73 @@ Example:
                 value: '5'
 ```
 
+## Debugging
+
+To execute the test scenarios outside the client Docker container, expose the Contile 
+and partner API ports in the docker-compose.yml, set environment variables and use a 
+pytest command. It is recommended to execute the tests within a Python virtual 
+environment to prevent dependency cross contamination.
+
+### Environment Setup
+
+Install all requirements via [pip-tools][pip-tools]:
+
+```shell
+pip-sync requirements.txt dev-requirements.txt
+```
+
+With requirements installed run the code checks and test via [tox][tox]:
+
+```shell
+tox
+```
+
+See the tox configuration in the `tox.ini` for the list of environments this
+will run.
+
+### Execution
+
+1. Modify `test-engineering/contract-tests/docker-compose.yml`
+
+    In the partner definition, expose port 5000 by adding the following:
+    ```yaml
+    ports:
+      - "5000:5000"
+    ```
+
+    In the Contile definition, expose port 8000 by adding the following:
+    ```yaml
+    ports:
+      - "8000:8000"
+    ```
+
+2. Run Contile and partner docker containers.
+
+   Execute the following from the project root:
+   ```shell
+   docker compose -f test-engineering/contract-tests/docker-compose.yml up contile
+   ```
+
+3. Run the contract tests
+
+    Execute the following from the project root:
+    ```shell
+    CONTILE_URL=http://localhost:8000 \
+        PARTNER_URL=http://localhost:5000 \
+        SCENARIOS_FILE=test-engineering/contract-tests/volumes/client/scenarios.yml \
+        pytest test-engineering/contract-tests/client/tests/test_contile.py
+    ```
+    * Environment variables can alternatively be set in a pytest.ini file or through an 
+      IDE configuration
+    * Tests can be run individually using [-k _expr_][pytest-k]. 
+
+      Example executing the `success_desktop_windows` scenario:
+      ```shell
+      pytest test-engineering/contract-tests/client/tests/test_contile.py \
+          -k success_desktop_windows
+      ```
+
 [contract_tests_readme]: ../README.md
+[pip-tools]: https://pypi.org/project/pip-tools/
+[tox]: https://pypi.org/project/tox/
+[pytest-k]: https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name

test-engineering/contract-tests/partner/README.md

@@ -4,34 +4,7 @@ This directory contains a Python-based web service. The HTTP API of this service
 implements the API specification of the partner API that MTS connects to when
 requesting tiles to pass along to Firefox for display.
 
-## Setup
-
-Install all requirements via [pip-tools][pip-tools]:
-
-```text
-pip-sync requirements.txt dev-requirements.txt
-```
-
-## Code checks and tests
-
-With requirements installed run the code checks and test via [tox][tox]:
-
-```text
-tox
-```
-
-See the tox configuration in the `tox.ini` for the list of environments this
-will run.
-
-## Running the service
-
-You can run the service using `docker compose` from the root directory:
-
-```text
-docker compose run -p 5000:5000 partner
-```
-
-## API
+## Overview
 
 Once the API service is running, API documentation can be found at 
 `http://0.0.0.0:5000/docs`.
@@ -44,7 +17,7 @@ Example:
 
 Request
 
-```text
+```shell
 curl \
   -X 'GET' \
   -H 'accept: application/json' \
@@ -136,7 +109,7 @@ Example:
 
 Request
 
-```text
+```shell
 curl \
   -X 'DELETE' \
   -H 'accept: */*' \
@@ -157,7 +130,7 @@ Example:
 
 Request
 
-```text
+```shell
 curl \
   -X 'GET' \
   -H 'accept: application/json' \
@@ -193,5 +166,102 @@ Body:
 }
 ```
 
+## Local Execution
+
+To run the service locally, execute the following from the contract-tests root:
+
+```shell
+docker compose run -p 5000:5000 partner
+```
+
+The mock partner runs, by default, on `http://localhost:5000/`.
+
+The test URI path is `tilesp/desktop` for desktop tiles, or `tilesp/mobile` for mobile 
+tiles.
+
+The following query arguments are required. Optional, undefined elements should be left 
+empty (e.g. `...&dma-code=&...`) Failure to include them will return a 400 error with 
+the missing variables listed in the response (NOTE: This will produce an unexpected 500 
+or 502 error in Contile.)
+
+* partner - _string_
+* sub1 - _string_
+* sub2 - _alphanumeric_
+* country-code - _2 CAPTIAL LETTER Alpha_
+* region-code - _1 to 3 CAPITAL LETTER AlphaNumeric_
+* dma-code - _Optional Numeric_
+* form-factor - _See `ACCEPTED_{MOBILE,DESKTOP}_FORM_FACTORS`_
+* v = `1.0`
+* out = `json`
+* results = _number of tiles to return, usually 2_
+
+## Debugging
+
+It is possible to run the mock partner app outside of docker. It is ___STRONGLY___ 
+suggested to run this within its own Python virtualenv, and possibly its own 
+shell to prevent environment variable cross contamination.
+
+### Environment Setup
+
+Install all requirements via [pip-tools][pip-tools]:
+
+```shell
+pip-sync requirements.txt dev-requirements.txt
+```
+
+With requirements installed run the code checks and test via [tox][tox]:
+
+```shell
+tox
+```
+
+See the tox configuration in the `tox.ini` for the list of environments this
+will run.
+
+The `services: partner` block of `contract-tests/docker-compose.yml` lists the 
+`environment` and `volumes` needed. The following environment variables are used by 
+the mock partner app.
+
+* PORT - _default port number_
+* RESPONSES_DIR - _directory to read the [Tile Values](#tile_values)_
+* ACCEPTED_MOBILE_FORM_FACTORS - _list of allowed `form-factors` for `tilesp/mobile` responses_
+* ACCEPTED_DESKTOP_FORM_FACTORS - _list of allowed `form-factors` for `tilesp/desktop` responses_
+
+### Execution
+
+Start the mock partner app from inside the mock partner virtualenv using
+
+```sh
+gunicorn -c config/gunicorn_conf.py --preload -k uvicorn.workers.UvicornWorker main:app
+````
+
+**Contile Configuring**
+
+Use the following environment variables for Contile to contact the mock partner server:
+
+```sh
+CONTILE_MAXMINDDB_LOC=${ProjectRoot}/mmdb/GeoLite2-City-Test.mmdb
+CONTILE_ADM_ENDPOINT_URL=http://localhost:5000/tilesp/desktop
+CONTILE_ADM_MOBILE_ENDPOINT_URL=http://localhost:5000/tilesp/mobile
+CONTILE_ADM_QUERY_TILE_COUNT=5
+CONTILE_ADM_SUB1=sub1_test
+CONTILE_ADM_PARTNER_ID=partner_id_test
+CONTILE_ADM_HAS_LEGACY_IMAGE='["Example ORG", "Example COM"]'
+```
+
+`CONTILE_ADM_TIMEOUT` determines how long to wait for a response from the partner server. 
+The default value is `5` seconds. You may wish to make this much longer if you're debugging.
+
+These would be in addition to any other settings you wish to use for the Contile server.
+
+**<a name="tile_values"></a>Tile Values**
+
+The returned tile values are stored in 
+`contract-tests/volumes/partner/${country-code}/${region-code}.yml`.
+
+If different values are desired, you can either alter these files or you can copy them 
+into a new directory and use the `RESPONSES_DIR` environment variable for the 
+mock partner app.
+
 [tox]: https://pypi.org/project/tox/
 [pip-tools]: https://pypi.org/project/pip-tools/