Add developer docs for CI debugging

docs/development/testing_continuous_integration.rst

@@ -102,3 +102,77 @@ To run the tests locally:
 You can use ``./devops/gce-nested/ci-runner.sh`` to provision the remote hosts
 while making changes, including rebuilding the Debian packages used in the
 Staging environment. See :doc:`virtual_environments` for more information.
+
+Debugging CI Issues and Connecting to Remote Instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For the staging tests, a container will be spawned on CircleCI, which will then
+create a Google Compute instance with nested virtualization and will set up the
+virtual environment and run the playbooks on that remote.
+
+Cloud instances are deleted after the test run is completed, whether a test run
+passes or fails. In order to debug the state of the remote instance, we must first
+ensure that the instance is not automatically destroyed. Note that there is also
+a cron job that destroys instances daily as well. The following is an example
+of a commit to apply to a branch in order disable the deletion for the Focal staging job:
+
+.. code:: Diff
+
+    diff --git a/.circleci/config.yml b/.circleci/config.yml
+    index 4a9b0bd4c..d9aea01b8 100644
+    --- a/.circleci/config.yml
+    +++ b/.circleci/config.yml
+    @@ -354,13 +354,6 @@ jobs:
+               BASE_OS=focal make ci-go
+             no_output_timeout: 35m
+
+    -      - run:
+    -          name: Ensure environment torn down
+    -          # Always report true, since env should will destroyed already
+    -          # if all tests passed.
+    -          command: make ci-teardown || true
+    -          when: always
+    -
+         - store_test_results:
+             path: ~/sd/junit
+
+    diff --git a/devops/gce-nested/ci-go.sh b/devops/gce-nested/ci-go.sh
+    index 850324ecc..776120df4 100755
+    --- a/devops/gce-nested/ci-go.sh
+    +++ b/devops/gce-nested/ci-go.sh
+    @@ -16,4 +16,3 @@ export BASE_OS="${BASE_OS:-xenial}"
+
+    ./devops/gce-nested/gce-start.sh
+    ./devops/gce-nested/gce-runner.sh
+    -./devops/gce-nested/gce-stop.sh
+    diff --git a/devops/scripts/create-staging-env b/devops/scripts/create-staging-env
+    index 3b9a2c7f8..df2ccfe3d 100755
+    --- a/devops/scripts/create-staging-env
+    +++ b/devops/scripts/create-staging-env
+    @@ -33,7 +33,7 @@ printf "Creating staging environment via '%s'...\\n" "${securedrop_staging_scena
+    virtualenv_bootstrap
+    # Are we in CI? Then lets do full testing post install!
+    if [ "$USER" = "sdci" ]; then
+    -    molecule test -s "${securedrop_staging_scenario}"
+    +    molecule test --destroy=never -s "${securedrop_staging_scenario}"
+    else
+       molecule "${MOLECULE_ACTION:-converge}" -s "${securedrop_staging_scenario}" "${EXTRA_ANSIBLE_ARGS[@]}"
+    fi
+
+Once that commit is pushed, run the appropriate ``staging-test-with-rebase`` job
+with ssh using with CircleCI. Once logged into that container, you can ssh into the
+Google Compute host:
+
+.. code:: sh
+
+    ssh -i /tmp/gce-nested/gce sdci@<ip adress>
+
+Once on the GCP host, the SecureDrop source is in ``/home/sdci/securedrop-source``
+and you may activate the virtualenv, list the molecule instances and connect to
+VM instances:
+
+.. code:: sh
+
+    cd securedrop-source
+    source .venv/bin/activate
+    molecule list
+    molecule login -s libvirt-staging-focal --host app-staging