Migrate docs to dedicated repo

[eloquence]

We currently manage docs in the main securedrop repo, which significantly complicates the docs maintenance story:

• We may want to make docs updates after a signed tag has been created. We now use a separate stable branch to track the current stable version of the documentation. To keep this branch up-to-date, we have to deal with potential for core-related merge-conflicts, and we are currently force-pushing this branch as part of the release process, which is not a great practice, esp. in a highly security-sensitive repo.

• We have to worry about side effects that docs merges have on journalists/admins, because Journalist and Admin Workstations hold a full git checkout of this repo and keep it up-to-date (we will eventually change this: Package Tails workstation code #3502). For example, forced updates to tags conflict with the current workstation updater logic, which is a big part of the reason we're using a stable branch instead of a tag.

• It's difficult to cleanly separate docs maintainership from SecureDrop core maintainership within a single repo, meaning that only folks who are trusted to merge code changes are currently able to merge docs changes.

In the context of the SecureDrop Workstation, we've already split out docs into a dedicated repo: https://github.com/freedomofpress/securedrop-workstation-docs

This has worked well in practice and maintainership of docs is more readily granted (e.g. @rocodes and I can merge there). The main complication is that we sometimes have to line up 2 PRs (a code change and its documentation) at the same time, e.g.: freedomofpress/securedrop-workstation#586 and freedomofpress/securedrop-workstation-docs#55 .

So let's do the same for SD Core. In previous discussions there seemed to be consensus towards this change, so the main questions are likely about the "how" not the "if".

[eloquence]

Should we start from a fork and then prune it down to docs-only? And if so, in that fork (e.g., a securedrop-docs repo), do we want to use a repo cleaner or history rewriting tool like https://github.com/newren/git-filter-repo to reduce the repo's size while still preserving a history of docs changes?

[zenmonkeykstop]

Would it be worth adding this repo as a submodule in securedrop core? Possibly more dev overhead, but it would help with associating a specific release with a specific docs version.

[eloquence]

associating a specific release with a specific docs version.

I defer to others on the submodule question. I'm not sure more precision than "applies to the current stable release" is required, but in principle, if we have a dedicated docs repo, we could also use tags again in the way RTD supports -- pushing e.g. a 1.6.0 tag the moment SecureDrop 1.6.0 is released, while still being able to easily forward that tag in the event we want to deploy a docs change after the release has been cut.

[rmol]

Just want to ask that translation be considered in the transition. See https://github.com/freedomofpress/securedrop/issues/2604

[eloquence]

@joaedwar and @eloquence will collaborate on this during the 8/20-9/2 sprint. As noted above, there are still open questions around translation & the best way to couple docs to a specific production release, which we likely won't be able to fully answer yet, but we should be able to get to a test setup & repo we can experiment with.

[eloquence]

@joaedwar has done some initial research on use of https://github.com/newren/git-filter-repo and will provide step-by-step instructions for arriving at a docs-only repo history from a SecureDrop repo clone. If the result looks good, we'll push it up to https://github.com/freedomofpress/securedrop-docs-alpha and then aim to get a first build working at an independent RTD URL.

[buildbricks]

For this migration effort, we can use git-filter-repo to set up a docs-only repo history. git-filter-repo is a single-file python script, so installation is straightforward and just requires copying it into $PATH.

Prerequisites

• To install git-filter-repo you'll need git version 2.22.0 at a minimum; some features require git 2.24.0 or later
• Python3 version 3.5 or higher

Procedure

1. To install, run the following command in the directory pointed to by git --exec-path:
   pip3 install git-filter-repo

• (Optional) You can also do a manual install by following these instructions.

2. You can then execute the git filter-repo command on the file paths that you would like to retain. For instance, to only keep the README.md file plus the docs directory, run the command:
   git filter-repo --path README.md --path docs

---

Summary:

1. Python should be installed and added to the system's path.
2. Git should be installed and git also added to the system's path.
3. Download git-filter-repo
4. Call git --exec-path
5. Move the git-filter-repo file into that location (git's path).
6. To use, type git filter-repo.

We can use these for reference:

• https://www.mankier.com/1/git-filter-repo
• https://github.com/newren/git-filter-repo

[eloquence]

Thanks @joaedwar! I've updated https://github.com/freedomofpress/securedrop-docs-alpha per the latest develop and run git filter-repo as follows:

git filter-repo --path README.md --path docs --path CODE_OF_CONDUCT.md

I've also removed all non-release branches. Here's an outline of how I would envision next steps:

• Peer review of the repo state. This is important so we are working from an agreed upon foundation and don't have to redo the repo cleanup later. Is it reasonable to filter the history as we've done here, or would folks prefer a different approach? Will raise at standup tomorrow.

• Agree upon branch/tagging strategy for initial use.

• Rename repo to securedrop-docs-beta. This indicates that we will no longer rewrite the repo history from this point forward: all changes will go through PRs and agreed upon maintainership rules. Configure branch protection, contributor write access, Slack integration, etc.

• Restore build/CI logic with a beta build target on ReadTheDocs. Update README; re-add CONTRIBUTING.md (customized for this purpose) and LICENSE. Update docs contributor guidelines https://docs.securedrop.org/en/stable/development/documentation_guidelines.html and other outdated references.

• After some beta testing, rename repo to securedrop-docs. This indicates that the repo is now official. Turn off builds from securedrop repo. Migrate docs-beta.securedrop.org to docs.securedrop.org. Remove docs from securedrop repo (we can always undo since we're not rewriting history in that repo).

[conorsch]

Took a look at the history of the new docs repo and it looks great to my eye. Talked through the proposed next steps with @eloquence just now, and those also sound great. The first PR into the new repo can includes a lot of the tracked follow-ups like build, CI, and README cleanup.

Once we're in the "beta" stage, we can use the readthedocs.io/org URLs for testing changes. Let's hold off on using docs*.securedrop.org until the final prod stage. Ideally we can pull off the whole migration to prod-ready by the string freeze for the 1.6 release, i.e. 2020-09-22, but that's not a hard deadline, and we can always kick back till after release if we must—it'll just mean repeating the repo setup steps documented above.

Thanks @joaedwar for driving on this!

[eloquence]

After discussion, we agreed that it'd make sense to switch this repo to LFS, given that the screenshots otherwise will continue to balloon it every time we update those.

This is now done on https://github.com/freedomofpress/securedrop-docs-alpha . I've used the very nice git lfs migrate tool (included in recent versions of git-lfs), as follows:

git lfs migrate import --include="*.png,*.xcf,*.draw,*.svg,*.vsdx,*.jpg" --everything

(SVG is debatable but I figured "all images" is the simplest rule of thumb to continue to enforce.) The .gitattributes file, which configures LFS usage for future file creation, has been automatically created by the migration tool and is part of the initial commit in the rewritten history.

I also removed all historical tags (we still have the release branches in case we want to continue to use the release branch model), and ensured that the LFS references are applied to all branches in this repo. As a result the non-LFS checked out size is now just under 7MB, and any checkout that includes the screenshots will only fetch the versions actually needed.

We'll need to document this after the switchover but that should be done through a PR process, so I think we're close to promoting this to beta once we agree on the tagging/branching strategy we want to use for 1.6.0.

[eloquence]

If I'm reading the discussion in readthedocs/readthedocs.org#1846 correctly, the move to LFS may necessitate some changes to our RTD build logic. https://test-builds.readthedocs.io/en/git-lfs/ has an example configuration that installs LFS as part of the build.

[conorsch]

That largely matches what we're doing in e.g. https://github.com/freedomofpress/securedrop-debian-packaging/blob/05eedc6e2a5d9a1c53ebb6dda0a2188365063e05/.circleci/config.yml#L225-L232, just specific to RTD. It'd be nice to run that lfs logic only inside RTD, maybe via env var.

[eloquence]

@zenmonkeykstop and I discussed this a bit today. For now, we would propose a slight variation of the process used for https://github.com/freedomofpress/securedrop-workstation-docs for versioning the standalone docs repo:

1. Use a single main branch in securedrop-docs as the primary working branch into which documentation changes get merged.

2. Use a semantic version tag to identify the commit that corresponds to the current production release (e.g., 1.5.0 for the commit that corresponds to the 1.5.0 version of the docs). RTD will then pick the most recent tag as its "stable" version per its versioning logic.

3. As we do in https://github.com/freedomofpress/securedrop-workstation-docs (and similar to the stable branch in SecureDrop Core), we would rely on developer-signed commits and tags, and not require a full signing ceremony. This would allow us to forward the tag relatively quickly when needed.

4. If we need to create a branch, e.g., to cherry-pick an emergency commit into the current stable release version without picking up other changes in main, we can always do so on an as-needed basis. Note that tags and the commits they describe are preserved and not garbage-collected even if a branch is later deleted, so we can preserve a full history of versions this way.

This seems like the simplest model to start with -- we can switch to a model with two or more long-lived branches if the frequency of branching warrants it.

Will discuss at standup tomorrow, and if this sounds like a reasonable plan, we may be ready to promote the securedrop-docs-alpha repo to a beta stage and start PR'ing changes into it, starting with the build & CI logic.

[conorsch]

Use a semantic version tag to identify the commit that corresponds to the current production release (e.g., 1.5.0 for the commit that corresponds to the 1.5.0 version of the docs). RTD will then pick the most recent tag as its "stable" version per its versioning logic.

In the case of hotfixing the docs, for example editing a typo immediately after release, we can use the postrelease syntax, e.g. 1.5.0 -> 1.5.0-1. That's in line with PEP440 https://www.python.org/dev/peps/pep-0440/#post-release-separators and would not require us to clobber tags to keep RTD accurate, even if the stable docs aren't exactly in sync with an SD release. Sounds great. 👍

[eloquence]

I've now renamed the repo to https://github.com/freedomofpress/securedrop-docs-beta and will file an infra issue to finalize permissions.

[eloquence]

I've also renamed the default branch to main per the above. I've not deleted the release/* branches yet, in case we change our mind about the branching strategy (they could also be useful if we want to-retag the tips of those branches with historical versions so we have a full tag history for docs).

[eloquence]

For simplicity we've decided to forego the -beta naming of the repo, but please note that https://github.com/freedomofpress/securedrop-docs is still experimental and not ready for docs PRs yet.

[eloquence]

Update on current state:

1. We now have https://securedrop-docs.readthedocs.io/ building both latest and stable from https://github.com/freedomofpress/securedrop-docs

2. The stable version has the tag 1.5.0-1. That's to make it clear that this includes some additional changes that have been merged into develop since SecureDrop 1.5.0 was released.

3. We now have the first docs PR that's targeting the securedrop-docs repo but not develop in the SecureDrop repo (Add documentation for /users endpoint securedrop-docs#3). I would recommend that we declare the formal cutover ASAP, to avoid divergence of docs in two repos.

4. Prior to cutover of https://docs.securedrop.org/, I'll prepare a draft PR into this repo which removes the docs and updates the README. We can merge that PR once https://docs.securedrop.org/ is pointing to the new repo.

5. After cutover, I would recommend transferring docs-only issues over into the securedrop-docs repo, following our general practice of filing issues in the repo in which the PR is expected.

[eloquence]

Status update: We are now using the standalone repo to build the docs at https://docs.securedrop.org/ 🎉 . You can verify it by looking at this page: https://docs.securedrop.org/en/latest/development/journalist_api.html#get-a-list-of-all-users-get

The documentation for the /users endpoint was merged only into securedrop-docs via freedomofpress/securedrop-docs#3 . Note also that the 1.5.0-1 tag describes an earlier commit, so this documentation is not included in the stable version at https://docs.securedrop.org/en/stable/development/journalist_api.html

Next up, in order, are:

• review of Remove docs, update CI config & tooling #5520, which removes the docs from this repo and updates the tooling to support the use of securedrop-docs
• Update release management guidelines securedrop-docs#8 , which updates the RM guidelines
• Add basic version update script securedrop-docs#6, which adds a version bump script to the securedrop-docs repo.

After that, still pending to close this issue:

• Update documentation guidelines securedrop-docs#4 (guidelines for documentation contributors)

[eloquence]

This is now fully resolved. I'll file a follow-up issue to migrate docs issues over in a few weeks, assuming we stick with this setup.