Fully automate updates in pinned requirements file

[lukpueh]

Description of issue or feature request:

TUF has a requirements-pinned.txt, which pins all direct and transitive dependencies for all supported Python versions. The file is created from the direct, unpinned dependencies listed in requirements.txt, using some shell commands (pip-compile for each Python version, etc.) plus manual dressing (combining files, adding per-version environment markers, etc.):
https://github.com/theupdateframework/tuf/blob/f7695dace85444041489b83d5a66cd39c761bbd6/requirements.txt#L26-L42

If a new version becomes available for any of the dependencies listed in requirements-pinned.txt it is usually detected by Dependabot, which automatically bumps the version in a PR and triggers tests, allowing us to immediately detect any breaking updates in direct and transitive dependencies.

However, Dependabot does not patch requirements-pinned.txt upon removal or addition of any direct or transitive dependency, or if any environment markers would need a change, e.g. because Python version support is removed or added. Thus re-running above routine on a regular basis is required, albeit easily missed.

Current behavior:
Semi-automatic update of pinned requirements

Expected behavior:
Fully-automatic update of pinned requirements.

Available options:

• Schedule regular run of above routine
  E.g. in a GitHub cron-style Action. Note that this also requires automation of the manual part, i.e. adding environment makers. Or alternatively, we could add one file per supported Python version, which is what the authors of pip-compile recommend (see Added cross-environment usage documentation to README.rst jazzband/pip-tools#651).
• Use state-of-the-art tooling
  pipenv? poetry?? Do they solve this issue? cc @trishankatdatadog who seems to know about "hypermodern" Python :)

[trishankatdatadog]

• Use state-of-the-art tooling
  pipenv? poetry?? Do they solve this issue? cc @trishankatdatadog who seems to know about "hypermodern" Python :)

They have their own set of problems: for example...

[lukpueh]

FYI: I recently did this for in-toto: in-toto/in-toto@4ce69ea

I think we can do the same here and go back to just using pip-compile without any postediting. Regardless, we should consider pip-compileing automatically on a regular basis to account for changes (addition/removal) in transitive dependencies.

[jku]

Yes, if dependabot won't do this for us then maybe should add Yet Another Recurring GH Action that runs pip-compile requirements.txt and, if the results are not equal to requirements-pinned.txt, files an issue. This would ensure we notice any transient dependency changes

Obviously this could lead to false alarms in cases -- like when there's a dependency update and we just haven't merged yet for one reason or another... but that might be something we can just document and live with.

[lukpueh]

NOTE: With #1867 we now also pin test requirements

[jku]

The tricky detail here (for automation) is different python versions: our current requirements-pinned.txt does not have different requirements but the requirements-test-pinned.txt in the linked PR does.

I think the only way to not manually update the test requirements file is to have separate ones for different python versions... this might be acceptable: tox could probably use a versioned file?

[lukpueh]

Yes. This is also what the pip-compile docs suggest:

Note that if you are deploying on multiple Python environments (read the section below), then you must commit a seperate output file for each Python environment. We suggest to use the {env}-requirements.txt format (ex: win32-py3.7-requirements.txt, macos-py3.10-requirements.txt, etc.).
...
If the generated requirements.txt remains exactly the same for all Python environments, then it can be used across Python environments safely. But users should be careful as any package update can introduce environment-dependant dependencies, making any newly generated requirements.txt environment-dependant too. As a general rule, it's advised that users should still always execute pip-compile on each targeted Python environment to avoid issues.

[lukpueh]

Just came across the GitHub's Dependency Graph API, while reviewing #1974.

Maybe we can use that API (or related tools) to automatically add/remove transitive dependencies from our pinned requirements files.