Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for PyPI trusted publisher and NPM provenance #511

Merged
merged 6 commits into from
Jun 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/source/get_started/making_release_from_releaser.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ already uses Jupyter Releaser.

- Add the token as `ADMIN_GITHUB_TOKEN` in the [repository secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) of your fork. The token must have `repo` and `workflow` scopes.

- Set up PyPI:

<details><summary>Using PyPI token (legacy way)</summary>

- If the repo generates PyPI release(s), create a scoped PyPI [token](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github). We recommend using a scoped token for security reasons.

- You can store the token as `PYPI_TOKEN` in your fork's `Secrets`.
Expand All @@ -34,8 +38,21 @@ already uses Jupyter Releaser.
owner1/repo1/path/to/package2,token2
```

</details>

<details><summary>Using PyPI trusted publisher (modern way)</summary>

- Set up your PyPI project by [adding a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
- if you use the example workflows, the _workflow name_ is `publish-release.yml` (or `full-release.yml`) and the
_environment_ should be left blank.
- Ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.pypi.org/trusted-publishers/using-a-publisher/))

</details>

- If the repo generates npm release(s), add access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens), saved as `NPM_TOKEN` in "Secrets".

> If you want to set _provenance_ on your package, you need to ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.npmjs.com/generating-provenance-statements#publishing-packages-with-provenance-via-github-actions)).

## Prep Release

- Go to the "Actions" tab in your fork of `jupyter_releaser`
Expand Down
19 changes: 18 additions & 1 deletion docs/source/how_to_guides/convert_repo_from_releaser.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,11 @@ A. Prep the `jupyter_releaser` fork:
[repository secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository).
The token will need "public_repo", and "repo:status" permissions.

- [ ] Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.
- [ ] Set up PyPI:

<details><summary>Using PyPI token (legacy way)</summary>

- Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.
_Note_ For security reasons, it is recommended that you scope the access
to a single repository, and use a variable called `PYPI_TOKEN_MAP` that is formatted as follows:

Expand All @@ -39,8 +43,21 @@ A. Prep the `jupyter_releaser` fork:
owner1/repo1/path/to/package2,token2
```

</details>

<details><summary>Using PyPI trusted publisher (modern way)</summary>

- Set up your PyPI project by [adding a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
- if you use the example workflows, the _workflow name_ is `publish-release.yml` (or `full-release.yml`) and the
_environment_ should be left blank.
- Ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.pypi.org/trusted-publishers/using-a-publisher/))

</details>

- [ ] If needed, add access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens), saved as `NPM_TOKEN`.

> If you want to set _provenance_ on your package, you need to ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.npmjs.com/generating-provenance-statements#publishing-packages-with-provenance-via-github-actions)).

B. Prep target repository:

- [ ] Switch to Markdown Changelog
Expand Down
23 changes: 21 additions & 2 deletions docs/source/how_to_guides/convert_repo_from_repo.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ See checklist below for details:
- Markdown changelog
- Bump version configuration (if using Python), for example [hatch](https://hatch.pypa.io/latest/)
- [Access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) with access to target GitHub repo to run GitHub Actions.
- Access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github)
- Set up:
- \[_modern way_\] [Add a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) to your PyPI project
- \[_legacy way_\] Access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github)
- If needed, access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens).

## Checklist for Adoption
Expand All @@ -22,10 +24,27 @@ See checklist below for details:
access token to allow for branch protection rules, which block the pushing
of commits when using the `GITHUB_TOKEN`, even when run from an admin user
account.
- [ ] Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.

- [ ] Set up PyPI:

<details><summary>Using PyPI token (legacy way)</summary>

- Add access token for the [PyPI registry](https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#saving-credentials-on-github) stored as `PYPI_TOKEN`.
_Note_ For security reasons, it is recommended that you scope the access
to a single repository. Additionally, this token should belong to a
machine account and not a user account.

</details>

<details><summary>Using PyPI trusted publisher (modern way)</summary>

- Set up your PyPI project by [adding a trusted publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/)
- if you use the example workflows, the _workflow name_ is `publish-release.yml` (or `full-release.yml`) and the
_environment_ should be left blank.
- Ensure the publish release job as `permissions`: `id-token : write` (see the [documentation](https://docs.pypi.org/trusted-publishers/using-a-publisher/))

</details>

- [ ] If needed, add access token for [npm](https://docs.npmjs.com/creating-and-viewing-access-tokens), saved as `NPM_TOKEN`. Again this should
be created using a machine account that only has publish access.
- [ ] Ensure that only trusted users with 2FA have admin access to the
Expand Down
11 changes: 8 additions & 3 deletions example-workflows/full-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ on:
jobs:
full_release:
runs-on: ubuntu-latest
permissions:
# This is useful if you want to use PyPI trusted publisher
# and NPM provenance
id-token: write
steps:
- uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1

Expand All @@ -51,9 +55,10 @@ jobs:
- name: Finalize Release
id: finalize-release
env:
PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
TWINE_USERNAME: __token__
# The following are needed if you use legacy PyPI set up
# PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
# PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
# TWINE_USERNAME: __token__
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
uses: jupyter-server/jupyter-releaser/.github/actions/finalize-release@v2
with:
Expand Down
11 changes: 8 additions & 3 deletions example-workflows/publish-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ on:
jobs:
publish_release:
runs-on: ubuntu-latest
permissions:
# This is useful if you want to use PyPI trusted publisher
# and NPM provenance
id-token: write
steps:
- uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1

Expand All @@ -30,9 +34,10 @@ jobs:
- name: Finalize Release
id: finalize-release
env:
PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
TWINE_USERNAME: __token__
# The following are needed if you use legacy PyPI set up
# PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
# PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }}
# TWINE_USERNAME: __token__
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
uses: jupyter-server/jupyter-releaser/.github/actions/finalize-release@v2
with:
Expand Down
5 changes: 5 additions & 0 deletions jupyter_releaser/lib.py
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,11 @@ def publish_assets( # noqa
if release_url and len(glob(f"{dist_dir}/*.whl")):
twine_token = python.get_pypi_token(release_url, python_package_path)

if twine_token:
# tell GitHub Actions to mask the token in any console logs,
# to avoid leaking it
util.run(f'echo "::add-mask::{twine_token}"')

if dry_run:
# Start local pypi server with no auth, allowing overwrites,
# in a temporary directory
Expand Down
7 changes: 7 additions & 0 deletions jupyter_releaser/npm.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,13 @@ def handle_npm_config(npm_token):
auth_entry = ""

text += f"\n{reg_entry}\n{auth_entry}"

if os.environ.get(util.GH_ID_TOKEN_TOKEN_VAR, ""):
util.log("Turning on NPM provenance as id-token permission is set.")
# See documentation https://docs.npmjs.com/generating-provenance-statements
# Also https://github.blog/2023-04-19-introducing-npm-package-provenance/
text += "\nprovenance=true"

text = text.strip() + "\n"
util.log(f"writing npm config to {npmrc}")
npmrc.write_text(text, encoding="utf-8")
Expand Down
55 changes: 55 additions & 0 deletions jupyter_releaser/python.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,26 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
import atexit
import json
import os
import os.path as osp
import re
import shlex
from glob import glob
from io import BytesIO
from pathlib import Path
from subprocess import PIPE, CalledProcessError, Popen
from tempfile import TemporaryDirectory

import requests

from jupyter_releaser import util

PYPROJECT = util.PYPROJECT
SETUP_PY = util.SETUP_PY

PYPI_GH_API_TOKEN_URL = "https://pypi.org/_/oidc/github/mint-token" # noqa


def build_dist(dist_dir, clean=True):
"""Build the python dist files into a dist folder"""
Expand Down Expand Up @@ -95,11 +101,60 @@ def check_dist(
util.run(cmd)


def fetch_pypi_api_token() -> "str":
"""Fetch the PyPI API token for trusted publishers

This implements the manual steps described in https://docs.pypi.org/trusted-publishers/using-a-publisher/
as of June 19th, 2023.

It returns an empty string if it fails.
"""
util.log("Fetching PyPI OIDC token...")

url = os.environ.get(util.GH_ID_TOKEN_URL_VAR, "")
auth = os.environ.get(util.GH_ID_TOKEN_TOKEN_VAR, "")
if not url or not auth:
util.log(
"Please verify that you have granted `id-token: write` permission to the publish workflow."
)
return ""

headers = {"Authorization": f"bearer {auth}", "Accept": "application/octet-stream"}

sink = BytesIO()
with requests.get(f"{url}&audience=pypi", headers=headers, stream=True, timeout=60) as r:
r.raise_for_status()
for chunk in r.iter_content(chunk_size=8192):
sink.write(chunk)
sink.seek(0)
oidc_token = json.loads(sink.read().decode("utf-8")).get("value", "")

if not oidc_token:
util.log("Failed to fetch the OIDC token from PyPI.")
return ""

util.log("Fetching PyPI API token...")
sink = BytesIO()
with requests.post(PYPI_GH_API_TOKEN_URL, json={"token": oidc_token}, timeout=10) as r:
r.raise_for_status()
for chunk in r.iter_content(chunk_size=8192):
sink.write(chunk)
sink.seek(0)
api_token = json.loads(sink.read().decode("utf-8")).get("token", "")

return api_token


def get_pypi_token(release_url, python_package):
"""Get the PyPI token

Note: Do not print the token in CI since it will not be sanitized
if it comes from the PYPI_TOKEN_MAP"""
trusted_token = os.environ.get(util.GH_ID_TOKEN_TOKEN_VAR, "")

if trusted_token:
return fetch_pypi_api_token()

twine_pwd = os.environ.get("PYPI_TOKEN", "")
pypi_token_map = os.environ.get("PYPI_TOKEN_MAP", "").replace(r"\n", "\n")
if pypi_token_map and release_url:
Expand Down
3 changes: 3 additions & 0 deletions jupyter_releaser/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,9 @@

GIT_FETCH_CMD = "git fetch origin --filter=blob:none --quiet"

GH_ID_TOKEN_URL_VAR = "ACTIONS_ID_TOKEN_REQUEST_URL" # noqa
GH_ID_TOKEN_TOKEN_VAR = "ACTIONS_ID_TOKEN_REQUEST_TOKEN" # noqa


def run(cmd, **kwargs):
"""Run a command as a subprocess and get the output as a string"""
Expand Down