-
-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build Docker image and push to GHCR #230
Changes from all commits
783267b
e453f8c
0d8d505
aed6c4b
f1f014b
cf5ce17
5ded531
a869dd3
a360fcb
f51682f
7ea8313
bacb626
d03addb
153ccde
cfb9d93
1b9f21a
80b1d50
da55441
36965cb
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
--- | ||
|
||
name: 🏗️ | ||
|
||
on: # yamllint disable-line rule:truthy | ||
pull_request: | ||
push: | ||
branches: ["release/*", "unstable/*"] | ||
workflow_dispatch: | ||
inputs: | ||
tag: | ||
description: Docker image tag | ||
required: true | ||
type: string | ||
|
||
jobs: | ||
smoke-test: | ||
uses: ./.github/workflows/reusable-smoke-test.yml | ||
build-and-push: | ||
if: github.event_name != 'pull_request' | ||
runs-on: ubuntu-latest | ||
needs: | ||
- smoke-test | ||
timeout-minutes: 10 | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Build Docker image | ||
run: | | ||
DOCKER_TAG="${DOCKER_TAG/'/'/'-'}" | ||
DOCKER_TAG_MAJOR=$(echo "$DOCKER_TAG" | cut -d '.' -f 1) | ||
DOCKER_TAG_MAJOR_MINOR=$(echo "$DOCKER_TAG" | cut -d '.' -f 1-2) | ||
IMAGE="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG}" | ||
IMAGE_MAJOR="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG_MAJOR}" | ||
IMAGE_MAJOR_MINOR="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG_MAJOR_MINOR}" | ||
echo "IMAGE=$IMAGE" >>"$GITHUB_ENV" | ||
echo "IMAGE_MAJOR=$IMAGE_MAJOR" >>"$GITHUB_ENV" | ||
echo "IMAGE_MAJOR_MINOR=$IMAGE_MAJOR_MINOR" >>"$GITHUB_ENV" | ||
docker build . \ | ||
--build-arg BUILDKIT_INLINE_CACHE=1 \ | ||
--cache-from $IMAGE \ | ||
--tag $IMAGE | ||
docker tag $IMAGE $IMAGE_MAJOR | ||
docker tag $IMAGE $IMAGE_MAJOR_MINOR | ||
env: | ||
DOCKER_TAG: ${{ inputs.tag || github.ref_name }} | ||
- name: Log in to GHCR | ||
run: >- | ||
echo ${{ secrets.GITHUB_TOKEN }} | | ||
webknjaz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
docker login ghcr.io -u $GITHUB_ACTOR --password-stdin | ||
- name: Push Docker image to GHCR | ||
run: | | ||
docker push $IMAGE | ||
docker push $IMAGE_MAJOR | ||
docker push $IMAGE_MAJOR_MINOR |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -91,15 +91,70 @@ branding: | |
color: yellow | ||
icon: upload-cloud | ||
runs: | ||
using: docker | ||
image: Dockerfile | ||
args: | ||
- ${{ inputs.user }} | ||
- ${{ inputs.password }} | ||
- ${{ inputs.repository-url }} | ||
- ${{ inputs.packages-dir }} | ||
- ${{ inputs.verify-metadata }} | ||
- ${{ inputs.skip-existing }} | ||
- ${{ inputs.verbose }} | ||
- ${{ inputs.print-hash }} | ||
- ${{ inputs.attestations }} | ||
using: composite | ||
steps: | ||
br3ndonland marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- name: Fail-fast in unsupported environments | ||
if: runner.os != 'Linux' | ||
run: | | ||
>&2 echo This action is only able to run under GNU/Linux environments | ||
exit 1 | ||
webknjaz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
shell: bash -eEuo pipefail {0} | ||
- name: Reset path if needed | ||
run: | | ||
# Reset path if needed | ||
webknjaz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# https://github.com/pypa/gh-action-pypi-publish/issues/112 | ||
if [[ $PATH != *"/usr/bin"* ]]; then | ||
echo "\$PATH=$PATH. Resetting \$PATH for GitHub Actions." | ||
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" | ||
echo "PATH=$PATH" >>"$GITHUB_ENV" | ||
echo "$PATH" >>"$GITHUB_PATH" | ||
echo "\$PATH reset. \$PATH=$PATH" | ||
fi | ||
shell: bash | ||
- name: Set repo and ref from which to run Docker container action | ||
id: set-repo-and-ref | ||
run: | | ||
# Set repo and ref from which to run Docker container action | ||
# to handle cases in which `github.action_` context is not set | ||
# https://github.com/actions/runner/issues/2473 | ||
REF=${{ env.ACTION_REF || env.PR_REF || github.ref_name }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @br3ndonland when running in another repo, wouldn't There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If another repo is using the marketplace action ( If the action is being run on a PR to this repo, If the action is being run from a push, release, or other event in this repo, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Alright. Thanks for clarifying. So I'll need to be careful in the release process to make an image before tagging… |
||
REPO=${{ env.ACTION_REPO || env.PR_REPO || github.repository }} | ||
REPO_ID=${{ env.PR_REPO_ID || github.repository_id }} | ||
echo "ref=$REF" >>"$GITHUB_OUTPUT" | ||
echo "repo=$REPO" >>"$GITHUB_OUTPUT" | ||
echo "repo-id=$REPO_ID" >>"$GITHUB_OUTPUT" | ||
shell: bash | ||
env: | ||
ACTION_REF: ${{ github.action_ref }} | ||
ACTION_REPO: ${{ github.action_repository }} | ||
PR_REF: ${{ github.event.pull_request.head.ref }} | ||
PR_REPO: ${{ github.event.pull_request.head.repo.full_name }} | ||
PR_REPO_ID: ${{ github.event.pull_request.base.repo.id }} | ||
- name: Check out action repo | ||
uses: actions/checkout@v4 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @br3ndonland could you remind me — is this necessary because it should be within the work dir? |
||
with: | ||
path: action-repo | ||
ref: ${{ steps.set-repo-and-ref.outputs.ref }} | ||
repository: ${{ steps.set-repo-and-ref.outputs.repo }} | ||
- name: Create Docker container action | ||
run: | | ||
# Create Docker container action | ||
python create-docker-action.py | ||
webknjaz marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry if I'm missing something here: why is creating the container action done dynamically like this? Is there a reason it can't be a static file? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, the reason is that otherwise there's no way to specify the correct Docker tag. Docker actions support pulling in pre-built Docker images by supplying a registry address to the # this works but the image tag can't be customized
runs:
using: docker
image: docker://ghcr.io/pypa/gh-action-pypi-publish:release-v1.8 # this doesn't work because `image:` doesn't support context
runs:
using: docker
image: docker://ghcr.io/pypa/gh-action-pypi-publish:${{ github.action_ref }} The workaround is to switch the top-level Originally this PR proposed to create the Docker container action as a YAML file with a single placeholder field, like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @woodruffw I requested that. I don't want to have another file with hard-coded contents in the repo. There was a template-like thing before. The published container version is being generated dynamically. |
||
env: | ||
REF: ${{ steps.set-repo-and-ref.outputs.ref }} | ||
REPO: ${{ steps.set-repo-and-ref.outputs.repo }} | ||
REPO_ID: ${{ steps.set-repo-and-ref.outputs.repo-id }} | ||
shell: bash | ||
working-directory: action-repo | ||
- name: Run Docker container | ||
uses: ./action-repo/.github/actions/run-docker-container | ||
with: | ||
user: ${{ inputs.user }} | ||
password: ${{ inputs.password }} | ||
repository-url: ${{ inputs.repository-url || inputs.repository_url }} | ||
packages-dir: ${{ inputs.packages-dir || inputs.packages_dir }} | ||
webknjaz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
verify-metadata: ${{ inputs.verify-metadata || inputs.verify_metadata }} | ||
skip-existing: ${{ inputs.skip-existing || inputs.skip_existing }} | ||
verbose: ${{ inputs.verbose }} | ||
print-hash: ${{ inputs.print-hash || inputs.print_hash }} | ||
attestations: ${{ inputs.attestations }} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
import json | ||
import os | ||
import pathlib | ||
|
||
DESCRIPTION = 'description' | ||
REQUIRED = 'required' | ||
|
||
REF = os.environ['REF'] | ||
REPO = os.environ['REPO'] | ||
REPO_ID = os.environ['REPO_ID'] | ||
REPO_ID_GH_ACTION = '178055147' | ||
|
||
|
||
def set_image(ref: str, repo: str, repo_id: str) -> str: | ||
if repo_id == REPO_ID_GH_ACTION: | ||
return '../../../Dockerfile' | ||
docker_ref = ref.replace('/', '-') | ||
return f'docker://ghcr.io/{repo}:{docker_ref}' | ||
|
||
|
||
image = set_image(REF, REPO, REPO_ID) | ||
|
||
action = { | ||
'name': '🏃', | ||
DESCRIPTION: ( | ||
'Run Docker container to upload Python distribution packages to PyPI' | ||
), | ||
'inputs': { | ||
webknjaz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
'user': {DESCRIPTION: 'PyPI user', REQUIRED: False}, | ||
'password': { | ||
DESCRIPTION: 'Password for your PyPI user or an access token', | ||
REQUIRED: False, | ||
}, | ||
'repository-url': { | ||
DESCRIPTION: 'The repository URL to use', | ||
REQUIRED: False, | ||
}, | ||
'packages-dir': { | ||
DESCRIPTION: 'The target directory for distribution', | ||
REQUIRED: False, | ||
}, | ||
'verify-metadata': { | ||
DESCRIPTION: 'Check metadata before uploading', | ||
REQUIRED: False, | ||
}, | ||
'skip-existing': { | ||
DESCRIPTION: ( | ||
'Do not fail if a Python package distribution' | ||
' exists in the target package index' | ||
), | ||
REQUIRED: False, | ||
}, | ||
'verbose': {DESCRIPTION: 'Show verbose output.', REQUIRED: False}, | ||
'print-hash': { | ||
DESCRIPTION: 'Show hash values of files to be uploaded', | ||
REQUIRED: False, | ||
}, | ||
'attestations': { | ||
DESCRIPTION: ( | ||
'[EXPERIMENTAL]' | ||
' Enable experimental support for PEP 740 attestations.' | ||
' Only works with PyPI and TestPyPI via Trusted Publishing.' | ||
), | ||
REQUIRED: False, | ||
}, | ||
}, | ||
'runs': { | ||
'using': 'docker', | ||
'image': image, | ||
}, | ||
} | ||
|
||
action_path = pathlib.Path('.github/actions/run-docker-container/action.yml') | ||
action_path.parent.mkdir(parents=True, exist_ok=True) | ||
action_path.write_text(json.dumps(action, ensure_ascii=False), encoding='utf-8') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we incorporate the prior art from the discussions in #45 and accept the 3-segment version as an input, then deduce everything else from that + push the Git tag and advance the proper branches? Perhaps, this could also happen before this PR so it exists as a separate base.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand what you're asking here. You want separate inputs for major, minor, and patch versions? What if you're building the Docker image from the
release/v1
branch? What is the "3-segment version" in that case?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@br3ndonland I was talking about the workflow inputs. Everything else should be computed from them.
The current workflow for me is as follows:
v1.10.1
on top ofunstable/v1
release/v1
tov1.10.1
release/v1.10
tov1.10.1
(or create it)git push --atomic
everything in one goWith this, my 3-segment “input” is
1.10.1
andv1
+v1.10
are being extracted from that. Do you think we could bake containers into this flow with not many changes? This will involve tagging the same image with multiple tags, as I understand. And the end-users referencing versions by both tags and branches should be able to retrieve whatever container there is, right?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@webknjaz thanks for explaining that. I think we're actually pretty close to the desired workflow. There's just a small update needed to the Docker build workflow so it pushes tags with the major and minor version numbers. I've pushed that change.
Your updated workflow would look like this:
v1.10.1
on top ofunstable/v1
release/v1
tov1.10.1
release/v1.10
tov1.10.1
(or create it)git push --atomic
everything in one goghcr.io/pypa/gh-action-pypi-publish:v1.10.1
ghcr.io/pypa/gh-action-pypi-publish:v1.10
ghcr.io/pypa/gh-action-pypi-publish:v1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@br3ndonland so with #45, I wanted to move all these steps inside the automation. For publishing Python projects, I usually use a workflow with
workflow_dispatch
that allows me to type in the desired version to release and that workflow creates the git tag, the gh release, signs stuff and so on.The idea is that tag+branches+gh-release represent the result of release automation being successful and aren't triggers or manual actions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So... you want the Docker build workflow to run automatically when a tag is created? Or when a GitHub Release is created?
Please be more specific here, and please limit the scope of your requests. The goal of this PR is "Build Docker image and push to GHCR." The goal is not to develop your entire release workflow.
PR #45 is currently a draft and I don't think I should have to make this PR dependent on some other draft PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No to both. In my release processes, tags + GH Releases are the results of publishing, not the triggers. So when I trigger
workflow_dispatch
(which is a button click on GH UI + the desired version entered into the input field), the workflows create what I want to publish, send that to the primary platform (PyPI in case of Python projects or a container registry in the context of this specific PR), and only after that the same workflow creates the Git tag and the GH Release.I only pointed to that PR as this one seems to be touching the same topic partially. I was thinking that since you're already making a workflow that publishes the container, that could piggyback on top of this effort. I wasn't thinking of it as a dependency but rather as a replacement, but wanted to point to it for the pre-existing context around what I wanted to implement for quite a while.
This PR is attempting to change my release workflow already, after all. So I wanted to make sure it doesn't go in the direction that would complicate things, given that I already envision the release process to develop in a certain way.
If it were possible to merge the PR and keep the workflow where the Git tag is pushed, and it works right away without a container published for a while, it'd be where we could scope it. However, AFAIU, this will no longer work after merging since when there's a tag and the end-users start using it, the action will attempt pulling the image that is not yet there. There's a race condition in this process, meaning that it is prone to human error and needs to be handled carefully. And that's why I was thinking that it'd be good to integrate all the other things in a way that would keep the process robust.
P.S. Would it be useful to still give the end-users some way to force building the container instead of pulling the cached one?