Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add actions to build and push images to Docker Hub #73

Merged
merged 21 commits into from
Nov 25, 2024
Merged

Conversation

amercader
Copy link
Member

@amercader amercader commented Jul 2, 2024

As discussed in ckan/ckan#8143, we want to automate the publication of artifacts as much as possible, in this case the building and pushing of CKAN Docker images to Docker hub.

This PR adds a new reusable workflow that is call for each CKAN version to build.

The actions work but there's stuff we need to decide before merging.

  1. The biggest question is when or how to trigger a build/push, right now these are triggered on every push to this repo, but this is not what we want. Some options:
  • When we push some tag(s)
  • When there's a new release created
  • When we add some labels to a Pull request
  1. There are CKAN tags hardcoded in the worflow files, so on every patch release we would need to update these files. There's probably a way to centralize these, reading them from an env var or a file in the repo

As discussed in ckan/ckan#8143

TODO: Decide when to trigger the builds
@wardi
Copy link
Contributor

wardi commented Jul 2, 2024

is it better to reference the latest ckan-X.Y image instead of a specific point release? IIUC the docker registry lets us update these links, users can get the latest by building and we don't need to update the tags inside the repo on every ckan-docker-base release.

@kowh-ai
Copy link
Contributor

kowh-ai commented Jul 8, 2024

Maybe once a new release has been pushed to DockerHub ie: 2.10.4 or 2.11.0 this remains ‘untouched’ as an image and can be used as a “point in time” release. Subsequent builds of, for instance 2.10 (which actually would be 2.10.4) could be tagged as 2.10.latest. These could be generated weekly.

It's not clear we are using this, and they were installing a system-wide
packaging package that could not be uninstalled when installing CKAN
requirements:

error: uninstall-distutils-installed-package

Cannot uninstall packaging 21.3
╰─> It is a distutils installed project and thus we cannot accurately...
Three main focuses:

* Consolidate version handling. We had version numbers all over the
  place and many different variables with slightly different meanings.
  Now we only have the version defined in one place (VERSION.txt), and
  the vars have been consolidated (Dockerfiles only use CKAN_REF, and
  the ones in the shell script should be hopefully named as what they
  are)

* Centralize building (and pushing). Rather than 8 different Makefiles
  mostly identical we now only have one bash script at the root of the
  file which builds the relevant images according to the passed
  parameters, e.g.:

  ```
  ./ckan-docker-base.sh build 2.11

  ./ckan-docker-base.sh build master
  ./ckan-docker-base.sh build 2.10 base
  ./ckan-docker-base.sh build 2.0 dev
  ```

  It also centralizes commands for pushing the images to DockerHub, but
  that will be soon done via GitHub Actions so it's discouraged.

* Reduce number of Dockerfiles. While we still keep separate images for
  prod and dev environments, they are all now built from the same
  Dockerfile using staged builds and the Buildkit ability to just build
  the stages needed to build the final image (so buildkit must be
  enabled with `DOCKER_BUILDKIT=1` when running `docker build`)
@amercader
Copy link
Member Author

@kowh-ai @wardi This is now ready for review. The latest changes to simplify the repo made this much easier and the workflows are pretty minimal and shouldn't require updates (e.g. version numbers) going forward.
The bulk of new logic is to generate the right image tags depending on what version we are building :)

Right now publishing will be triggered when a new release is "released". According to the docs this means that:

A release was published, or a pre-release was changed to a release

Seems like a good event to trigger it, as we can create drafts without publishing the images yet

@amercader amercader marked this pull request as ready for review November 22, 2024 13:22
Copy link
Contributor

@kowh-ai kowh-ai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @amercader - it's nice and lean

@kowh-ai kowh-ai merged commit ab6e040 into main Nov 25, 2024
6 checks passed
@kowh-ai kowh-ai deleted the build-push-actions branch November 25, 2024 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants