Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from CircleCI runners to Github actions #907

Merged
merged 4 commits into from
Oct 2, 2024

Conversation

almet
Copy link
Contributor

@almet almet commented Sep 2, 2024

This is still pretty much a WIP. I'm pushing it here mainly to see how the CI is reacting.

Fixes #674

@almet almet force-pushed the 674-circleci-to-gh-actions branch 2 times, most recently from 3a9bb59 to 1454b39 Compare September 17, 2024 14:54
@almet
Copy link
Contributor Author

almet commented Sep 17, 2024

Here are the changes I did in this PR, which is up for discussion:

"dev" environments (dangerzone.rocks/build/distro-version) are now built daily, and pushed to the github container registry, so that the CI can reuse them during the day, without having to build them.

I'm currently using the ghcr, but I'm wondering if using a cache instead would be more beneficial, because we could invalidate the cache more easily, and rebuild the dev images on the fly if needed.

The dangerzone container image is also built daily, cached and uploaded to the ghcr.

"end user" (dangerzone.rocks/distro-version) images are currently not uploaded because they require a .deb or .rpm to be built, which is part of the CI pipeline.

While doing these changes, the dangerzone.rocks prefix shown to be cumbersome, as it is taking the place of the container registry. As a result, I now changed this prefix to dangerzone instead, only for the "dev" and "end-user" images.

We might want to follow the same lead for the main container image dangerzone.rocks/dangerzone.

With these changes, the CI is now taking ~9mn to complete all the tests! 🐎

@almet almet marked this pull request as ready for review September 17, 2024 16:09
@almet
Copy link
Contributor Author

almet commented Sep 17, 2024

I'm currently using the ghcr for the dev images, but I'm leaning towards getting it (almost) out of the picture, as I'm not sure it makes more sense than using a cache with podman save / podman load -i.

This last option will make it easier to invalidate the cache if for instance the dependencies change. But maybe I'm missing the main point why you wanted to have a look at the ghcr in the first place @apyrgio ?

@apyrgio
Copy link
Contributor

apyrgio commented Sep 17, 2024

Damn, super exciting work Alexis! I'll dig into it soon.


On the subject of caching, I'm wondering how is the invalidation story different for caches, instead of container images? I'm definitely missing context here since I have theorized about this working, but I didn't implement it as you did.

In any case, I suggested pushing to a container registry instead of creating a tarball for the following reasons:

  1. We can re-use these container images in other CI actions (think apt-tools-prod / yum-tools-prod). We can even give them a check locally in case of a CI error, without waiting for image rebuilds.
  2. Pushing the Dangerzone container image to GHCR may help with independent container updates.
  3. It will speed up image uploads/downloads, in case some layers are already present (think the base Debian/Fedora layers, and maybe some extra ones if only a later portion of the Dockerfile changes)."
  4. Artifacts like Tesseract data (soon to come) won't be evicted in case of lots of dev container images.

That's mostly it, but I'd like to know your experience so far.

.github/workflows/ci.yml Outdated Show resolved Hide resolved
@almet almet changed the title WIP: Migrate to Github Actions Migrate to Github Actions Sep 18, 2024
@almet almet force-pushed the 674-circleci-to-gh-actions branch 4 times, most recently from bd9b500 to f3197f5 Compare September 19, 2024 13:50
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/build.yml Outdated Show resolved Hide resolved
.github/workflows/build.yml Outdated Show resolved Hide resolved
dev_scripts/env.py Outdated Show resolved Hide resolved
.github/workflows/build.yml Show resolved Hide resolved
dev_scripts/env.py Outdated Show resolved Hide resolved
dev_scripts/env.py Outdated Show resolved Hide resolved
dev_scripts/env.py Outdated Show resolved Hide resolved
dev_scripts/env.py Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
@almet almet force-pushed the 674-circleci-to-gh-actions branch from f3d6f0b to 6ac7892 Compare October 2, 2024 12:24
@almet almet force-pushed the 674-circleci-to-gh-actions branch from 09aaee2 to e177c8d Compare October 2, 2024 14:41
@almet almet changed the title Migrate to Github Actions Switch from CircleCI runners to Github actions. Oct 2, 2024
almet and others added 4 commits October 2, 2024 16:47
As part of this change, the dev (build) and end-user test images names
changed from `dangerzone.rocks/*` to `ghcr.io`.

A new `--sync` option is provided in the `env.py` command, in order to
retrieve the images from the registry, or build and upload otherwise.
Add a dummy entry in debian/changelog, to signal that the latest
Dangerzone version is 0.7.1.
Make sure that the Debian package we build conforms to the expected
naming scheme else, it's possible that something is off. A scenario
we've encountered is bumping `share/version.txt`, but not
`debian/changelog`, which would create a Debian package with an older
version.
@almet almet force-pushed the 674-circleci-to-gh-actions branch from e177c8d to a001b54 Compare October 2, 2024 14:49
@almet almet changed the title Switch from CircleCI runners to Github actions. Switch from CircleCI runners to Github actions Oct 2, 2024
@almet almet merged commit a001b54 into main Oct 2, 2024
69 of 75 checks passed
@almet almet deleted the 674-circleci-to-gh-actions branch October 2, 2024 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate from CircleCI to GitHub actions
3 participants