Publish acceptance-test-harness docker image on release

[timja]

Service(s)

infra.ci.jenkins.io

Summary

see https://github.com/jenkinsci/acceptance-test-harness/blob/master/src/main/resources/ath-container/Dockerfile
and jenkinsci/acceptance-test-harness#875

What would be the best way to get this publishing automatically?

Reproduction steps

No response

[dduportal]

Proposal:

• Creates a Docker repository (in DockerHub) named jenkinsciinfra/ath - use jenkins/ath
• Adding a file Jenkinsfile.infra.ci.jenkins.io that defines the "lint/build/(test)/deploy on main/tag" process in this repository
• Add the repository in infra.ci's "Docker Images" folder (might require a GH app update to allow cloning/webhooks/checks)
• Eventually add the updatecli process to bump maven/JDK/etc. dependencies

WDYT @timja ? we'll discuss this issue in the next infra weekly meeting

[timja]

Why in jenkinsciinfra? (probably just missing context that I should have included)

It already exists in the jenkins org, see https://hub.docker.com/r/jenkins/ath

Apart from that sounds good.

[dduportal]

Oh, then easier to have the same name then. Thanks for the pointer!

[dduportal]

Created jenkins-infra/kubernetes-management#2730 as first step to have infra.ci scanning the repository.

Ping @timja , as I'm not admin on the jenkinsci org., we need a GitHub app (ID + private key) to allow infra.ci to authenticate.

[lemeurherve]

Created jenkins-infra/kubernetes-management#2730 as first step to have infra.ci scanning the repository.

Ping @timja , as I'm not admin on the jenkinsci org., we need a GitHub app (ID + private key) to allow infra.ci to authenticate.

@dduportal we can create the GitHub App and ask to install it on jenkinsci org/repositories, @timja will only have to accept the installation.

[dduportal]

Created jenkins-infra/kubernetes-management#2730 as first step to have infra.ci scanning the repository.
Ping @timja , as I'm not admin on the jenkinsci org., we need a GitHub app (ID + private key) to allow infra.ci to authenticate.

@dduportal we can create the GitHub App and ask to install it on jenkinsci org/repositories, @timja will only have to accept the installation.

So I might be missing something but I can't find a way to install the "infra.ci.jenkins.io" GitHub application on the jenkinsci organization.
This app is already installed in the jenkins-infra organization to allow the controller infra.ci.jenkins.io to scan/checkout/add checks the GitHub API for these repositories. I thought that we could install an application on different organizations (to make it easier to keep the same set of permissions, only repository list would be different between installtions).

@timja I'm adding a new GitHub app in the jenkinsci that will have the same name and settings (permissions) to target the ath repository. That would be a duplicate but that would allow to go forward. If there is a way to avoid duplication, please let us know and we'll change the credentials even afterward.

[dduportal]

• App created: https://github.com/organizations/jenkinsci/settings/apps/infra-ci-jenkins-io-org-jenkinsci , need to install it on jenkinsci. Followed https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/github-app-auth#_creating_the_github_app for the permissions.
• I'm adding the (converted) secrets in the SOPS system so that feat(infra.ci) add job definition to build the jenkinsci ATH Docker image kubernetes-management#2730 can be merged.

N.B. found why we can't install the current "infra.ci" somewhere else: there is the following option only available when creating the app:

[timja]

Yeah check that button and you'll be able to install it

[timja]

(I've installed the other one in the meantime)

[dduportal]

Yeah check that button and you'll be able to install it

I can't find the option on an existing app. It seems it's only on the creation form.
Might be worth it to create a brand new one for both cases (that would rotate credentials at the same time) ;)

[timja]

I can't find the option on an existing app. It seems it's only on the creation form.

It's on the advanced page, e.g. https://github.com/organizations/jenkinsci/settings/apps/jenkins-incrementals-publisher/advanced

(Make public)

[dduportal]

@timja tested the job configuration manually and it worked (with jenkins-infra/kubernetes-management#2737 applied).

Once merged, I'll open a PR in the repo to start building the image.

[dduportal]

First step: jenkinsci/acceptance-test-harness#905

[dduportal]

Checking pull request [#905](https://github.com/jenkinsci/acceptance-test-harness/pull/905)

  Found [#905](https://github.com/jenkinsci/acceptance-test-harness/pull/905). has no labels 

  Has no labels. Includes this pull request.
    (not from a trusted source)
      ‘Jenkinsfile.infra.ci.jenkins.io’ not found
    Does not meet criteria

@timja Can I ask you to set me developer for the acceptance-test-harness so that I can die-and-retry the build part in jenkinsci/acceptance-test-harness#905 ?

[NotMyFault]

Added you.

There's a dockerhub-admins team that was already added to the repository, we may should add infra people to that team, for future cases.

[basil]

I verified that docker pull jenkins/ath:latest pulls the freshly built image. Thank you! However, I expected a 1.118 or acceptance-test-harness-1.118 (consistent with historical precedent) Docker image tag corresponding to the acceptance-test-harness-1.118 Git tag. The reason for this is that https://github.com/jenkinsci/jenkins/blob/1e56822e62803eeedf84911201ddd548a8544850/essentials.yml#L4-L5 defines both an athRevision (currently 1.115, which is almost the latest version) and an athImage (currently 1.97-pre, which is a very old version that only works with an athRevision of 1.115 in the subset of tests we run in core by accident), the latter of which must be able to run tests for the former. While the latest image likely satisfies this condition, I do not believe this is guaranteed in the general case. In any case we want to be able to pin to a specific version in core to for deterministic builds.

[jglick]

Would suggest switching to CD (jenkins-infra/repository-permissions-updater#2714) in which case you do not need to deal with tags.

[timja]

Would suggest switching to CD (jenkins-infra/repository-permissions-updater#2714) in which case you do not need to deal with tags.

?

It uses CD, Basil was just referring to an old tag I assume

[basil]

Right, core's current athRevision of 1.115 is (as I wrote) almost the latest version (which is 5414.v5b_9d1cd476cd). The point is that for deterministic builds, both the ATH revision and the Docker image must be compatible, and this is not guaranteed when using a fixed athRevision but the latest Docker image. So there is a need for tagging of the Docker image for each version (as was done in the past).

[dduportal]

Right, core's current athRevision of 1.115 is (as I wrote) almost the latest version (which is 5414.v5b_9d1cd476cd). The point is that for deterministic builds, both the ATH revision and the Docker image must be compatible, and this is not guaranteed when using a fixed athRevision but the latest Docker image. So there is a need for tagging of the Docker image for each version (as was done in the past).

Thanks for the summary: we are in agreement on this target.

The issue is not closed because only the latest tag is currently published, scanning tags is the next step (did not want to start a build wave before validating the main feature).

[dduportal]

Ping @timja @basil , can you create a new tag on the ath repository?

I've double checked: the job is configured to discover (git) tags and to publish the Docker image with the (Docker) tag associated.

Existing tags were not built because they were created prior to the PR introducing the Docker image build (so the MB job does not pick it).

Please note that our default setup also prevent automatically building tags older than 3 days to avoid build waves when scanning repository.

[timja]

needs the master branch to pass, I think it flaked on https://ci.jenkins.io/blue/organizations/jenkins/Core%2Facceptance-test-harness/detail/master/567/tests

[dduportal]

Gotcha. I’m keeping this issue open and i’ll watch

[timja]

https://github.com/jenkinsci/acceptance-test-harness/releases/tag/5417.v87b_d3db_92981

[dduportal]

Checking tags...

  Getting remote tag 5417.v87b_d3db_92981...

    Checking tag [5417.v87b_d3db_92981](https://github.com/jenkinsci/acceptance-test-harness/tree/5417.v87b_d3db_92981)
      ‘Jenkinsfile.infra.ci.jenkins.io’ found
    Met criteria
No automatic build triggered for 5417.v87b_d3db_92981

=> tag is picked up. I'm not sure why it is not triggered though (I assume I mis-configured something).

[timja]

Suppress log output in checks is enabled as well, which doesn't seem ideal for a public build?

[dduportal]

Checking tags...

  Getting remote tag 5417.v87b_d3db_92981...

    Checking tag [5417.v87b_d3db_92981](https://github.com/jenkinsci/acceptance-test-harness/tree/5417.v87b_d3db_92981)
      ‘Jenkinsfile.infra.ci.jenkins.io’ found
    Met criteria
No automatic build triggered for 5417.v87b_d3db_92981

=> tag is picked up. I'm not sure why it is not triggered though (I assume I mis-configured something).

Ok, got it: the tag has a timestamp when the Draft release was created. If you look at the GH release, the artifacts are dated "5 days ago". Since the job policy says "no build of tags older than 3 days ago", then it explains why the build was not triggered.

We had this issue with our own "CD" system for the infra and we solved by by annotating tag (so that the timestamp is changed when the release is triggered) (ref. https://github.com/jenkins-infra/pipeline-library/blob/master/vars/buildDockerAndPublishImage.groovy#L187).

Not sure how to do it here:

• I assume that the tag creation is either CD process, or a manual publication of the release drafter's "draft" version?
• If we removes the job policy forbidding to build older tags, then infra.ci will rebuild and publish ALL the ath tags (that would be bad...)

[dduportal]

Suppress log output in checks is enabled as well, which doesn't seem ideal for a public build?

Gotcha, it's a leftover or me not understanding properly gh status API. Gotta fix it.

[dduportal]

Mmmh that would mean annotating the tag there: https://github.com/jenkins-infra/jenkins-maven-cd-action/blob/master/run.sh#L12.

[jglick]

I am not very familiar with git tag -a, but would it suffice to define a workflow that runs on tagging events?

[jglick]

(or release creation events)

[dduportal]

@jglick Let me open an issue describing a bit more in the reusable action for CD, but https://github.com/orgs/community/discussions/4924

[dduportal]

but would it suffice to define a workflow that runs on tagging events?

If this workflow has a the correct GITHUB_TOKEN to ensure that it can git tag -a <..> && git push origin <..>

[dduportal]

• The Docker image jenkins/ath:5417.v87b_d3db_92981 was built successfully manually 2 days ago so it can be used in the ATH pinning
• Use annotated tags when releasing jenkins-maven-cd-action#25 was merged. As soon as a new release of the ATH is rolled up, we should see a new Docker image published with the new tag: we'll close this issue if it's ok

[timja]

As of merging jenkins-infra/github-reusable-workflows#16 and releasing the PR, we should now have annotated tags on new releases

[basil]

Thank you very much! As a result of these efforts, core is back on track with the latest version of ATH in jenkinsci/jenkins#7099 🎉

[dduportal]

Closing the issue:

• https://github.com/jenkinsci/acceptance-test-harness/releases/tag/5439.vb_929a_7e53585 was released 3 days ago
• The jenkins/ath:5439.vb_929a_7e53585 (https://hub.docker.com/layers/jenkins/ath/5439.vb_929a_7e53585/images/sha256-b70d0dd765bf4050216941caa58e70711f85ac15ff70fe32c101b8a879a4b1fb?context=explore) was deployed right after.