[Devtools Week] Investigate usage of renovate to keep registry image up-to-date #1126

thepetk · 2023-05-25T18:10:19Z

Which area/kind this issue is related to?

/area registry

Issue Description

In order to be sure that we the images used from registry stacks are always up-to-date we can investigate a tool like renovate.
More details for renovate here: https://github.com/renovatebot/renovate

Target Odo version

Target Odo version:

thepetk · 2023-05-28T17:29:05Z

I've added a renovate bot for my fork, following this tutorial. Investigating usage for the bot with this config: https://github.com/thepetk/registry/blob/main/renovate.json

thepetk · 2023-06-05T19:49:20Z

I'm currently investigating the correct configuration of renovate. A first result can be found here: https://github.com/thepetk/registry/pulls

Notes:

We cannot update an image that has the latest tag, so it might be better to first merge the (Update stacks that reference images with the latest tag #849 -> which is actually blocked from Registry validate stacks test failing on the java-quarkus stack #1115).
We need to define:
1. Schedule for those runs (e.g once a day every afternoon)
2. The quantity of the PRs per time (for example, we may only allow 5 PRs max per run, as it might be overwhelming for the
  person responsible to monitor)
3. Who will be monitoring the opened PRs. Do we need any assignees on them? Ping someone?
We might also need to decide how we will host the bot (most probably we can rely on the mend environment, but it could be also useful to host it ourselves?)

thepetk · 2023-06-05T20:28:37Z

I've listed all images that cannot be updated due to invalid tags:

image	Version
icr.io/appcafe/websphere-liberty-devfile-stack	{{liberty-version}}-gradle
icr.io/appcafe/websphere-liberty-devfile-stack	{{liberty-version}}
icr.io/appcafe/open-liberty-devfile-stack	{{liberty-version}}-gradle
icr.io/appcafe/open-liberty-devfile-stack	{{liberty-version}}
registry.access.redhat.com/ubi8/openjdk-11	latest
quay.io/eclipse/che-java11-maven	next
registry.access.redhat.com/ubi8/nodejs-16	latest
quay.io/devfile/universal-developer-image	latest

thepetk · 2023-06-05T20:41:18Z

We might also need to decide how we will host the bot (most probably we can rely on the mend environment, but it could be also useful to host it ourselves?)

I've checked other configurations of renovate inside redhat-developer or redhat-appstudio and they are using pretty much the same setup with https://github.com/thepetk/registry/blob/main/renovate.json. So I don't see a need of self hosting etc.

michael-valdron · 2023-06-05T21:53:03Z

I've listed all images that cannot be updated due to invalid tags:

image Version

icr.io/appcafe/websphere-liberty-devfile-stack {{liberty-version}}-gradle

icr.io/appcafe/websphere-liberty-devfile-stack {{liberty-version}}

icr.io/appcafe/open-liberty-devfile-stack {{liberty-version}}-gradle

icr.io/appcafe/open-liberty-devfile-stack {{liberty-version}}

registry.access.redhat.com/ubi8/openjdk-11 latest

quay.io/eclipse/che-java11-maven next

registry.access.redhat.com/ubi8/nodejs-16 latest

quay.io/devfile/universal-developer-image latest

There might be a PR coming to change registry.access.redhat.com/ubi8/nodejs-16:latest into a fixed version tag.

thepetk · 2023-06-06T11:36:12Z

@michael-valdron I've tried to add fixed values to all of the above images in this commit [73febe1b7bb750ba7fbc934db55873ab2e33b3f3] (thepetk/registry@73febe1)
& tried to trigger update by putting in purpose an older image.
I'll try to monitor in case we fix those values if renovate would be in position to keep them up-to-date

thepetk · 2023-06-06T11:50:48Z

From the above images, if we fix the tag in all of them I see:

The only image not able to be updated I see is quay.io/devfile/universal-developer-image:

DEBUG: Dependency quay.io/devfile/universal-developer-image has unsupported/unversioned value ubi8-5e2d33a (versioning=docker)

thepetk · 2023-06-06T11:56:28Z

I'll now check if there is a way to fix the regex manager and checking as tag the variable field for open-liberty cases. This way we will keep having a variable to define the open-liberty version

thepetk · 2023-06-06T14:11:54Z

For using it on samples:

We want to have another renovate bot inside our org for devfile samples.
The case here is pretty simple. We will only enable dockerfile manager in the configuration and we will only keep updated the tags inside dockerfiles (i think)

thepetk · 2023-06-07T13:52:40Z

Invesigating the update of github-actions (following documentation here). The trickiest part if we want to include renovate to all the repos will be to combine the configuration for devfile with the configuration for github-actions

thepetk · 2023-06-07T14:41:30Z

Invesigating the update of github-actions (following documentation here). The trickiest part if we want to include renovate to all the repos will be to combine the configuration for devfile with the configuration for github-actions

I see is pretty simple to combine the two actions (devfile images & github-actions).

Github Actions & Renovate
I've gone through registry and applied the config to update github actions, while also pining the digest to the action.

Results:
As we want to first pin automatically all digest there is a PR created here:

Pin dependencies - autoclosed thepetk/registry#38 > pins digest instead of version

For specific action updates we have:

As we are not using digest, the first PR is dedicated to pin digest everywhere so as a first time it might conflicts with the updates. Now looking for actions that we weren't able to update

thepetk · 2023-06-09T10:22:34Z

FInal thing to investigate before opening a PR, is if the usage of packageRules will improve the configuration of renovate

thepetk · 2023-06-09T13:17:36Z

As a result of this investigation issue, I'd say that we should add a renovate bot in our repos. Moreover:

Registry Images Update

For images mentioned inside devfiles, I've tried using a custom regexManager which only checks for dependency updates inside a devfile.y[a]ml file and tries to find an image : string to check for any dependency updates.

Every new update has its own PR, as it might need different approach on testing etc.

Note, that renovate cannot update tags that have latest or next as value. The value has to be a valid version. As a result some of the images inside stacks have to be updated and use fixed tags. For the record, during investigation I have tried to fix the version in my fork and everything was being updated correctly.

Some of the PRs we will see on the first run right now on registry would be:

Github Actions Update

For github actions I've introduced the github-actions manager and I've included all these updates into the same group as a packageRule (it helps reduce the noise from many different PRs opened). This way we will see each time only one PR opened. One more addition is to extend the renovate.json file to use the helpers:pinGitHubActionDigests in order to use digest instead of versions.

An example of github actions PR would be:

Update github actions (major) thepetk/registry#51

Configuration used for Investigation

This is the proposed configuration for renovate. After bot installation, the bot will create a PR called add renovate so I'd propose to update this PR with the below configuration:

{
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "extends": [
      "config:base",
    // Using this helper in order to pin digest of github action
      "helpers:pinGitHubActionDigests"
    ],
    // Helps the bot only to run these managers and skip all others
    "enabledManagers": ["regex", "github-actions"],
    "regexManagers": [
      {
        // Using regex manager to define a custom way to fetch devfile image tags
        "fileMatch": [
          "devfile.*y[a]?ml$"
        ],
        "matchStrings": [
          "image: (?<depName>\\S*):(?<currentValue>\\S*)"
        ],
        "datasourceTemplate": "docker",
        "depNameTemplate": "{{{depName}}}"
      }
    ],
    "packageRules": [
        // Grouping github-actions in order to reduce renovate PR noise
        {
          "matchManagers": ["github-actions"],
          "groupName": "github actions",
          "groupSlug": "github-actions",
          "commitMessageTopic": "{{depName}}"
        }
    ],
    // Ignoring these paths in order to reduce PR noise
    "ignorePaths": [
      "**/docker/**",
      ".ci/**",
      "tests/**"
    ]
}

Configuration of Renovate

As I've seen in other issues inside redhat-developer and redhat-appstudio orgs, we are not self hosting the renovate bot and most of the configuration files don't have a schedule defined. Some points of interest could be:

If a PR is closed, in the next run renovate will take this PR as ignored, so it will not open one. It will open a new one only if the dependency has a new update. For example if you close Update docker to 2.1.0 it will open a PR only if it finds a version 2.1.1
Do we need a specific schedule for renovate runs?
Do we need a limit for PR created by the bot?

Automatic Alizer Update

For alizer, as it is a go module used by another application we can follow this docs and using a packageRule for alizer only.

Next steps

As a next step for renovate bot we should add the renovate bot to devfiles organization (if we want to include samples it should be added to this org too). For HAS the renovate bot is already running on its organization on github so we will just have to add a renovate.json file (for github-actions).

The bot creation process requires a specific level of access to the org and is pretty simple following this guide.

thepetk · 2023-06-13T13:28:16Z

I've created a new issue, dedicated only to next steps sections (#1133)

thepetk self-assigned this May 25, 2023

openshift-ci bot added the area/registry Devfile registry for stacks and infrastructure label May 25, 2023

thepetk changed the title ~~Investigate usage of renovate to keep registry image up-to-date~~ [Devtools Week] Investigate usage of renovate to keep registry image up-to-date Jun 5, 2023

thepetk mentioned this issue Jun 13, 2023

Add renovate to devfiles registry #1133

Closed

4 tasks

thepetk closed this as completed Jun 13, 2023

thepetk added this to Devfile Project Jun 13, 2023

thepetk moved this to Done ✅ in Devfile Project Jun 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Devtools Week] Investigate usage of renovate to keep registry image up-to-date #1126

[Devtools Week] Investigate usage of renovate to keep registry image up-to-date #1126

thepetk commented May 25, 2023

thepetk commented May 28, 2023

thepetk commented Jun 5, 2023 •

edited

Loading

thepetk commented Jun 5, 2023

thepetk commented Jun 5, 2023

michael-valdron commented Jun 5, 2023

thepetk commented Jun 6, 2023 •

edited

Loading

thepetk commented Jun 6, 2023 •

edited

Loading

thepetk commented Jun 6, 2023 •

edited

Loading

thepetk commented Jun 6, 2023

thepetk commented Jun 7, 2023

thepetk commented Jun 7, 2023 •

edited

Loading

thepetk commented Jun 9, 2023

thepetk commented Jun 9, 2023 •

edited

Loading

thepetk commented Jun 13, 2023

[Devtools Week] Investigate usage of renovate to keep registry image up-to-date #1126

[Devtools Week] Investigate usage of renovate to keep registry image up-to-date #1126

Comments

thepetk commented May 25, 2023

Which area/kind this issue is related to?

Issue Description

Target Odo version

thepetk commented May 28, 2023

thepetk commented Jun 5, 2023 • edited Loading

thepetk commented Jun 5, 2023

thepetk commented Jun 5, 2023

michael-valdron commented Jun 5, 2023

thepetk commented Jun 6, 2023 • edited Loading

thepetk commented Jun 6, 2023 • edited Loading

thepetk commented Jun 6, 2023 • edited Loading

thepetk commented Jun 6, 2023

thepetk commented Jun 7, 2023

thepetk commented Jun 7, 2023 • edited Loading

thepetk commented Jun 9, 2023

thepetk commented Jun 9, 2023 • edited Loading

Registry Images Update

Github Actions Update

Configuration used for Investigation

Configuration of Renovate

Automatic Alizer Update

Next steps

thepetk commented Jun 13, 2023

thepetk commented Jun 5, 2023 •

edited

Loading

thepetk commented Jun 6, 2023 •

edited

Loading

thepetk commented Jun 6, 2023 •

edited

Loading

thepetk commented Jun 6, 2023 •

edited

Loading

thepetk commented Jun 7, 2023 •

edited

Loading

thepetk commented Jun 9, 2023 •

edited

Loading