Use a non-alpha tagged defaultBaseImage image for ko builds

[skaegi]

Tekton is currently using cgr.dev/chainguard/static as our defaultBaseImage. This recently has caused us problems as apparently in the 0.46 Tekton release the /etc/alpine-release is set to 3.18_alpha20230208 which is leading out security team to think this image contains alpha packages.

We are in a really awkward situation here as we cannot even rebuild easily as apparently the tags are not available outside of an agreement with Chainguard so without deep thought can only rebuild with "latest".

My ask here is...

1. Can we work with Chainguard so for "static" Tekton (and anyone) can use tags and stop use of "latest" without an agreement.
2. Can we use a tag that is non-alpha

[afrittoli]

Thanks @skaegi - I agree we should fix this - I wonder if there is any special reason for pulling in an alpha release of alpine? /cc @wlynch

[imjasonh]

Hey, thanks for raising this. Sorry I didn't see this earlier. I'm not sure why the /etc/alpine-release points to alpha, I'll look into that.

Secondly, there's currently no need for an agreement between Chainguard and Tekton to pull any available tag of static. There are currently 500+ tags for cgr.dev/chainguard/static, all of which should be accessible to anybody. I'm not sure where you got the impression that they weren't.

If you can identify a date that this last worked for you, you can pin to cgr.dev/chainguard/static:latest-20230301 (or whatever date) and that should work fine. You can also pin to any previous digest, if a previous digest worked for you.

I'll look into when alpine-release changed, and in the meantime you should be able to pin to an older version to unblock releases.

[imjasonh]

It looks like the change was made some time in early Feb:

# Cat file in image
function ccat() {
  crane export $1 --platform ${3:-linux/amd64} - | tar -Oxf - $2
}

$ ccat cgr.dev/chainguard/static:latest-20230207 etc/alpine-release
3.17.0
$ ccat cgr.dev/chainguard/static:latest-20230208 etc/alpine-release
3.18_alpha20230208

This happens because cgr.dev/chainguard/static tracks Alpine's edge repository (configured here), and I suspect around Feb 7/8 they cut a new release branch for 3.18 which edge tracks.

These Alpine-based Chainguard Images are a bit of a weird bird, since they depend on Alpine's packages and release schedules. This is a good example of where we don't have as much control/visibility as we'd like, and where changes outside our control can cause confusion downstream. I'm sorry about that.

[skaegi]

Thanks @imjasonh -- skopeo list-tags docker://cgr.dev/chainguard/static indeed shows many tags we can use.

The pre-amble in the README here -- https://github.com/chainguard-images/images/tree/main/images/static -- led me to think we could only use "latest".

[imjasonh]

The pre-amble in the README here -- https://github.com/chainguard-images/images/tree/main/images/static -- led me to think we could only use "latest".

That's helpful to know. The intention of that wasn't to indicate that older tags aren't available for static, but that tags for older versions of "real" images (e.g., python, nginx) might be available by contacting sales.

Since static doesn't really contain any particular software in it by design, this isn't really appropriate. I'll remove that warning for static: chainguard-images/images#412

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[skaegi]

/reopen

@imjasonh @afrittoli @pritidesai -- sorry to flag but really need help here
This is still causing us grief (and now an escalated version of grief) as it was never fixed. Could we please switch to using an image version with an alpine release instead of alpha. I'd create a PR but I've tried to look at the tags cgr.dev/chainguard/static and have no idea what I'm looking at and also don't understand how these images are getting built.

[tekton-robot]

@skaegi: Reopened this issue.

In response to this:

/reopen

@imjasonh @afrittoli @pritidesai -- sorry to flag but really need help here
This is still causing us grief (and now an escalated version of grief) as it was never fixed. Could we please switch to using an image version with an alpine release instead of alpha. I'd create a PR but I've tried to look at the tags cgr.dev/chainguard/static and have no idea what I'm looking at and also don't understand how these images are getting built.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[skaegi]

/remove-lifecycle rotten -- more housekeeping -- we really need this addressed.

[jerop]

/lifecycle frozen

[afrittoli]

@imjasonh @skaegi is this something we could fix for now by pinning a specific version of the static image on Tekton side and making a new patch release?

[skaegi]

Yes, what is triggering the problem is the contents of /etc/alpine-release and /etc/os-release. The scanner that is giving us grief is clair based so this might also eventually be relevant -- quay/claircore#923

... but I think even if clair had support we would still get flagged because of using an alpha release...

/tmp # STATIC_ID=$(docker create cgr.dev/chainguard/static:latest sh) && docker export $STATIC_ID | tar -xC /tmp/static
/tmp # cd /tmp/static
/tmp/static # cat etc/alpine-release 
3.19_alpha20230901
/tmp/static # cat etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19_alpha20230901
PRETTY_NAME="Alpine Linux edge"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

Just to be clear, as this is our base image you get the same result when using -- gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.53.0@sha256:9cef507c33127c488938fd6af10c0c2242b4b667732e488545338f290025fa08 as the image.

[afrittoli]

Using

curl -H "Authorization: Bearer $tok" \
  https://cgr.dev/v2/chainguard/static/_chainguard/history/latest | jq

The last five releases of static:

    {
      "updateTimestamp": "2023-07-11T20:32:40.602Z",
      "digest": "sha256:6b35c7e7084349b3a71e70219f61ea49b22d663b89b0ea07474e5b44cbc70860"
    },
    {
      "updateTimestamp": "2023-08-10T17:44:41.743Z",
      "digest": "sha256:67a1b00e0134e2b3a614c7198a26f7deed9d11b7acad4d52c79c0cfd47a2eae7"
    },
    {
      "updateTimestamp": "2023-09-02T00:27:17.261Z",
      "digest": "sha256:a432665213f109d5e48111316030eecc5191654cf02a5b66ac6c5d6b310a5511"
    },
    {
      "updateTimestamp": "2023-09-28T00:23:34.6Z",
      "digest": "sha256:ef5add7fd46cf1ce7d33d6de517833ac5c7e749db9b15249f9c472a772f3af27"
    },
    {
      "updateTimestamp": "2023-10-30T23:01:43.037Z",
      "digest": "sha256:d3465871ccaba3d4aefe51d6bb2222195850f6734cbbb6ef0dd7a3da49826159"
    }

Checking the alpine version for each of them:

curl -H "Authorization: Bearer $tok" \
  https://cgr.dev/v2/chainguard/static/_chainguard/history/latest | jq -r | grep digest | awk '{ print $2 }' | sed 's/"//g' | while read aa; do \
echo "${aa}: $(crane export cgr.dev/chainguard/static@${aa} --platform linux/amd64 - | tar -Oxf - etc/alpine-release)"; done

I see that the switch to edge happened in September:

sha256:6b35c7e7084349b3a71e70219f61ea49b22d663b89b0ea07474e5b44cbc70860: 3.18.0
sha256:67a1b00e0134e2b3a614c7198a26f7deed9d11b7acad4d52c79c0cfd47a2eae7: 3.18.0
sha256:a432665213f109d5e48111316030eecc5191654cf02a5b66ac6c5d6b310a5511: 3.19_alpha20230901
sha256:ef5add7fd46cf1ce7d33d6de517833ac5c7e749db9b15249f9c472a772f3af27: 3.19_alpha20230901
sha256:d3465871ccaba3d4aefe51d6bb2222195850f6734cbbb6ef0dd7a3da49826159: 3.19_alpha20230901

We should pin Tekton builds to sha256:67a1b00e0134e2b3a614c7198a26f7deed9d11b7acad4d52c79c0cfd47a2eae7 instead of latest, and backport to LTS and rebuild as well.

[afrittoli]

This is required too for a full fix #7366

[mattmoor]

As of yesterday, the default base is Wolfi-based, so this shouldn't be an issue.