Publish a ko image

[bobcatfish]

In tektoncd/pipeline#529 and tektoncd/pipeline#528 I'm adding dogfooding CI/CD logic for publishing tekton pipelines images and yamls.

This means I want to create a Task step or steps which invoke ko, since this is how we currently build and publish our images (via a bash script).

To do this, I need a container with ko installed. For now I'll have a first step that installs ko, but would be great if a ko image could be published and available for folks to use.

(Looks like #358 might be related but I'm not sure?)

[bobcatfish]

Additionally can this image also know how to auth with provided kubernetes style service accounts (maybe via mounted files or environment variables)? 🙏

[bobcatfish]

After some discussion with @jonjohnsonjr , a couple options for the auth interface:

1. Compile a different ko binary for use in the image which does something similar to how kaniko detects if its running in k8s and uses k8schain in that case
2. Create a k8schain-based cred helper that is bundled in the image, e.g. something like https://github.com/docker/docker-credential-helpers <-- the better option!

[jonjohnsonjr]

Best option, I think, is to mount docker push secrets at ~/.docker/config.json, which ko would then pick up. Similar to https://github.com/tektoncd/pipeline/blob/master/docs/auth.md#basic-authentication-docker

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[06kellyjac]

I'd still like an official ko image or at least a reason why not to have one.

[jonjohnsonjr]

I'm not opposed to having one, but credentials are tricky.

[bobcatfish]

@jonjohnsonjr the way we solve this in our tekton release infra is not super great

we build an image with both ko and gcloud (and kubectl) and explicitly call out to gcloud to activate a service account and configure docker:

      # Activate service account
      gcloud auth activate-service-account --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
      # Setup docker-auth
      gcloud auth configure-docker

[jonjohnsonjr]

I was looking into how minikube configures credentials for an unrelated issue and noticed that they use this project: https://github.com/upmc-enterprises/registry-creds/

Possibly we could do something similar?

Also, for crane, somebody mentioned they were just piggybacking off of the kaniko image: google/go-containerregistry#731 (comment)

[jonjohnsonjr]

To sketch out what I'd probably want to do for this:

1. Create a docker-credential-k8schain binary. This would speak the docker credential helper protocol, but fetch credentials from the environment using the same logic as k8schain. Implementing this would mean we can use the standard docker credential file lookup logic, and change nothing about ko to support this. I'm not sure if it's easy to make this always "just work", especially when pulling public images -- do we fall back to anonymous appropriately?
2. Create an image containing:
   1. The go toolchain, probably just a golang base.
   2. docker-credential-k8schain.
   3. ko, on $PATH.
   4. ~/.docker/config.json, wired up to use docker-credential-k8schain.
3. A pipeline on commit/release to create that image.
4. Some tests that verify this doesn't break. It is hard to test against different kube providers :/

An easier approach would be to just merge gcr.io/kaniko-project/executor and golang, then slap a ko binary on top.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[jonjohnsonjr]

This issue spawned https://medium.com/@jdolitsky/docker-credential-magic-a-magic-shim-for-docker-credential-helpers-deae9e78c2df

We should look at that as a solution.

[jonjohnsonjr]

#528 👀

[imjasonh]

FYI the ko image doesn't have golang or creds installed at this time, and we shouldn't recommend it for general use just yet. But I'd like it to end up there, and for that to be a way to use it, with Tekton, GCB, etc.

[mattmoor]

Can I also plug for us to make sure that the appropriate credential helpers are installed for this to work in cluster contexts 🙏

In cosign we added a --k8s-keychain flag to avoid the need for lots of heavy binaries, but if we're basing on golang already then YOLO

[imjasonh]

Sure, how do you want that to work?

I think a cred-less ko image is more useful than nothing, so if giving it creds blocks a release I'd say we can still punt and figure it out later.

[mattmoor]

So I'd say "Just docker-credential-magician the golang image" but that has problems because it'd invoke docker-credential-gcr for the distroless base in non-GCP contexts and fail to auth.

Honestly, the best answer I'm aware of is to use k8schain but nobody listens to me about that 🤣

[jonjohnsonjr]

So I'd say "Just docker-credential-magician the golang image" but that has problems because it'd invoke docker-credential-gcr for the distroless base in non-GCP contexts and fail to auth.

I'd rather fix this than use k8schain

[mattmoor]

I'd rather fix this than use k8schain

Yes, we've met 😉

[imjasonh]

I'd rather do anything besides expose a --k8schain flag to users, a cryptic string that holds no meaning to anyone but maybe four cursed individuals.

[mattmoor]

We didn't call it k8schain 🙃

https://github.com/sigstore/cosign/blob/7e5ff000f6af25793631eb590964b9a861d672dc/cmd/cosign/cli/options/registry.go#L44-L45

[imjasonh]

We didn't call it k8schain 🙃

ohthankgod 😅

[imjasonh]

This is done now. ghcr.io/google/ko