RFP: Using OCI artifacts as sources

[squaremo]

It's been suggested several times, in different places, to make OCI artifacts* available as a source. A few things recommend using OCI:

• artifacts are immutable, like git
• they can be signed, like git
• there's lots of tooling for building, distributing and serving images (that mostly also works for artifacts)
• if you are running a CI pipeline to verify your configuration, it makes sense to stick the verified configuration in an immutable, distributable form at the end

*What is an OCI artifact as opposed to an OCI image? An OCI image is the standardised form of a Docker image, that is something you can run. An artifact is the more general specification, which shares the manifest, index and layer bits of OCI images, but you can define your own format. This article does a good job of explaining what artifacts are: https://dlorenc.medium.com/oci-artifacts-explained-8f4a77945c13

Related issues and discussions:

• There's an issue already for using Helm charts from OCI registries.

Design and implementation notes:

Handling different media types

OCI artifacts come in different formats -- it's not just runnable container images any more, and not just an overlay filesystem in tarballs. In general, formats will not be suitable for distributing Kubernetes configuration; and, each format will have its own layout of files in layers. So I would expect an implementation to handle specific media types, rather than try to be generic across media types.

A particular case is Helm charts in OCI artifacts: these are suitable for consumption by the helm-controller, and not by anything else. But, in the spirit of remaining duck-typed, it's OK to let people mix and match as they wish.

Relation to image.toolkit.fluxcd.io API

The image.toolkit.fluxcd.io API has these types:

• ImageRepository, representing an image repository to be scanned (and referring to secrets needed to do so)
• ImagePolicy, representing a rule for picking a "latest image" from an ImageRepository
• ImageUpdateAutomation, which uses the calculate "latest image" from ImagePolicy objects to make changes in a source (which is currently restricted to a git repository).

These stand alone, but there is clear overlap in the arrangement of ImageRepository and ImagePolicy, and what an OCI source would do. There are some alternatives for how to account for this:

1. Refer to a policy from the source type. That is, have a type OCIArtifact that refers to an image policy which selects the particular image to serve to consumers.
2. Make an image policy (optionally) behave like a source. This might involve a switch in the ImagePolicy type useAsSource, which is understood by a separate controller (source-controller or another); or, something to switch it on for some subset of ImagePolicy types, from outside, e.g., by namespace.
3. Use a disjoint type. At least the GitRepository type mixes definition of the repository and the policy, so why not do this for OCI artifacts, and have an OCIArtifact type which combines ImageRepository, ImagePolicy, and acting as a source.

[souleb]

As an example the tekton project uses OCI artifacts to store tasks in a container registry instead of a crd in etcd. The tasks can then be referenced with an objectRef in a pipeline declaration.
https://tekton.dev/docs/pipelines/pipelines/#tekton-bundles

[stefanprodan]

Use a disjoint type proposal

I propose we add a command to Flux CLI that bundles a directory containing Kubernetes manifests and/or Kustomize overlays, pushes the artifact to an OCI registry (using the local docker config) and optinally signs the artifact with a sigstore key.

Example command:

flux bundle -f ./deploy -t ghcr.io/stefanprodan/podinfo-deploy:v1.0.0 -s cosig.key

As for the API, I opt for Use a disjoint type. I propose we add a new API type to the source-controller named ArtifactRepository.

Example of a source that pulls a fixed version from the OCI registry and verifies its authenticity using sigstore:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: ArtifactRepository
metadata:
  name: podinfo-release
  namespace: flux-system
spec:
  interval: 10m
  image: ghcr.io/stefanprodan/podinfo-deploy
  secretRef:
    name: regcred
  tag: "1.0.0"
  verify:
    secretRef:
      name: sigstore-keys

Example of a source that pulls the latest tag based on timestamp ordering:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: ArtifactRepository
metadata:
  name: podinfo-nighlty-builds
  namespace: flux-system
spec:
  interval: 10m
  image: ghcr.io/stefanprodan/podinfo-deploy
  secretRef:
    name: regcred
  filterTags:
    pattern: '^main-[a-fA-F0-9]+-(?P<ts>.*)'
    extract: '$ts'
  policy:
    numerical:
      order: asc

Example of a source that pulls the latest tag based on semver ordering:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: ArtifactRepository
metadata:
  name: podinfo-release-candidates
  namespace: flux-system
spec:
  interval: 10m
  image: ghcr.io/stefanprodan/podinfo-deploy
  secretRef:
    name: regcred
  filterTags:
    pattern: '.*-rc.*'
  policy:
    semver:
      range: '^1.x-0'

[squaremo]

people would just have to change the sourceRef.kind from GitRepository to ArtifactRepository.

or GitRepository to ArtifactPolicy; is that somehow more difficult?

[squaremo]

I don't think ArtifactPolicy is necessarily the best word there, by the way. Just using Artifact might be better, off the top of my head.

[stefanprodan]

Looks like we hit a deadlock, you want to change the Source API into Repos+Policies while I want to preserve the current design. I don't see how we could do the Repos+Policies for OCI only, should be the same all sources, means breaking the current API, drop sourceRef from all reconcilers and replace it with an artifactRef or something like that.

[squaremo]

No deadlock, I'm pushing back on your arguments because I want to have the thinking put on record. If I had an immovable opinion, I would have put it in the original post :-)

It's certainly true that there is no more reason to separate access and policy for ArtifactRepository than there is for GitRepository (and no less either). Fair enough if you want to stick to the current scheme of combining both concerns, and kick consideration of separating them down the road. It works fine for end users, I think, just less so for people wanting to extend or build on GitOps toolkit.

The question remains: why have another type representing images from an image repository, when those are already in the image[...] API? I think it would be different enough in purpose, and detail, that it warrants being its own thing. By "purpose" I mean that the reason it's there is to act as a source, whereas an ImagePolicy calculates an image ref for automation -- similar machinery, but not the same desired outcome. By "detail" I mean things like handling different media types, and checking signatures (though perhaps we'd want that latter for image policy too).

[stefanprodan]

I'm for making Flux easy to extend but not at the detriment of Flux end-users. Doubling the amount of YAML files needed to define a source is something that I think will have grave implications on our user base. I'm also concerned about introducing a major breaking change in one of Flux's core APIs (if we split the sources into repos/policies) that would push GA to at least 6+ months, as we need to build a migration tool for Flux manifests in Git (we promised no breaking changes without conversion and this change can't be fixed on the cluster side, we'll need a flux migrate bootstrap --url=my-repo).