-
Flux v1 has an image update automation feature that is in widespread use. Flux v2 will need this to have parity with Flux v1 -- but of course we want it to work in the spirit of GitOps Toolkit. I have been prototyping a set of components that will do the image update automation as well as set the pattern for other kinds of automation. The design, in progress, is here: fluxcd/image-reflector-controller#5. The code is in https://github.com/squaremo/image-reflector-controller and https://github.com/squaremo/image-automation-controller. The main areas of uncertainty are:
Update: I have marked the kyaml setters design as the answer, since the implementation seems to work satisfactorily in practice. There are nonetheless lots of good ideas and points elsewhere in the discussion, and I'll be referring back here. Thanks everyone! |
Beta Was this translation helpful? Give feedback.
Replies: 13 comments 51 replies
-
There's now a run-through of the "working at all" implementation: https://github.com/squaremo/image-automation-controller#readme (The implementation is split into two controllers: image-reflector, which scans image repositories, and image-automation, which updates git by looking at the outcome of the scans) This is what the manifests look like: # image.yaml
# Declares an image repository to scan. This is where things like registry
# auth, and scan interval, would go, for example
apiVersion: image.fluxcd.io/v1alpha1
kind: ImageRepository
metadata:
name: app-image
spec:
image: squaremo/cuttlefacts-app # policy.yaml
# Defines a rule for "latest image" of some image repo. There could be
# different ways of filtering and ordering (similar to Flux v1, with glob and
# regexp, but potentially also things like "order by commit history")
apiVersion: image.fluxcd.io/v1alpha1
kind: ImagePolicy
metadata:
name: app-policy
spec:
imageRepository:
name: app-image
policy:
semver:
range: 1.x # repo.yaml
# This is a type from source-controller, but used here to give access to
# the git repository, for making updates.
apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: cuttlefacts-auto
spec:
url: ssh://git@github.com/squaremo/cuttlefacts-app-automated
interval: 1m
secretRef:
name: cuttlefacts-auto-deploy # update.yaml
# This defines an automated update to do; in this case, update files
# in the given repo to use the latest image as calculated by the given
# ImagePolicy. It also gives parameters for the commits that will be made;
# here you could tell it to push to a fresh branch, or even open a PR, say.
apiVersion: image.fluxcd.io/v1alpha1
kind: ImageUpdateAutomation
metadata:
name: update-app
spec:
gitRepository:
name: cuttlefacts-auto
update:
imagePolicy:
name: app-policy
commit:
authorName: Updatebot
authorEmail: bot@example.com
messageTemplate: |
Here look what I did
|
Beta Was this translation helpful? Give feedback.
-
Things that this sets out to solve:
|
Beta Was this translation helpful? Give feedback.
-
Uses case: automated updates for a public container imageGit-to-cluster reconciliationHaving the following deployment in Git: apiVersion: apps/v1
kind: Deployment
metadata:
name: podinfo
namespace: default
spec:
selector:
matchLabels:
app: podinfo
template:
metadata:
labels:
app: podinfo
spec:
containers:
initContainers:
- name: init
image: stefanprodan/podinfo:4.0.0
command:
- sh
- -c
- "sleep 1"
- name: podinfod
image: docker.io/stefanprodan/podinfo:4.0.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 9898
protocol: TCP
command:
- ./podinfo
- --port=9898 Having a GitRepository registered with a read-write deploy key: apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
name: podinfo
namespace: gitops-system
spec:
interval: 1m
url: ssh://git@github.com/stefanprodan/podinfo-deploy
secretRef:
name: ssh-podinfo
ref:
branch: master Having a Kustomization reconciling podinfo: apiVersion: kustomize.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
name: podinfo
namespace: gitops-system
spec:
interval: 5m
sourceRef:
kind: GitRepository
name: podinfo
path: "./default/"
prune: true
healthChecks:
- kind: Deployment
name: podinfo
namespace: default
timeout: 2m Image update based on semver rangeRegister the podinfo container registry: apiVersion: image.fluxcd.io/v1alpha1
kind: ImageRepository
metadata:
name: podinfo
spec:
image: docker.io/stefanprodan/podinfo Define the image updated policy using a semver range: apiVersion: image.fluxcd.io/v1alpha1
kind: ImagePolicy
metadata:
name: podinfo
spec:
imageRepository:
name: podinfo
policy:
semver:
range: ">=4.0.0 <5.0.0" Define the image update automation: apiVersion: image.fluxcd.io/v1alpha1
kind: ImageUpdateAutomation
metadata:
name: podinfo
spec:
gitRepository:
name: podinfo
update:
imagePolicy:
name: podinfo
commit:
authorName: fluxcdbot
authorEmail: fluxcdbot@@users.noreply.github.com
messageTemplate: |
Bump podinfo version Questions
|
Beta Was this translation helpful? Give feedback.
-
Proposal: select targetsThe image update automation could target Kubernetes objects by label/namespace or target a kustomization.yaml file by path. Target Kubernetes objectsSelect a group of Kubernetes objects such as deployments, daemonsets, statefulsets, cronjobs, jobs or pods: kind: ImageUpdateAutomation
spec:
update:
imagePolicy:
name: app-policy
selector:
app: my-app Select Kubernetes objects filtered by labels and namespaces: kind: ImageUpdateAutomation
spec:
update:
imagePolicy:
name: app-policy
selector:
app: my-app
namespaces:
- dev
- staging The above proposal addresses only Kubernetes native kinds and doesn't allow targeting a specific container/initContainer but all of them. I don't think there is valid use case for targeting containers. Target kustomizationsSelect a kind: ImageUpdateAutomation
spec:
gitRepository:
name: app-repo
kustomization:
path: "./deploy/overlays/dev" # <- kustomize edit set image app=app:semver |
Beta Was this translation helpful? Give feedback.
-
Chasing up a couple of things I mentioned ...
For example, naming several policies to dereference when doing an update. Pros: no need to have an automation object for each image policy; commits can group together changes. Cons: complication; using in combination with a single selector might not make much sense (in general, you'd want to pick out which policies apply to which workloads).
For example, a fixed value that is updated by hand.
Pros: simple, easy to use as a primitive for a higher layer. Cons: a proliferation of objects; you don't usually want to change the same image in lots of workloads, making selectors less useful. |
Beta Was this translation helpful? Give feedback.
-
How about something like the below? apiVersion: image.fluxcd.io/v1alpha1
kind: ImageUpdateAutomation
metadata:
name: podinfo
spec:
gitRepository:
name: podinfo
policy:
semver:
range: ">=4.0.0 <5.0.0"
images:
registries:
- name: docker.io
include:
- docker.io/*
exclude:
- docker.io/library/*
commit:
authorName: fluxcdbot
authorEmail: fluxcdbot@@users.noreply.github.com
messageTemplate: |
Bump podinfo version apiVersion: image.fluxcd.io/v1alpha1
kind: ImageRegistry
metadata:
name: docker.io
spec:
url: docker.io
secretRef: docker.io-auth
interval: 10m |
Beta Was this translation helpful? Give feedback.
-
To paraphrase some of the things that have been mentioned in the discussion above ... Limitations in the toytown implementationThe current implementation mentions a single image policy and a git repository, and updates every workload using the image in question, according to the policy. This is limited in several ways: You can't restrict it to updating only certain workloads, either by file path or by properties of the workload (e.g., its name, or labels) For some scenarios this is fine, since you'll only be automating the update of say, a few images you build in your own CI. But it might cause problems if there's different configs in the git repo, needing different policies -- say if you have different directories for staging vs production. You can only update one image at a time Some installations might have hundreds of images to be automated. That's 1. a lot of Taking these one at a time:
When applying a group of policies, it makes less sense to have narrow selection of workloads -- there is a tension between coarse-grained operations, and pin-pointing exactly what you want to change, because images often are close to 1-1 with workloads. Targeting kustomizationsKustomizations are explicitly supported by GOTK. To automate these, you have to target the kustomization.yaml file, rather than the YAMLs in the base. This would be an alternative to targeting specific files or workloads. In a design where you mark images for update (see above), ideally you would mark a field in the kustomization in the same way. Parameterising the location of the kustomization file @adusumillipraveen suggested that the path to the kustomization might be parameterised, similarly to how .flux.yaml commands receive various arguments in environment variables. This could end up being quite brittle, though; it's probably better to simply have automation objects for each of the variations you want (assuming the automation objects become a bit more expressive). Targeting custom resourcesIt's a reasonable expectation that
This requires either a blunt instrument (replace the image string wherever you see it; not useful if, say, the tag is in its own field), or a little language for pointing at fields that contain image references.
Kustomize knows how to update image fields of the usual Kubernetes types, but not how to patch the values in a HelmRelease file. A different factoring entirely, that focuses on which images to automate@moshloop suggested a different factoring of the definition of an automation; instead of naming This is quite close to how you might think of what you want as a user ("I want images in my own dockerhub org to be automated"). A potential downside is that it might become fiddly once you want to narrow down which workloads are affected, say. Another is that it may be hard to compose automation objects, since there's a lot of opportunity for specifications to overlap. |
Beta Was this translation helpful? Give feedback.
-
Design: Automation via kyaml setters2In this design:
Target scenarios
Questions ...How does this translate into ImageRepository and ImagePolicy objects? There are some alternatives: A) Implicitly -- look at the current value for the setters, and perhaps some auxiliary information, and coalesce into objects. B) Higher-level description: the repositories and policies are defined in the same place as the setters (or define the setters), and they are created by the controller interpreting that (again, perhaps with auxiliary information, like credentials, from elsewhere) C) Explicitly -- there's a map from setter to image policy name, and the latter is assumed to exist. One question that might decide it is: if the markers are in the files, where do the setter definitions live? Is there any sense in having them as a resource, or should they just be in a file in the repo? There's no particular benefit to having another object over and above the automation object, so you might expect to put this information in the automation object. If the definitions live in a resource in the cluster, it is a little harder to make sure they are in sync with the markers in the files in git. If they live in a file or files in the repo, that's one extra file that needs to be maintained -- but maybe that's not too bad, since you need to set up the markers anyway. Providing credentials Making available the credentials for accessing a particular image registry or image repo is a runtime concern. This implies it should be an object in the cluster (even if it's synced from git). In the case C) above, image repository objects are created explicitly, and they will include (a reference to) the credentials. But in A) and B) above, the image repository objects may be created implicitly, and something will have to decide which credentials to use. ** Using tags vs whole image vs other bits .. ** Sometimes you might want to set only the tag, or set the tag and repo separately. This might need to be specified alongside the setter/policy, or perhaps synthetic names used e.g., Unanswered questions
Hypothetical user guide for C)You are here because you want to use Flux to automate updates in one or more git repositories. You might run the automation in the same cluster as you run your apps that are being automated; or you might run it in another cluster, or at least in a separate namespace. !!! note Unlike Flux v1, automation in Flux v2 does not require the workloads to be running in the same cluster.
To start the process, you'll have to give the automation access to your git repository. Create a GitRepository with the Flux command-line:
First we'll create a policy for updating a particular image; then we'll apply that policy in the deployment file for the app.
Now we have an image policy that yields a particular image with its tag, which we'll attach to a deployment.
The apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld
spec:
template:
spec:
containers:
- name: hello
- image: squaremo/helloworld:v1.0.0 # {"$ref": "app-image-policy"}
# ... Commit the change so it'll be seen by automation:
Lastly, create an automation object to tell the controller to run automation using this file:
This will update any files under app/, so if you had other images to be updated, you could go through the above steps to create a policy and mark fields to be updated. You can monitor the automation object to see when the automation has run:
|
Beta Was this translation helpful? Give feedback.
-
Does this proposal include chart version update automation? I know the helm-controller has support for specifying a semver range already, but if you want more control/history/observability of your chart version updates than that, it would be useful to drive those through git commits (or even pull requests if you want to gate with human code review) instead. |
Beta Was this translation helpful? Give feedback.
-
Any thoughts about supporting code review gating of the git updates? So rather than direct git commits, it would create pull requests / patchsets (gerrit terminology). If there is an open pull request when an update comes in it would update the existing pull request rather than creating a new one, so that outdated pull requests aren't left around. Note that this is similar to Dependabot. There have been requests there for container image, HelmRelease and Helm chart dependency versions. However those are not implemented yet, and Dependabot is specific to Github and doesn't support Gerrit for example. So it would be great to have that functionality in gotk. |
Beta Was this translation helpful? Give feedback.
-
I was wondering if there was any thought around supporting other templating solutions here (i.e. not just helm or kustomize). In flux v1. the solution seems fairly generic (via This was useful because it basically overwrites whatever is in your rendered manifests (that seemed to meet our use-case). It seems like my question is similar to those around HelmRelease CRDs or Charts (although in my case, it's just Jsonnet). |
Beta Was this translation helpful? Give feedback.
-
Is the ImagePolicy semver only? Or can I configure it to get the latest tag. So that the latest sha256 id of the image gets replaced by the setter? This would be my dev-environment use-case to always deploy the latest version of my app. |
Beta Was this translation helpful? Give feedback.
-
Hello! Is there are roadmap for other policies in addition to semver? I'm thinking in the way flux v1 has regexp and glob with either full regexp or some simple patterns. Also, does semver work with prefix like vX.X.X? Or with suffixes like -mybuild ? |
Beta Was this translation helpful? Give feedback.
Design: Automation via kyaml setters2
In this design:
https://github.com/kubernetes-sigs/kustomize/tree/master/kyaml/setters2 (but this is mostly transparent for the user)
Target scenarios
Questions ...
How does this translate into ImageRepository and ImagePolicy objects?
There are some alternatives:
A) Implicitly -- look at the current value for the setters, and perhaps some auxil…