Migration path for .flux.yaml users not using kustomize

[squaremo]

https://toolkit.fluxcd.io/guides/flux-v1-migration/ gives instructions for migrating a Flux v1 installation to Flux v2, if you are either syncing plain YAMLs, or using .flux.yaml to run kustomize. There is a constituency of people who use .flux.yaml to run things other than kustomize -- e.g., envsubst. It would be good to give those people a plausible path to migrate.

The key bit will be to have something runs an arbitrary command, and can be hooked up to GOTK controllers. Tekton and GitHub Actions spring to mind, and have different strengths.

1. Try to find (from the community) some examples of .flux.yaml without kustomize. We know some people wanted to use it with envsubst, so that might be a decent example to start with, albeit fairly simple.
2. Make worked examples for your chosen technologies from above -- e.g., construct a Tekton pipeline that is triggered by new commits, runs envsubst, and stashes the result somewhere for kustomize-controller to pick up.
3. Generalise the examples into guides -- perhaps one for each of the technologies -- showing how to migrate from a Flux v1 configuration using .flux.yaml, to Flux v2 plus the pipeline-running technology.

[squaremo]

@alisondy This is what I was thinking of as a good way to get familiar with Flux -- both the old and the new!

[erkannt]

An example application that isn't covered by the Kustomize functionality of flux v2 is Sealed Secrets. They use jsonnet in their repo to build a manifest.yaml that is part of their github releases.

• SourceController can access the repo, but KustomizeController can't handle jsonnet
• KustomizeController could handle the manifest, but SourceController can't access gh-releases

[stefanprodan]

KustomizeController could handle the manifest, but SourceController can't access gh-releases

Any public GH artifact URL can used as a kustomize remote base, example here: https://github.com/fluxcd/flux2-multi-tenancy/blob/main/infrastructure/kyverno/kustomization.yaml

[erkannt]

Thanks Stefan, I had only read the kustomize.toolkit.fluxcd.io/v1beta1 Kustomization.

I'm guessing doing this would have to rely on some other update mechanism if we wanted to track resource releases? Happy to use something like renovatebot regex if having artifact URLs managed by SourceController is out of scope.

[stefanprodan]

@erkannt here is an example on how to use GitHub Actions to check for new releases and open a PR when a new version is released:

• GitHub Workflow: https://github.com/fluxcd/flux2-multi-tenancy/blob/main/.github/workflows/update.yaml
• Automated PR: Update kyverno to v1.3.0-rc7 flux2-multi-tenancy#5

[groodt]

I've got a question. Is the following supported or planned to be supported in flux2?

We currently use flux1. We render our k8s manifests with jsonnet and our container images in ECR.

How might it be possible to implement the following:

• Newly built and tagged image is pushed to ECR
• Jsonnet source code in k8s repo is updated with new image tag
• Jsonnet source is rendered into manifests (via build.sh script or jsonnet directly)
• Newly rendered manifests are synced to the cluster

[kingdonb]

I would like to spike an attempt at supporting this use case, I had in mind to use either Jenkins, tekton, or GitHub Actions, to push those commits representing the "jsonnet" build output, (consider this implementation will not be specific to jsonnet, a "bring your own template rendering" use case) with rendered manifests outputting to either a git repository or an S3 bucket.

There are some ways to make this deeply integrated with gitops toolkit that are coming to mind, but since flux does not seek to replace CI and this really seems like a CI job thing, I'm going to assume you have some kind of CI that can stand in this role and run jsonnet, whether it's one of those from the list above or not. Some solutions could be driven by a signal from source-controller GitRepo, so that nothing really changes in the main Flux workflow (flux is still deploying rendered manifests from a git branch with kustomization-controller) but now Kustomization refers to the branch where those outputs are committed, and our CI process is responsible for watching for commits on the main branch and pushing up those rendered manifest commits.

The most flux/gitops way that I can think of to make this sort of downstream behavior work is of course (not by using an unversioned S3 bucket) to have the output of eg. jsonnet published into a separate branch on the git repository by the CI process. This implies that when a failure in the render-manifest process comes along, there may be cases where it is not enough to simply roll back the upstream git repo and repeat the build. Some tools like jsonnet will be idempotent and for this spike it may be good to assume the tool used will be safe and idempotent, but in a real solution I think you cannot assume this, it may be some other conditions that cause it to fail again with inputs that succeeded before. The build output being versioned implies the existence of another controller to manage health assessments and regulate the progression states and the separate rollback, or perhaps even separate workflow for independently suspending the manifest generation loop.

You could roll back the commit manually on that website branch, but I think with gitops toolkit maybe there's a better solution in here. Seems like a spike could produce a meaningful proposal that would help to flesh out that information.

I think it goes without saying that if you use eg. an S3 bucket that is not versioned for output, then your template manifest generation should be completely pure and idempotent (else you will break most rollback guarantees). Of course that's bad, we want those guarantees and we don't want to add knobs that allow users to seriously compromise them. Flux is at the mercy of whatever arbitrary outside tool is chosen (it won't be only jsonnet, it may be some tool that even depends on some outside network, like a service that selects images to deploy based on A/B testing decisions, or arbitrary other inputs like phases of the moon). So even if jsonnet is well behaved, this is not a good concession. But the use case is important.

So that can be any CI process, which produces the rendered manifests and publishes a new commit on that "rendered" manifests branch. Right now I can imagine at least 6 different ways to do this, I would limit the spike and try to present no more than 4 ways, probably in a 2x2 matrix of (Git source vs S3 bucket), built on (GitHub Actions vs Tekton). Would this help?

[groodt]

Would this help?

Yes. I think it would. I agree with everything youv'e said.

It could also be said that what I'm proposing is not very different from:

kustomize build myapp/src --output myapp/rendered

kustomize happens to be the currently blessed template rendering tool for k8s because it is tightly integrated with kubectl.
Helm is equally popular because it built a community around charts and has many adopters.

However, there are many others in the ecosystem: jsonnet, envsubst, cue, pulumi etc.

These other tools do have reasonable levels of adoption and they are useful for many teams. Sometimes people need a bit more power than kustomize and a bit less ceremony or conventions than Helm.

I agree with what you say, that arbitrary rendering scripts that are non-total and non-deterministic can cause issues, but obviously this is something that can be documented and that obviously people using this feature should aim to use deterministic configuration tools.

So yes, if you think the rendering should happen after a CI hook triggered from fluxcd, that sounds fine. Alternatively, a custom pod that could somehow be invoked by the image-automation-controller to trigger what it does currently with updates to YAML. From my perspective, I'm not looking for much more than that. The way that we invoke jsonnet in our repo is very similar to this:

kustomize build myapp/src --output myapp/rendered

I'm just looking for a general way to take an update from ECR and update some configuration files in my repo, which are then used to render YAML manifests and all get committed to the git repo.

[kingdonb]

So, say you're using cdk8s and your "hydrate" command is cdk8s synth that runs somewhere in your CI pipeline, with any necessary dependencies taken care of by the CI pipeline or Dockerfile in advance of getting to this stage of the build. That was the toolkit I used today in my first attempt at solving this from an end-user perspective, I made some assumptions, including "you will already have a CI process in place and Flux should remain agnostic about it."

I had my CI built in Jenkins, for example, (to be clear... please don't interpret this as meaning I expect you to use Jenkins. You should expect to use whatever CI process is in place already, and only take my example of the Jenkins server as one possible idea for how it could work. I had this CI environment laying around and I was comfortable working inside it, so I used this.)

(Edit: the next iteration on this example will use jsonnet and GitHub Actions, which simplifies a lot of things drastically, ... it will be a better demo example. I got some feedback about this example and so, now ready to iterate again.)

So, you make a change in your main.ts file that represents the intent to change the resources on the cluster like this one:
kingdon-ci/example-cdk8s-ruby@8d5f965

The output is captured during the CI process, committed into a special branch that is just for jsonnet output, or cdk8s synth output, or whatever your chosen yaml un-minifier, and output pushed up to the branch in a commit like this one:
https://github.com/kingdon-ci/example-cdk8s-ruby/blob/synths/synths.k8s.yaml#L30
– I've made a lot of assumptions already, including one: your CI process will need write access to a git repository. Flux has to apply the generated manifest from somewhere, I chose to use a Git repo, you may choose an S3 bucket or something else, ... keeping in mind with this solution that you will need a separate branch or bucket for each downstream environment.

We can perhaps do better than all of this, opinions may vary, but I sort of like how this turned out...

...so, then, the cluster applies it as a regular Kustomization, with eg.
this: GitRepository
and this: Kustomization (edit: oops, fixed visibility so these are public...)

This certainly will not work for everyone, but it is a start. I did this as a spike, which in my process means "there is no rule or inviolable requirement because this is just a spike." So I have undoubtedly made some concessions which are not production-worthy and this whole idea must go back to proposal stage before we can consider including this as an example in Flux. RFC

I have not thought about how the CI process could be triggered from inside of Flux. I will have to consider the other feedback you provided, I expect this solution will change a little bit when it's replacing cdk8s with jsonnet, or jenkins with jenkinsx (or tekton, or whatever other solution is in the next iteration of the spike.)

Hopefully this example is easy to follow, but as it is my first try at solving this, please expect to find some rough edges and tell me about it, any feedback in kind is all welcome. This is not polished or demo quality, but if you have specific feedback do feel free to open an issue right on the example repo.

If you take a look at this solution and let me know how it meets your requirements well, or if there are ways that it does not meet them, I will be happy to take any comments or criticism. I'm also happy to explain how it works. Questions, comments, or any other feedback is supremely valuable. 🙏

I plan to iterate on this further and show some different solutions that for example are using jsonnet or something else lighter weight besides cdk8s, and different CI solutions besides Jenkins. There may be a natural point in the CI process where this makes sense to commit back. Depending on your CI, there may not be this already. I had to add hooks along the way to attach the SSH key as a secret in CI, and there is not much elaboration on how this is done here. In a finished tutorial page, handling secrets properly will be a more focal and central idea, and it will likely depend on the specific CI environment.

Maybe this is an overreaching dependency into CI, maybe there is a better way to run from within Flux incorporating source-controller, and that does not depend on users presenting with their own elaborate CI processes in place. (And hopefully so!)

[stefanprodan]

Relevant RFC: fluxcd/kustomize-controller#253

[kingdonb]

Now that #1200 merged, we have a use cases guide in the documentation that covers one migration path for .flux.yaml from end to end (this one is using GitHub Actions, for Jsonnet):

https://toolkit.fluxcd.io/use-cases/gh-actions-manifest-generation/

Jenkins + Flux guide is planned next in this series, coming soon 👍