[EKS] [addon]: Support Flux V2 in EKS addons

[neillturner]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Please add Flux V2 in EKS addons. There is a helm chart now so should be relatively easy. See https://github.com/fluxcd-community/helm-charts

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Need to get flux installed reliably and simply in the EKS cluster in a way that is easy to upgrade without breaking all the installed resources by flux.
Are you currently working around this issue?
Flux and kubernetes provider in terraform but this is unreliable and can fail silently.
Flux have a command line flux bootstrap but this needs organisational access to the github repo and is unnecessary once a git repo has been setup.

Additional context
Once flux is installed then flux can automatically install the rest of the kubernets resources so its a big win in eks can do this in the addons.

Attachments
See https://github.com/fluxcd-community/helm-charts for the charts

[mikestef9]

We are working on this.

Several questions:

• Are you using the Flux v2 image automation controllers?
• Any interest in seeing EKS integrate with CodeStar Connections to make it simpler/secure to connect to Git repositories? Who is your Git provider?
• Are you using Flux v2 in a multi-tenant manner? Note recent Flux upstream enhancement proposal to make doing so simpler
• How are you monitoring actions taken by Flux? (notification controller, prometheus metrics, etc)
• Are you also using a progressive delivery operator like Flagger? How do you control rollouts to multiple clusters? For example, if you make a change in central repo, does it get applied all clusters at once, or do you have a process to roll out in waves.

[neillturner]

All good questions:

Are you using the Flux v2 image automation controllers?
Not currently

Any interest in seeing EKS integrate with CodeStar Connections to make it simpler/secure to connect to Git repositories? Not currently

Who is your Git provider? github

Are you using Flux v2 in a multi-tenant manner? Note recent Flux upstream enhancement proposal to make doing so simpler
No. we have one github repository with a directory per cluster

/clusters/development
/clusters/production

and inside that base, infrastructure and apps directories as per flux recommendations https://fluxcd.io/docs/guides/repository-structure/ ie a monorepo
we currently don't use dependson and multiple customizations for dependences. When we first built we can manually first add base via commit and push, then infrastructure via commit and push, and apps via commit and push.

How are you monitoring actions taken by Flux? (notification controller, prometheus metrics, etc)
we use the slack provider and and configure alerts. this done in the flux repo infrastructure directory as it relies on external secrets installed in base so doesn't need to be handled by the eks addon.

Are you also using a progressive delivery operator like Flagger?
No

How do you control rollouts to multiple clusters? For example, if you make a change in central repo, does it get applied all clusters at once, or do you have a process to roll out in waves.
We have separate directories for each cluster ie

./clusters/production
./cluster/development

i would recommend you create 2 adds one for each chart https://github.com/fluxcd-community/helm-charts/tree/main/charts
flux. - does install https://github.com/fluxcd-community/helm-charts/tree/main/charts/flux2
flux-sync - does sync https://github.com/fluxcd-community/helm-charts/tree/main/charts/flux2-sync
we would need to be able to config the values of each chart.

NOTE: I wouldn't worry about the flux bootstrap functionality that creates the git repo and ssh key etc as this is just a one time thing that doesn't need to be done every time an EKS cluster is created or updated.

NOTE: After running EKS if it created the flux addons then you should see a message in the source controller log saying:
source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-system","namespace":"flux-system","error":"auth secret error: Secret \"flux-system\" not found
then we would manually add the secret via kubectl which we copy for the secret manager which then starts off the sync. I think that is fine. If the addon was really smart you could specify the secret manager secret value for the flux secret and it could add that as a last step which would kick off the sync.

[mikestef9]

Thx @neillturner this is helpful. For your last point "If the addon was really smart you could specify the secret manager secret value for the flux secret and it could add that as a last step which would kick off the sync."

Wouldn't there also need to be initial GitRepository and Kustomization custom resources applied for the sync to kick off?

This is why we are thinking about CodeStar Connection integration, because you could store Git credential connection info there, specify the connection Arn and repo path as part of the add-on, and EKS add-ons would write out the secret, GitRepository, and Kustomization necessary for initial syncing to start.

[neillturner]

Sorry i have no experience with CodeStar or AWS Code Pipelines. If you have a smarter way to automate things that would be great although i don't want to be forced to use code pipelines or anything. We use github actions.

Generally i like to keep things simpler rather that having lots of different components. If you support CodeStar it would be good if it could still work if folks don't want to use CodeStar.

You also don't want the EKS Cluster install to break if github is down (something that does happen -)) so relying on a connection to the git provider to get the secret maybe will create more install dependencies which create more potential failures. -)

What we do is have a secret in secret manager called say
mycluster/flux-system
that is plaintext and contains the file for the secret that is encrypted:

apiVersion: v1
data:
  identity: XXXXXXX
  identity.pub: YYYYY
  known_hosts: ZZZZZZ
kind: Secret
metadata:
  annotations:
  managedFields:
  name: flux-system
  namespace: flux-system
type: Opaque

currently we manually get this into a file flux-system.yaml and then do a
kubectl apply -f flux-system.yaml

[neillturner]

to install flux via the helm charts we do the following:

1. create namespace
   helm apply -f namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: flux-system

2. install flux2

helm repo add fluxcd-community https://fluxcd-community.github.io/helm-charts
helm install flux2 fluxcd-community/flux2  --namespace flux-system  --version 0.6.1

Currently we just take default values

3. install flux2-sync
   helm install flux2-sync fluxcd-community/flux2-sync -f values.yaml --namespace flux-system --version 0.2.0
   where values.html like this:

gitRepository:
  spec:
    secretRef: 
      name: flux-system
    url: "ssh://git@github.com:/mycompany/mycompany-flux.git"
    ref:
      branch: main
kustomization:
  spec:
    force: false
    interval: 10m
    path: ./clusters/development
    sourceRef:
      kind: GitRepository
      name: flux2-sync
    prune: true

4. install secret as above
   kubectl apply -f flux-system.yaml

People could have multiple Kustomisations with dependsOn etc however we don't currently have that setup and i'm no expert on that yet. As a first cut of the eks addons you could just allow one kustomisation. Maybe your smarter and know how to do that. -)

I'm guessing that the current EKS addons are deployed via helm charts so this would be consistent with the others rather than trying to do flux bootstrap which eksctl seems to be trying to do. https://eksctl.io/usage/gitops-v2/. I think companies will much prefer the helm chart approach as it seems simpler and more reliable to upgrade and not dependent on access to the git repo even if it doesn't initialise the git repo first time. -)

[mikestef9]

CodeStar certainly won't be required, and the workflow you've outlined above will work just fine. Steps 1 and 2 would be handled through EKS add-ons. Additional steps you can still do yourself to start the sync process.

[hiddeco]

As a member of the Flux organization, I want to highlight that fluxcd-community != fluxcd. We endorse the projects under the fluxcd-community umbrella, but are not involved as maintainers (of Flux), nor do we provide official support for it.

This means AWS relying on a project under this umbrella may result in a burden for the maintainers there who are doing this on a "best effort" basis. Please try to keep this in mind while bringing Flux as an add-on to EKS, and maybe get in touch with the maintainers of the repository first.

[haarchri]

we are open for implementation in eks addons - feel free to open requests in fluxcd-community if we can support with stuff

// CC: @dwerder @haarchri

[cdenneen]

Currently we use Private GitLab for repos and Hosted JFrog for container registry. So wouldn't be using ECR for secrets but SM or SSM would be useful in connecting to these thirdparty backends.

[adaoud-ims]

we plan to use the CDK to provision EKS, Flux and CodeCommit through codepipeline.

I am wondering how Flux will be bootstrapped with CodeCommit ? Where the credentials will be ? in SSM ?
Do you think it will be possible to use CodeCommit without CodeStar ?

[landbaychrisburrell]

We are in the process of migrating our workloads as we speak. using fluxcd 2, with ECR for our OCI and docker registry, and image automation controllers with semver matching.

Currently we're bootstraping flux via its terraform provider. We're indeed looking to integration the notification controller via Slack and looking to use flagger for progressive delivery (though haven't yet prototyped this yet). Our monitoring will be through prometheus initially.

Happy to offer up more info on demand or via our account manager.