This repo is a clone of the fluxcd example repo that has been updated to work with multi-tenancy. Azure GitOps enables Flux multi-tenancy by default, thus this example repo can be used for simple proof of concept following this tutorial.
This repo is a tutorial repository and does not come with any guarantees pertaining to breaking changes. Version upgrades may be performed to this repository that may cause breaks on upgrades. In cases where installation of helm charts or Kubernetes manifests upgrades fail, you may have to manually purge the releases from the cluster and allow the HelmRelease to re-reconcile the fresh install of the chart.
helm delete -n <namespace> <helm-chart-name>
flux reconcile helmrelease -n <helmrelease-namespace> <helmrelease-name>
Optionally, you may also re-create the fluxConfiguration to re-deploy all of the deployed manifests from start.
az k8s-configuration flux delete -g flux-demo-rg -c flux-demo-arc -n cluster-config --namespace cluster-config -t connectedClusters
az k8s-configuration flux create -g flux-demo-rg -c flux-demo-arc -n cluster-config --namespace cluster-config -t connectedClusters --scope cluster -u https://github.com/Azure/gitops-flux2-kustomize-helm-mt --branch main --kustomization name=infra path=./infrastructure prune=true --kustomization name=apps path=./apps/staging prune=true dependsOn=["infra"]
Redhat Openshift clusters impose security context constraints on all containers that run on the Kubernetes cluster. For all containers in the example application to run, you will need to add the following SCCs to your cluster
oc adm policy add-scc-to-user privileged system:serviceaccount:nginx:nginx-nginx-nginx-ingress-controller
oc adm policy add-scc-to-user nonroot system:serviceaccount:redis:default
For this example we assume a scenario with two clusters: staging and production. The end goal is to leverage Flux and Kustomize to manage both clusters while minimizing duplicated declarations.
We will configure Flux to install, test and upgrade a demo app using
HelmRepository
and HelmRelease
custom resources.
Flux will monitor the Helm repository, and it will automatically
upgrade the Helm releases to their latest chart version based on semver ranges.
You will need a Kubernetes cluster version 1.16 or newer and kubectl version 1.18. For a quick local test, you can use Kubernetes kind. Any other Kubernetes setup will work as well though.
In order to follow the guide you'll need a GitHub account and a
personal access token
that can create repositories (check all permissions under repo
).
Install the Flux CLI on MacOS and Linux using Homebrew:
brew install fluxcd/tap/flux
Or install the CLI by downloading precompiled binaries using a Bash script:
curl -s https://fluxcd.io/install.sh | sudo bash
The Git repository contains the following top directories:
- apps dir contains Helm releases with a custom configuration per cluster
- infrastructure dir contains common infra tools such as NGINX ingress controller and Helm repository definitions
- clusters dir contains the Flux configuration per cluster
├── apps
│ ├── base
│ ├── production
│ └── staging
├── infrastructure
│ ├── nginx
│ ├── redis
│ └── sources
└── clusters
├── production
└── staging
The apps configuration is structured into:
- apps/base/ dir contains namespaces and Helm release definitions
- apps/production/ dir contains the production Helm release values
- apps/staging/ dir contains the staging values
./apps/
├── base
│ └── podinfo
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
├── production
│ ├── kustomization.yaml
│ └── podinfo-patch.yaml
└── staging
├── kustomization.yaml
└── podinfo-patch.yaml
In apps/base/podinfo/ dir we have a HelmRelease with common values for both clusters:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: podinfo
namespace: podinfo
spec:
releaseName: podinfo
chart:
spec:
chart: podinfo
sourceRef:
kind: HelmRepository
name: podinfo
namespace: flux-system
interval: 5m
values:
cache: redis-master.redis:6379
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
In apps/staging/ dir we have a Kustomize patch with the staging specific values:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: podinfo
spec:
chart:
spec:
version: ">=1.0.0-alpha"
test:
enable: true
values:
ingress:
hosts:
- host: podinfo.staging
Note that with version: ">=1.0.0-alpha"
we configure Flux to automatically upgrade
the HelmRelease
to the latest chart version including alpha, beta and pre-releases.
In apps/production/ dir we have a Kustomize patch with the production specific values:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: podinfo
namespace: podinfo
spec:
chart:
spec:
version: ">=1.0.0"
values:
ingress:
hosts:
- host: podinfo.production
Note that with version: ">=1.0.0"
we configure Flux to automatically upgrade
the HelmRelease
to the latest stable chart version (alpha, beta and pre-releases will be ignored).
Infrastructure:
./infrastructure/
├── nginx
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
├── redis
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
└── sources
├── bitnami.yaml
├── kustomization.yaml
└── podinfo.yaml
In infrastructure/sources/ dir we have the Helm repositories definitions:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: podinfo
spec:
interval: 5m
url: https://stefanprodan.github.io/podinfo
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: bitnami
spec:
interval: 30m
url: https://charts.bitnami.com/bitnami
Note that with interval: 5m
we configure Flux to pull the Helm repository index every five minutes.
If the index contains a new chart version that matches a HelmRelease
semver range, Flux will upgrade the release.
The clusters dir contains the Flux configuration:
./clusters/
├── production
│ ├── apps.yaml
│ └── infrastructure.yaml
└── staging
├── apps.yaml
└── infrastructure.yaml
In clusters/staging/ dir we have the Kustomization definitions:
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
interval: 10m0s
dependsOn:
- name: infrastructure
sourceRef:
kind: GitRepository
name: flux-system
path: ./apps/staging
prune: true
wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infrastructure
namespace: flux-system
spec:
interval: 10m0s
sourceRef:
kind: GitRepository
name: flux-system
path: ./infrastructure
prune: true
Note that with path: ./apps/staging
we configure Flux to sync the staging Kustomize overlay and
with dependsOn
we tell Flux to create the infrastructure items before deploying the apps.
Fork this repository on your personal GitHub account and export your GitHub access token, username and repo name:
export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<repository-name>
Verify that your staging cluster satisfies the prerequisites with:
flux check --pre
Set the kubectl context to your staging cluster and bootstrap Flux:
flux bootstrap github \
--context=staging \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--branch=main \
--personal \
--path=clusters/staging
The bootstrap command commits the manifests for the Flux components in clusters/staging/flux-system
dir
and creates a deploy key with read-only access on GitHub, so it can pull changes inside the cluster.
Watch for the Helm releases being install on staging:
$ watch flux get helmreleases --all-namespaces
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
nginx nginx 5.6.14 False True release reconciliation succeeded
podinfo podinfo 5.0.3 False True release reconciliation succeeded
redis redis 11.3.4 False True release reconciliation succeeded
Verify that the demo app can be accessed via ingress:
$ kubectl -n nginx port-forward svc/nginx-ingress-controller 8080:80 &
$ curl -H "Host: podinfo.staging" http://localhost:8080
{
"hostname": "podinfo-59489db7b5-lmwpn",
"version": "5.0.3"
}
Bootstrap Flux on production by setting the context and path to your production cluster:
flux bootstrap github \
--context=production \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--branch=main \
--personal \
--path=clusters/production
Watch the production reconciliation:
$ flux get kustomizations --watch
NAME REVISION READY
apps main/797cd90cc8e81feb30cfe471a5186b86daf2758d True
flux-system main/797cd90cc8e81feb30cfe471a5186b86daf2758d True
infrastructure main/797cd90cc8e81feb30cfe471a5186b86daf2758d True
In order to store secrets safely in a Git repository, you can use Mozilla's SOPS CLI to encrypt Kubernetes secrets with OpenPGP or KMS.
brew install gnupg sops
Generate a GPG key for Flux without specifying a passphrase and retrieve the GPG key ID:
$ gpg --full-generate-key
Email address: fluxcdbot@users.noreply.github.com
$ gpg --list-secret-keys fluxcdbot@users.noreply.github.com
sec rsa3072 2020-09-06 [SC]
1F3D1CED2F865F5E59CA564553241F147E7C5FA4
Create a Kubernetes secret on your clusters with the private key:
gpg --export-secret-keys \
--armor 1F3D1CED2F865F5E59CA564553241F147E7C5FA4 |
kubectl create secret generic sops-gpg \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin
Generate a Kubernetes secret manifest and encrypt the secret's data field with sops:
kubectl -n redis create secret generic redis-auth \
--from-literal=password=change-me \
--dry-run=client \
-o yaml > infrastructure/redis/redis-auth.yaml
sops --encrypt \
--pgp=1F3D1CED2F865F5E59CA564553241F147E7C5FA4 \
--encrypted-regex '^(data|stringData)$' \
--in-place infrastructure/redis/redis-auth.yaml
Add the secret to infrastructure/redis/kustomization.yaml
:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: redis
resources:
- namespace.yaml
- release.yaml
- redis-auth.yaml
Enable decryption on your clusters by editing the infrastructure.yaml
files:
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: infrastructure
namespace: flux-system
spec:
# content omitted for brevity
decryption:
provider: sops
secretRef:
name: sops-gpg
Export the public key so anyone with access to the repository can encrypt secrets but not decrypt them:
gpg --export -a fluxcdbot@users.noreply.github.com > public.key
Push the changes to the main branch:
git add -A && git commit -m "add encrypted secret" && git push
Verify that the secret has been created in the redis
namespace on both clusters:
kubectl --context staging -n redis get secrets
kubectl --context production -n redis get secrets
You can use Kubernetes secrets to provide values for your Helm releases:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: redis
spec:
# content omitted for brevity
values:
usePassword: true
valuesFrom:
- kind: Secret
name: redis-auth
valuesKey: password
targetPath: password
Find out more about Helm releases values overrides in the docs.
If you want to add a cluster to your fleet, first clone your repo locally:
git clone https://github.com/${GITHUB_USER}/${GITHUB_REPO}.git
cd ${GITHUB_REPO}
Create a dir inside clusters
with your cluster name:
mkdir -p clusters/dev
Copy the sync manifests from staging:
cp clusters/staging/infrastructure.yaml clusters/dev
cp clusters/staging/apps.yaml clusters/dev
You could create a dev overlay inside apps
, make sure
to change the spec.path
inside clusters/dev/apps.yaml
to path: ./apps/dev
.
Push the changes to the main branch:
git add -A && git commit -m "add dev cluster" && git push
Set the kubectl context and path to your dev cluster and bootstrap Flux:
flux bootstrap github \
--context=dev \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--branch=main \
--personal \
--path=clusters/dev
If you want to spin up an identical environment, you can bootstrap a cluster
e.g. production-clone
and reuse the production
definitions.
Bootstrap the production-clone
cluster:
flux bootstrap github \
--context=production-clone \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--branch=main \
--personal \
--path=clusters/production-clone
Pull the changes locally:
git pull origin main
Create a kustomization.yaml
inside the clusters/production-clone
dir:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- flux-system
- ../production/infrastructure.yaml
- ../production/apps.yaml
Note that besides the flux-system
kustomize overlay, we also include
the infrastructure
and apps
manifests from the production dir.
Push the changes to the main branch:
git add -A && git commit -m "add production clone" && git push
Tell Flux to deploy the production workloads on the production-clone
cluster:
flux reconcile kustomization flux-system \
--context=production-clone \
--with-source
Any change to the Kubernetes manifests or to the repository structure should be validated in CI before a pull requests is merged into the main branch and synced on the cluster.
This repository contains the following GitHub CI workflows:
- the test workflow validates the Kubernetes manifests and Kustomize overlays with kubeconform
- the e2e workflow starts a Kubernetes cluster in CI and tests the staging setup by running Flux in Kubernetes Kind