Add pvc helm charts for storage, update all s3 secret keys to be cons…

…istent, document backup/restores (#695) * add nextcloud PVCs helm chart * update nextcloud to use the pvc helm chart we just made * fix pvc enabled variable * change name of persistence app for nextcloud * set schedules sooner for test * switch to cronjob helm chart to use appset for setting schedules; update postgres and s3 pvc schedules to use secret vars * make matrix use PVC helm chart instead of PVCs directly * add configurable matrix schedule for backups * update matrix and nextcloud pvc appsets to point at feature branch * try settting creation policy to owner for external secret for backups * use feature branch for external secrets * update both files and config pvcs to always be enabled for now * fix nextcloud s3 pvc to point at feature branch and make master volume for swfs smaller * clean up serverInfoToken for nextcloud external secrets and add readme * update helm external secret server info token parameter * fix nextcloud server info token rendered external secret * fix maintenance mode schedule values * fix nextcloud cronjob helm chart appset * fix more maintenance types * add tip about potential zitadel failure with nextcloud * set maintanence mode for nextcloud to be 23:00 till 03:00AM * add default k8up dashboard for all of smol-k8s-lab default supported apps * update default k8up dashboard to have more default charts for at least nextcloud * make sure we only backup annotated PVCs for scheduled backups * remove create policy for s3 backup * add creation policy owner back to s3 backups for nextcloud * add nexcloud namespace * update the s3 backup credentials to be more consistent with what's expected * update postgres credentials naming scheme * fix nextcloud s3 credentials namign scheme * fix k8up backup secret for s3 provider helm chart * base64 encode the true value for the k8up secret * add configurable storageclass name * use global pvc storage class for nextcloud pvcs * shrink data pvc by default * add pvc enabled secret vars for nextcloud * take string values instead of booleans for enabled * switch to beta channel before installing certain apps * always enable oidc login * temporarily still point at feature branch for nextcloud s3 provider * make k8up operator use BACKUP_SKIP_WITHOUT_ANNOTATION=true env var, so ensure we don't backup ephemeral volumes * Update k8up_argocd_appset.yaml - use k8up.skipWithoutAnnotation * start attempts to backup nextcloud to local then remote 123 backup style * try new cors solution for webdav stuff as per nextcloud/helm#410 (comment) * try suggest for forwarding real ip with nextcloud suggest is here nextcloud/helm#410 (comment) * disable cors in a test and also remove robots.txt * convert s3 pvc appset in mastodon to use a valuesObject * switching back to normal pvc backup annotation * try dav with comma seperated allowed methods * disable cors again * try overriding the default nginx.conf with realip settings suggested in nextcloud/helm#410 * turn on cors again * try moving the real ip settings into the server block * try moving the real ip settings further down the server block * try to add real ip settings inside of location / * solve duplicate location "/" in /etc/nginx/conf.d/zz-custom.conf:142 * give up on nextcloud and try preserving real ip with https://stackoverflow.com/questions/66787939/preserving-source-ip-in-nginx-ingress-controller * try the solution from https://stackoverflow.com/a/68347429 for preserving source ip * disable cors again for nextcloud * try real ip cidr for nextcloud * that's all the gas left in the tank for nextcloud ingress testing 🤷 * add default phone region for nextcloud * add home assistant persistence and backups * fix getting home assistant pvc capacity * fix branch for pvc in home assistant * make sure we have s3 backup credentials ahead of time for home assistant * udpate external secrets for home assistant to use feature branch * annotate home assistant pvc for k8up backups * ignore restic env * fix typo for pvc scheduled backups and add some info on checking restic for home assistant backups and restores * remove user 82 for backup for home assistant * update ignore for restic env and add basic readme for backups and restore for home assistant * do consistant external secrets naming * more renaming of xternal_secrets_appset.yaml to external_secrets_argocd_appset.yaml everywhere * comment out the resources * update backups-s3-credentials secret to be s3-backups-credentials; change resticRepoPass to resticRepoPassword for matrix * use feature branch for external secrets for matrix * fix secretAccessKey and accessKeyID for matrix secrets * switch zitadel external secrets to use add-pvc-helm-chart-for-nextcloud feature branch * update mastodon s3 backup credentials * update zitadel secret from backups-s3-credentials to s3-backups-credentials * add more info about sample restore job * turn on generic device plugin again * k8up.io/backup: true added to all matrix pvcs * fix pvc enabled variables for matrix * fix access credentials for s3 backups * update zitadel s3 pvc to point at the right feature branch * switch to calling it s3-postgres-credentials instead of postgres-s3-credentials for nextcloud and matrix * add WAL compression and encryption * add WAL compression and encryption * add WAL compression and encryption * update matrix backups * do specify wal for matrix backups * allow max parellel for wal to be 8 * remove compression and adjust max parallel to 4 * removing tenant appsets b/c those live in thier own app dirs for instance nextcloud's tenant chart, now called cluster, is here: nextcloud/app_of_apps/postgres_argocd_appset.yaml * clean whitespace * only retain two days worth of postgres backups, so that you can just restore from seaweedfs if needed for matrix * updated max parallel to be 8 * update matrix to use the new default credentials for postgres and new schedule template values * add correct backup schedule for home assistant * update zitadel to use the new default credentials for postgres and new schedule template values * update nextcloud to use the new default credentials for postgres and new schedule template values * update mastodon to use the new default credentials for postgres and new schedule template values * use feature branch for matrix * clean up backup crendentials accessKeyID vs accessKeyId * update to always point at latest nextcloud 29 image * verify php occ is available before proceeding * update post install job to have an init container that checks for occ first * try once more to make nextcloud init prcoess work for installing apps * prettier logs for nextcloud install apps job * fix accessKeyId casing for nextcloud s3 pvc backups * rename zitadel external secrets appset to have the word external secrets * fix naming everywhere * update wal archive settings * fix s3 backup credentials access key id typo * switching back to main for all branches that were previously pointed at the feature branches --------- Co-authored-by: Max! <admin@cloudydev.net>
small-hack · May 15, 2024 · d48c2a8 · d48c2a8
1 parent 5d6de11
commit d48c2a8
Show file tree

Hide file tree

Showing 94 changed files with 3,406 additions and 475 deletions.
diff --git a/.../app_of_apps/external_secrets_appset.yaml → ..._apps/external_secrets_argocd_appset.yaml b/.../app_of_apps/external_secrets_appset.yaml → ..._apps/external_secrets_argocd_appset.yaml
diff --git a/demo/argo-workflows/app-of-apps/external_secret_appset.yaml b/demo/argo-workflows/app-of-apps/external_secret_appset.yaml
diff --git a/...stant/external_secrets_argocd_appset.yaml → ..._apps/external_secrets_argocd_appset.yaml b/...stant/external_secrets_argocd_appset.yaml → ..._apps/external_secrets_argocd_appset.yaml
diff --git a/...sistant/home_assistant_argocd_appset.yaml → ...of_apps/home_assistant_argocd_appset.yaml b/...sistant/home_assistant_argocd_appset.yaml → ...of_apps/home_assistant_argocd_appset.yaml
diff --git a/home-assistant/themes-config-map.yaml → ...istant/app_of_apps/themes-config-map.yaml b/home-assistant/themes-config-map.yaml → ...istant/app_of_apps/themes-config-map.yaml
diff --git a/home-assistant/backups_and_restores/.gitignore b/home-assistant/backups_and_restores/.gitignore
@@ -0,0 +1,3 @@
+.restic.env
+.restic-env
+.home-assistant-restic-password
diff --git a/home-assistant/backups_and_restores/.sample-restic-env b/home-assistant/backups_and_restores/.sample-restic-env
@@ -0,0 +1,10 @@
+# this contains both the s3 endpoint (this example uses b2, but you can use any s3 compliant endpoint) AND the s3 bucket
+export RESTIC_REPOSITORY="s3:s3.eu-central-003.backblazeb2.com/my-home-assistant-bucket"
+
+# Create this file, with a single line with your restic repo password. Make sure it's `chmod`ed to 600 and has only your user as the owner
+export RESTIC_PASSWORD_FILE=./.home-assistant-restic-password
+
+export AWS_ACCESS_KEY_ID="access key id goes here"
+export AWS_SECRET_ACCESS_KEY="secret key goes here"
+
+# after sourcing this file, you can do the following to test: restic snapshots
diff --git a/home-assistant/backups_and_restores/README.md b/home-assistant/backups_and_restores/README.md
@@ -0,0 +1,52 @@
+This app is already backed up on a schedule, however, you can also trigger a manual backup through the TUI.
+
+We've included an example `.sample-restic-env` that you can use to get started filling out the correct info for exporting your environment variables.
+
+```bash
+# copy the sample file, and then use vi, vim, nvim, or nano to edit
+# the file with your s3 credentials and restic password command
+cp .sample-restic-env .restic-env
+```
+
+You can also check your backups with
+
+```bash
+source .restic-env
+restic snapshots
+```
+
+We've also included a very basic restore job you can run in this directory to get started. Below are the values that you must change before you can run this restore job:
+
+This must be changed to YOUR S3 endpoint and home assistant backup bucket:
+```yaml
+value: s3:my.s3.endpoint/my-home-assistant-bucket
+```
+
+Replace the word TIMESTAMP with an actual timestamp:
+```yaml
+name: home-assistant-restic-restore-TIMESTAMP
+```
+
+Replace the snapshot ID "latest" with the ID of the snapshot you actually want to use:
+```yaml
+ - name: SNAPSHOT_ID
+   value: latest
+```
+
+If you're NOT using affinity and tolerations remove these two sections, otherwise, change the keys, values, operators, and effect to match your own needs:
+```yaml
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: home-assistant
+                operator: In
+                values:
+                - true
+      tolerations:
+        - effect: NoSchedule
+          key: home-assistant
+          operator: Equal
+          value: true
+```
diff --git a/home-assistant/backups_and_restores/restore_job.yaml b/home-assistant/backups_and_restores/restore_job.yaml
@@ -0,0 +1,67 @@
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: home-assistant-restic-restore-TIMESTAMP
+spec:
+  template:
+    metadata:
+      name: restic-restore-job
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: home-assistant
+                operator: In
+                values:
+                - true
+      tolerations:
+        - effect: NoSchedule
+          key: home-assistant
+          operator: Equal
+          value: true
+      volumes:
+        - name: restic-repo-password
+          secret:
+            secretName: s3-backups-credentials
+        - name: home-assistant
+          persistentVolumeClaim:
+            claimName: home-assistant
+      containers:
+        - name: restic-restore
+          image: instrumentisto/restic:latest
+          # runs: restic restore $SNAPSHOT_ID:/data/pvc-name --target config/
+          command:
+            - restic
+            - restore
+            - $SNAPSHOT_ID:/data/$PVC
+            - --target
+            - /config
+          volumeMounts:
+            - name: restic-repo-password
+              readOnly: true
+              mountPath: "/secrets/"
+            - name: home-assistant
+              mountPath: /config
+          env:
+            - name: SNAPSHOT_ID
+              value: latest
+            - name: PVC
+              value: home-assistant
+            - name: RESTIC_REPOSITORY
+              value: s3:my.s3.endpoint/my-home-assistant-bucket
+            - name: RESTIC_PASSWORD_FILE
+              value: "/secrets/resticRepoPassword"
+            - name: AWS_ACCESS_KEY_ID
+              valueFrom:
+                secretKeyRef:
+                  key: accessKeyID
+                  name: s3-backups-credentials
+            - name: AWS_SECRET_ACCESS_KEY
+              valueFrom:
+                secretKeyRef:
+                  key: secretAccessKey
+                  name: s3-backups-credentials
+      restartPolicy: Never
diff --git a/home-assistant/external_secrets/templates/bitwarden/s3_backup_credentials.yaml b/home-assistant/external_secrets/templates/bitwarden/s3_backup_credentials.yaml
@@ -0,0 +1,48 @@
+{{- if eq .Values.provider "bitwarden" }}
+---
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: s3-backups-credentials
+  namespace: home-assistant
+spec:
+  target:
+    name: s3-backups-credentials
+    deletionPolicy: Delete
+    template:
+      type: Opaque
+      data:
+        accessKeyID: |-
+          {{ `{{ .accessKey }}` }}
+        secretAccessKey: |-
+          {{ `{{ .secretKey }}` }}
+        resticRepoPassword: |-
+          {{ `{{ .resticRepoPass }}` }}
+  data:
+    - secretKey: accessKey
+      sourceRef:
+        storeRef:
+          name: bitwarden-login
+          kind: ClusterSecretStore
+      remoteRef:
+        key: {{ .Values.s3BackupCredentialsBitwardenID }}
+        property: username
+
+    - secretKey: secretKey
+      sourceRef:
+        storeRef:
+          name: bitwarden-login
+          kind: ClusterSecretStore
+      remoteRef:
+        key: {{ .Values.s3BackupCredentialsBitwardenID }}
+        property: password
+
+    - secretKey: resticRepoPass
+      sourceRef:
+        storeRef:
+          name: bitwarden-fields
+          kind: ClusterSecretStore
+      remoteRef:
+        key: {{ .Values.s3BackupCredentialsBitwardenID }}
+        property: resticRepoPassword
+{{- end }}
diff --git a/home-assistant/external_secrets/values.yaml b/home-assistant/external_secrets/values.yaml
@@ -4,3 +4,6 @@ provider: "true"
 
 # the item ID of the Bitwarden admin credentials
 bitwardenAdminCredentialsID: ""
+
+# -- existing kubernetes secret with s3 credentials for the remote backups
+s3BackupCredentialsBitwardenID: ""
diff --git a/home-assistant/storage/Chart.yaml b/home-assistant/storage/Chart.yaml
@@ -0,0 +1,24 @@
+apiVersion: v2
+name: home-assistance-persistence-chart
+description: A Helm chart for deploying a home assistant pvc on Kubernetes including backups using k8up
+
+# A chart can be either an 'application' or a 'library' chart.
+#
+# Application charts are a collection of templates that can be packaged into versioned archives
+# to be deployed.
+#
+# Library charts provide useful utilities or functions for the chart developer. They're included as
+# a dependency of application charts to inject those utilities and functions into the rendering
+# pipeline. Library charts do not define any templates and therefore cannot be deployed.
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 1.0.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "1.0.0"
diff --git a/home-assistant/storage/README.md b/home-assistant/storage/README.md
@@ -0,0 +1,29 @@
+# home-assistance-persistence-chart
+
+![Version: 1.0.0](https://img.shields.io/badge/Version-1.0.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.0.0](https://img.shields.io/badge/AppVersion-1.0.0-informational?style=flat-square)
+
+A Helm chart for deploying a home assistant pvc on Kubernetes including backups using k8up
+
+## Values
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| k8up | object | `{"backup_name":"home-assistant-nightly","backup_type":"s3","local":{"mountPath":""},"podSecurityContext":{"runAsUser":0},"prometheus_url":"","repoPasswordSecretRef":{"key":"","name":""},"s3":{"accessKeyIDSecretRef":{"key":"","name":"","optional":false},"bucket":"","endpoint":"","secretAccessKeySecretRef":{"key":"","name":"","optional":false}},"schedules":{"backup":"","check":"","prune":""}}` | for enabling backups to a remote s3 provider or local disk backup |
+| k8up.backup_type | string | `"s3"` | can be set to 's3' or 'local' |
+| k8up.podSecurityContext | object | `{"runAsUser":0}` | user to run the backups as |
+| k8up.prometheus_url | string | `""` | url to push to for prometheus gateway |
+| k8up.repoPasswordSecretRef | object | `{"key":"","name":""}` | secret for your restic repo |
+| k8up.repoPasswordSecretRef.key | string | `""` | key in secret to use for repo password |
+| k8up.repoPasswordSecretRef.name | string | `""` | name of the secret to use |
+| k8up.s3.accessKeyIDSecretRef.key | string | `""` | key in the secret to use for access key id |
+| k8up.s3.accessKeyIDSecretRef.name | string | `""` | name of the secret to use |
+| k8up.s3.bucket | string | `""` | s3 bucket to backup to |
+| k8up.s3.endpoint | string | `""` | s3 endpoint to backup to |
+| k8up.s3.secretAccessKeySecretRef.key | string | `""` | key in the secret to use for secret access key |
+| k8up.s3.secretAccessKeySecretRef.name | string | `""` | name of the secret to use |
+| k8up.schedules | object | `{"backup":"","check":"","prune":""}` | schedules for backups, checks, and prunes |
+| pvc_capacity | string | `"10Gi"` |  |
+| pvc_storageClassName | string | `"local-path"` |  |
+
+----------------------------------------------
+Autogenerated from chart metadata using [helm-docs v1.13.1](https://github.com/norwoodj/helm-docs/releases/v1.13.1)
diff --git a/home-assistant/pvc.md → home-assistant/storage/templates/pvc.yaml b/home-assistant/pvc.md → home-assistant/storage/templates/pvc.yaml
@@ -5,9 +5,12 @@ kind: PersistentVolumeClaim
 metadata:
   namespace: home-assistant
   name: home-assistant
+  annotations:
+    k8up.io/backup: 'true'
 spec:
+  storageClassName: {{ .Values.pvc_storageClassName }}
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
-      storage: 15Gi
+      storage: {{ .Values.pvc_capacity }}
diff --git a/home-assistant/storage/templates/scheduled_backups.yaml b/home-assistant/storage/templates/scheduled_backups.yaml
@@ -0,0 +1,50 @@
+---
+apiVersion: k8up.io/v1
+kind: Schedule
+metadata:
+  name: {{ .Values.k8up.backup_name }}
+  annotations:
+    "helm.sh/hook": post-install
+    "helm.sh/resource-policy": keep
+spec:
+  backend:
+    repoPasswordSecretRef:
+      name: {{ .Values.k8up.repoPasswordSecretRef.name }}
+      key: {{ .Values.k8up.repoPasswordSecretRef.key }}
+    {{- if eq .Values.k8up.backup_type "s3" }}
+    s3:
+      endpoint: {{ .Values.k8up.s3.endpoint }}
+      bucket: {{ .Values.k8up.s3.bucket }}
+      accessKeyIDSecretRef:
+        name: {{ .Values.k8up.s3.accessKeyIDSecretRef.name }}
+        key: {{ .Values.k8up.s3.accessKeyIDSecretRef.key }}
+        optional: {{ .Values.k8up.s3.accessKeyIDSecretRef.optional }}
+      secretAccessKeySecretRef:
+        name: {{ .Values.k8up.s3.secretAccessKeySecretRef.name }}
+        key: {{ .Values.k8up.s3.secretAccessKeySecretRef.key }}
+        optional: {{ .Values.k8up.s3.secretAccessKeySecretRef.optional }}
+    {{- end }}
+    {{- if eq .Values.k8up.backup_type "local" }}
+    local:
+      mountPath: {{ .Values.k8up.local.mountPath }}
+    {{- end }}
+  # Backup nightly at scheduled time (Central Europe Time)
+  backup:
+    schedule: {{ .Values.k8up.schedules.backup }}
+    failedJobsHistoryLimit: 5
+    successfulJobsHistoryLimit: 2
+    promURL: {{ .Values.k8up.prometheus_url }}
+    {{- if .Values.k8up.podSecurityContext }}
+    podSecurityContext:
+      runAsUser: {{ .Values.k8up.podSecurityContext.runAsUser }}
+    {{- end }}
+  # verify the backups are ok
+  check:
+    schedule: {{ .Values.k8up.schedules.check }}
+    promURL: {{ .Values.k8up.prometheus_url }}
+  # delete old backups
+  prune:
+    schedule: {{ .Values.k8up.schedules.prune }}
+    retention:
+      keepDaily: 7
+      keepWeekly: 8