Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pvc helm charts for storage, update all s3 secret keys to be consistent, document backup/restores #695

Merged
merged 117 commits into from
May 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
a5140c8
add nextcloud PVCs helm chart
jessebot Apr 10, 2024
7b11334
update nextcloud to use the pvc helm chart we just made
jessebot Apr 10, 2024
d3ffed9
fix pvc enabled variable
jessebot Apr 10, 2024
63ba21b
change name of persistence app for nextcloud
jessebot Apr 10, 2024
b98af81
set schedules sooner for test
jessebot Apr 28, 2024
a30a6ec
switch to cronjob helm chart to use appset for setting schedules; upd…
jessebot Apr 28, 2024
590066a
make matrix use PVC helm chart instead of PVCs directly
jessebot Apr 11, 2024
03cffd5
add configurable matrix schedule for backups
jessebot Apr 11, 2024
bc9b4e2
update matrix and nextcloud pvc appsets to point at feature branch
jessebot Apr 12, 2024
da76873
try settting creation policy to owner for external secret for backups
jessebot Apr 12, 2024
41fee1a
use feature branch for external secrets
jessebot Apr 12, 2024
252956c
update both files and config pvcs to always be enabled for now
jessebot Apr 12, 2024
5379eaa
fix nextcloud s3 pvc to point at feature branch and make master volum…
jessebot Apr 13, 2024
9bc9a0c
clean up serverInfoToken for nextcloud external secrets and add readme
jessebot Apr 13, 2024
d544b50
update helm external secret server info token parameter
jessebot Apr 13, 2024
5a3cef9
fix nextcloud server info token rendered external secret
jessebot Apr 13, 2024
f95865b
fix maintenance mode schedule values
jessebot Apr 13, 2024
a90857b
fix nextcloud cronjob helm chart appset
jessebot Apr 13, 2024
bf5c8e3
fix more maintenance types
jessebot Apr 13, 2024
a599330
add tip about potential zitadel failure with nextcloud
jessebot Apr 13, 2024
fc04e32
set maintanence mode for nextcloud to be 23:00 till 03:00AM
jessebot Apr 13, 2024
224a089
add default k8up dashboard for all of smol-k8s-lab default supported …
jessebot Apr 25, 2024
f5647d1
update default k8up dashboard to have more default charts for at leas…
jessebot Apr 26, 2024
75c6244
make sure we only backup annotated PVCs for scheduled backups
jessebot Apr 26, 2024
1cda60e
remove create policy for s3 backup
jessebot Apr 26, 2024
06cd98c
add creation policy owner back to s3 backups for nextcloud
jessebot Apr 26, 2024
42c250a
add nexcloud namespace
jessebot Apr 26, 2024
5e0b0ef
update the s3 backup credentials to be more consistent with what's ex…
jessebot Apr 27, 2024
3783382
update postgres credentials naming scheme
jessebot Apr 27, 2024
27f15d1
fix nextcloud s3 credentials namign scheme
jessebot Apr 27, 2024
0984e32
fix k8up backup secret for s3 provider helm chart
jessebot Apr 27, 2024
f17fd06
base64 encode the true value for the k8up secret
jessebot Apr 27, 2024
4e5536d
add configurable storageclass name
jessebot Apr 27, 2024
8a1ce81
use global pvc storage class for nextcloud pvcs
jessebot Apr 27, 2024
6e41b73
shrink data pvc by default
jessebot Apr 27, 2024
dfd306d
add pvc enabled secret vars for nextcloud
jessebot Apr 27, 2024
54a49c0
take string values instead of booleans for enabled
jessebot Apr 27, 2024
de114ef
switch to beta channel before installing certain apps
jessebot Apr 27, 2024
c505667
always enable oidc login
jessebot Apr 27, 2024
b8fcabd
temporarily still point at feature branch for nextcloud s3 provider
jessebot Apr 28, 2024
54b6408
make k8up operator use BACKUP_SKIP_WITHOUT_ANNOTATION=true env var, s…
jessebot Apr 28, 2024
57bd5ed
Update k8up_argocd_appset.yaml - use k8up.skipWithoutAnnotation
jessebot Apr 30, 2024
21a903d
start attempts to backup nextcloud to local then remote 123 backup style
jessebot Apr 30, 2024
84def2f
try new cors solution for webdav stuff
jessebot Apr 30, 2024
e002f0f
try suggest for forwarding real ip with nextcloud
jessebot Apr 30, 2024
3dab095
disable cors in a test and also remove robots.txt
jessebot Apr 30, 2024
26d55c5
convert s3 pvc appset in mastodon to use a valuesObject
jessebot May 1, 2024
1fedb5e
switching back to normal pvc backup annotation
jessebot May 1, 2024
be03dab
try dav with comma seperated allowed methods
jessebot May 1, 2024
6cd89a6
disable cors again
jessebot May 1, 2024
a261f6d
try overriding the default nginx.conf with realip settings suggested …
jessebot May 1, 2024
5c5bf44
turn on cors again
jessebot May 1, 2024
fc6ce9d
try moving the real ip settings into the server block
jessebot May 1, 2024
67dd326
try moving the real ip settings further down the server block
jessebot May 1, 2024
9cb4ce7
try to add real ip settings inside of location /
jessebot May 1, 2024
e0a8517
solve duplicate location "/" in /etc/nginx/conf.d/zz-custom.conf:142
jessebot May 1, 2024
56071b9
give up on nextcloud and try preserving real ip with https://stackove…
jessebot May 1, 2024
b8c01cc
try the solution from https://stackoverflow.com/a/68347429 for preser…
jessebot May 1, 2024
92bbdb1
disable cors again for nextcloud
jessebot May 1, 2024
b778b1f
try real ip cidr for nextcloud
jessebot May 1, 2024
b6e12ac
that's all the gas left in the tank for nextcloud ingress testing :sh…
jessebot May 1, 2024
8e61cf1
add default phone region for nextcloud
jessebot May 2, 2024
d7fffa8
add home assistant persistence and backups
jessebot May 2, 2024
95ef6e9
fix getting home assistant pvc capacity
jessebot May 2, 2024
f560203
fix branch for pvc in home assistant
jessebot May 2, 2024
5dd64df
make sure we have s3 backup credentials ahead of time for home assistant
jessebot May 2, 2024
4da81f9
udpate external secrets for home assistant to use feature branch
jessebot May 2, 2024
37636ef
annotate home assistant pvc for k8up backups
jessebot May 2, 2024
b0921f1
ignore restic env
jessebot May 2, 2024
f8da5ab
fix typo for pvc scheduled backups and add some info on checking rest…
jessebot May 2, 2024
8b9c0ff
remove user 82 for backup for home assistant
jessebot May 2, 2024
b66e4a6
update ignore for restic env and add basic readme for backups and res…
jessebot May 2, 2024
c01d066
do consistant external secrets naming
jessebot May 3, 2024
dc77d54
more renaming of xternal_secrets_appset.yaml to external_secrets_argo…
jessebot May 12, 2024
4e6bdb3
comment out the resources
jessebot May 3, 2024
f5c4a5a
update backups-s3-credentials secret to be s3-backups-credentials; ch…
jessebot May 3, 2024
11c6200
use feature branch for external secrets for matrix
jessebot May 3, 2024
836ac5e
fix secretAccessKey and accessKeyID for matrix secrets
jessebot May 3, 2024
5d2cff7
switch zitadel external secrets to use add-pvc-helm-chart-for-nextclo…
jessebot May 3, 2024
8f6a077
update mastodon s3 backup credentials
jessebot May 3, 2024
70710f5
update zitadel secret from backups-s3-credentials to s3-backups-crede…
jessebot May 3, 2024
3b3042a
add more info about sample restore job
jessebot May 4, 2024
120d680
turn on generic device plugin again
jessebot May 4, 2024
96fa81b
k8up.io/backup: true added to all matrix pvcs
jessebot May 5, 2024
b421771
fix pvc enabled variables for matrix
jessebot May 5, 2024
5628717
fix access credentials for s3 backups
jessebot May 5, 2024
f391b52
update zitadel s3 pvc to point at the right feature branch
jessebot May 5, 2024
8ff16b1
switch to calling it s3-postgres-credentials instead of postgres-s3-c…
jessebot May 6, 2024
c648dcd
add WAL compression and encryption
cloudymax May 6, 2024
9c2b7ab
add WAL compression and encryption
cloudymax May 6, 2024
41036bb
add WAL compression and encryption
cloudymax May 6, 2024
a1d3d78
update matrix backups
jessebot May 7, 2024
a6296f0
do specify wal for matrix backups
jessebot May 7, 2024
1e19185
allow max parellel for wal to be 8
jessebot May 7, 2024
e9b32d7
remove compression and adjust max parallel to 4
jessebot May 7, 2024
af2d699
removing tenant appsets b/c those live in thier own app dirs
jessebot May 7, 2024
7f03c30
clean whitespace
jessebot May 7, 2024
6f21aaa
only retain two days worth of postgres backups, so that you can just …
jessebot May 7, 2024
04666f6
updated max parallel to be 8
jessebot May 7, 2024
81d212b
update matrix to use the new default credentials for postgres and new…
jessebot May 8, 2024
baec9c4
add correct backup schedule for home assistant
jessebot May 8, 2024
87c80fe
update zitadel to use the new default credentials for postgres and ne…
jessebot May 8, 2024
d1f6207
update nextcloud to use the new default credentials for postgres and …
jessebot May 8, 2024
456f390
update mastodon to use the new default credentials for postgres and n…
jessebot May 8, 2024
b96bd77
use feature branch for matrix
jessebot May 8, 2024
3fd7be7
clean up backup crendentials accessKeyID vs accessKeyId
jessebot May 8, 2024
9e1ca72
update to always point at latest nextcloud 29 image
jessebot May 9, 2024
158b86a
verify php occ is available before proceeding
jessebot May 9, 2024
3f6ad39
update post install job to have an init container that checks for occ…
jessebot May 9, 2024
0758d42
try once more to make nextcloud init prcoess work for installing apps
jessebot May 9, 2024
5cb0317
prettier logs for nextcloud install apps job
jessebot May 9, 2024
e6a4191
fix accessKeyId casing for nextcloud s3 pvc backups
jessebot May 9, 2024
77bbc8e
rename zitadel external secrets appset to have the word external secrets
jessebot May 9, 2024
c343406
fix naming everywhere
jessebot May 9, 2024
41e1a7d
update wal archive settings
jessebot May 9, 2024
0887732
fix s3 backup credentials access key id typo
jessebot May 10, 2024
ffbe736
switching back to main for all branches that were previously pointed …
jessebot May 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 0 additions & 42 deletions demo/argo-workflows/app-of-apps/external_secret_appset.yaml

This file was deleted.

3 changes: 3 additions & 0 deletions home-assistant/backups_and_restores/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.restic.env
.restic-env
.home-assistant-restic-password
10 changes: 10 additions & 0 deletions home-assistant/backups_and_restores/.sample-restic-env
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# this contains both the s3 endpoint (this example uses b2, but you can use any s3 compliant endpoint) AND the s3 bucket
export RESTIC_REPOSITORY="s3:s3.eu-central-003.backblazeb2.com/my-home-assistant-bucket"

# Create this file, with a single line with your restic repo password. Make sure it's `chmod`ed to 600 and has only your user as the owner
export RESTIC_PASSWORD_FILE=./.home-assistant-restic-password

export AWS_ACCESS_KEY_ID="access key id goes here"
export AWS_SECRET_ACCESS_KEY="secret key goes here"

# after sourcing this file, you can do the following to test: restic snapshots
52 changes: 52 additions & 0 deletions home-assistant/backups_and_restores/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
This app is already backed up on a schedule, however, you can also trigger a manual backup through the TUI.

We've included an example `.sample-restic-env` that you can use to get started filling out the correct info for exporting your environment variables.

```bash
# copy the sample file, and then use vi, vim, nvim, or nano to edit
# the file with your s3 credentials and restic password command
cp .sample-restic-env .restic-env
```

You can also check your backups with

```bash
source .restic-env
restic snapshots
```

We've also included a very basic restore job you can run in this directory to get started. Below are the values that you must change before you can run this restore job:

This must be changed to YOUR S3 endpoint and home assistant backup bucket:
```yaml
value: s3:my.s3.endpoint/my-home-assistant-bucket
```

Replace the word TIMESTAMP with an actual timestamp:
```yaml
name: home-assistant-restic-restore-TIMESTAMP
```

Replace the snapshot ID "latest" with the ID of the snapshot you actually want to use:
```yaml
- name: SNAPSHOT_ID
value: latest
```

If you're NOT using affinity and tolerations remove these two sections, otherwise, change the keys, values, operators, and effect to match your own needs:
```yaml
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: home-assistant
operator: In
values:
- true
tolerations:
- effect: NoSchedule
key: home-assistant
operator: Equal
value: true
```
67 changes: 67 additions & 0 deletions home-assistant/backups_and_restores/restore_job.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
apiVersion: batch/v1
kind: Job
metadata:
name: home-assistant-restic-restore-TIMESTAMP
spec:
template:
metadata:
name: restic-restore-job
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: home-assistant
operator: In
values:
- true
tolerations:
- effect: NoSchedule
key: home-assistant
operator: Equal
value: true
volumes:
- name: restic-repo-password
secret:
secretName: s3-backups-credentials
- name: home-assistant
persistentVolumeClaim:
claimName: home-assistant
containers:
- name: restic-restore
image: instrumentisto/restic:latest
# runs: restic restore $SNAPSHOT_ID:/data/pvc-name --target config/
command:
- restic
- restore
- $SNAPSHOT_ID:/data/$PVC
- --target
- /config
volumeMounts:
- name: restic-repo-password
readOnly: true
mountPath: "/secrets/"
- name: home-assistant
mountPath: /config
env:
- name: SNAPSHOT_ID
value: latest
- name: PVC
value: home-assistant
- name: RESTIC_REPOSITORY
value: s3:my.s3.endpoint/my-home-assistant-bucket
- name: RESTIC_PASSWORD_FILE
value: "/secrets/resticRepoPassword"
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: accessKeyID
name: s3-backups-credentials
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: secretAccessKey
name: s3-backups-credentials
restartPolicy: Never
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{{- if eq .Values.provider "bitwarden" }}
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: s3-backups-credentials
namespace: home-assistant
spec:
target:
name: s3-backups-credentials
deletionPolicy: Delete
template:
type: Opaque
data:
accessKeyID: |-
{{ `{{ .accessKey }}` }}
secretAccessKey: |-
{{ `{{ .secretKey }}` }}
resticRepoPassword: |-
{{ `{{ .resticRepoPass }}` }}
data:
- secretKey: accessKey
sourceRef:
storeRef:
name: bitwarden-login
kind: ClusterSecretStore
remoteRef:
key: {{ .Values.s3BackupCredentialsBitwardenID }}
property: username

- secretKey: secretKey
sourceRef:
storeRef:
name: bitwarden-login
kind: ClusterSecretStore
remoteRef:
key: {{ .Values.s3BackupCredentialsBitwardenID }}
property: password

- secretKey: resticRepoPass
sourceRef:
storeRef:
name: bitwarden-fields
kind: ClusterSecretStore
remoteRef:
key: {{ .Values.s3BackupCredentialsBitwardenID }}
property: resticRepoPassword
{{- end }}
3 changes: 3 additions & 0 deletions home-assistant/external_secrets/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,6 @@ provider: "true"

# the item ID of the Bitwarden admin credentials
bitwardenAdminCredentialsID: ""

# -- existing kubernetes secret with s3 credentials for the remote backups
s3BackupCredentialsBitwardenID: ""
24 changes: 24 additions & 0 deletions home-assistant/storage/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
apiVersion: v2
name: home-assistance-persistence-chart
description: A Helm chart for deploying a home assistant pvc on Kubernetes including backups using k8up

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 1.0.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.0.0"
29 changes: 29 additions & 0 deletions home-assistant/storage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# home-assistance-persistence-chart

![Version: 1.0.0](https://img.shields.io/badge/Version-1.0.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.0.0](https://img.shields.io/badge/AppVersion-1.0.0-informational?style=flat-square)

A Helm chart for deploying a home assistant pvc on Kubernetes including backups using k8up

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| k8up | object | `{"backup_name":"home-assistant-nightly","backup_type":"s3","local":{"mountPath":""},"podSecurityContext":{"runAsUser":0},"prometheus_url":"","repoPasswordSecretRef":{"key":"","name":""},"s3":{"accessKeyIDSecretRef":{"key":"","name":"","optional":false},"bucket":"","endpoint":"","secretAccessKeySecretRef":{"key":"","name":"","optional":false}},"schedules":{"backup":"","check":"","prune":""}}` | for enabling backups to a remote s3 provider or local disk backup |
| k8up.backup_type | string | `"s3"` | can be set to 's3' or 'local' |
| k8up.podSecurityContext | object | `{"runAsUser":0}` | user to run the backups as |
| k8up.prometheus_url | string | `""` | url to push to for prometheus gateway |
| k8up.repoPasswordSecretRef | object | `{"key":"","name":""}` | secret for your restic repo |
| k8up.repoPasswordSecretRef.key | string | `""` | key in secret to use for repo password |
| k8up.repoPasswordSecretRef.name | string | `""` | name of the secret to use |
| k8up.s3.accessKeyIDSecretRef.key | string | `""` | key in the secret to use for access key id |
| k8up.s3.accessKeyIDSecretRef.name | string | `""` | name of the secret to use |
| k8up.s3.bucket | string | `""` | s3 bucket to backup to |
| k8up.s3.endpoint | string | `""` | s3 endpoint to backup to |
| k8up.s3.secretAccessKeySecretRef.key | string | `""` | key in the secret to use for secret access key |
| k8up.s3.secretAccessKeySecretRef.name | string | `""` | name of the secret to use |
| k8up.schedules | object | `{"backup":"","check":"","prune":""}` | schedules for backups, checks, and prunes |
| pvc_capacity | string | `"10Gi"` | |
| pvc_storageClassName | string | `"local-path"` | |

----------------------------------------------
Autogenerated from chart metadata using [helm-docs v1.13.1](https://github.com/norwoodj/helm-docs/releases/v1.13.1)
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@ kind: PersistentVolumeClaim
metadata:
namespace: home-assistant
name: home-assistant
annotations:
k8up.io/backup: 'true'
spec:
storageClassName: {{ .Values.pvc_storageClassName }}
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15Gi
storage: {{ .Values.pvc_capacity }}
50 changes: 50 additions & 0 deletions home-assistant/storage/templates/scheduled_backups.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
apiVersion: k8up.io/v1
kind: Schedule
metadata:
name: {{ .Values.k8up.backup_name }}
annotations:
"helm.sh/hook": post-install
"helm.sh/resource-policy": keep
spec:
backend:
repoPasswordSecretRef:
name: {{ .Values.k8up.repoPasswordSecretRef.name }}
key: {{ .Values.k8up.repoPasswordSecretRef.key }}
{{- if eq .Values.k8up.backup_type "s3" }}
s3:
endpoint: {{ .Values.k8up.s3.endpoint }}
bucket: {{ .Values.k8up.s3.bucket }}
accessKeyIDSecretRef:
name: {{ .Values.k8up.s3.accessKeyIDSecretRef.name }}
key: {{ .Values.k8up.s3.accessKeyIDSecretRef.key }}
optional: {{ .Values.k8up.s3.accessKeyIDSecretRef.optional }}
secretAccessKeySecretRef:
name: {{ .Values.k8up.s3.secretAccessKeySecretRef.name }}
key: {{ .Values.k8up.s3.secretAccessKeySecretRef.key }}
optional: {{ .Values.k8up.s3.secretAccessKeySecretRef.optional }}
{{- end }}
{{- if eq .Values.k8up.backup_type "local" }}
local:
mountPath: {{ .Values.k8up.local.mountPath }}
{{- end }}
# Backup nightly at scheduled time (Central Europe Time)
backup:
schedule: {{ .Values.k8up.schedules.backup }}
failedJobsHistoryLimit: 5
successfulJobsHistoryLimit: 2
promURL: {{ .Values.k8up.prometheus_url }}
{{- if .Values.k8up.podSecurityContext }}
podSecurityContext:
runAsUser: {{ .Values.k8up.podSecurityContext.runAsUser }}
{{- end }}
# verify the backups are ok
check:
schedule: {{ .Values.k8up.schedules.check }}
promURL: {{ .Values.k8up.prometheus_url }}
# delete old backups
prune:
schedule: {{ .Values.k8up.schedules.prune }}
retention:
keepDaily: 7
keepWeekly: 8
Loading