Simple Backups for Persisting Volumes

[creib]

Hi,
I am struggling with the backup of my databases which are stored on pvc in the hetzner-k3s cluster. Do you have any proposial for a simple and daily backup of various Persisting Volumes?
I checked Valero, but I am not sure, if it can be used in this case to backup PV to a Hetzner Volume.
Thanks for your ideas and hints :)

[vitobotta]

Hi, for database the recommended way is to use backup tools specific for the databases. Velero is an awesome backup tool and can back up persistent volumes using Hetzner block storage (using restic behind the scenes), but there is the risk of data corruption with the database when you try to restore, because some data might be changed while Velero is backing up. There is actually a potential fix for this with Velero using fsfreezs with hooks - see https://velero.io/docs/v0.11.0/hooks/. Basically it freezes the filesystem before starting the backup and unfreezes it soon after. This guarantees that the backups are crash consistent, but it has the downside that if the backup takes long, the filesystem is frozen for a while so writes hang. It works well if backups are quick, and since Velero does incremental backups, it may work. You need to try it and see it for yourself.

The nice thing with Velero is that you can back up an entire namespace and restore it in a new cluster as it is. But there is the problem I described above.

For databases it's anyway better to use the database's own backup tool (like pg_dump for Postgres or mysqldump for MySQL and so on) because it guarantees crash consistent backups, and also allows for restoring the database anywhere, not just in a Kubernetes cluster.

Which databases are you referring to?

[vitobotta]

For Postgres I can share my Docker image that I use to back up my databases, but I am not familiar with MongoDB unfortunately. As for Grafana and Prometheus I usually don't back them up because in new clusters I just reinstall them. I haven't had a need to back them up so far.

[vitobotta]

BTW I forgot to mention that another very good backup tool is Kasten K10 (https://www.kasten.io/product/), in case you want to take a look. It's a commercial product but it has a free tier as well.

[creib]

For Postgres I can share my Docker image that I use to back up my databases, but I am not familiar with MongoDB unfortunately. As for Grafana and Prometheus I usually don't back them up because in new clusters I just reinstall them. I haven't had a need to back them up so far.

This sounds very good. In MongoDb it should be more or less the same using mongodump instead of pg_dump. It would be great if you can share this docker image.

[vitobotta]

Hi, sorry for the delay. To use my image first create a secret with both the Postgres password and the S3 credentials:

apiVersion: v1
kind: Secret
metadata:
  name: s3-postgres-backup
  namespace: postgres
type: Opaque
stringData:
  POSTGRES_PASSWORD: ...
  S3_ACCESS_KEY_ID: ...
  S3_BUCKET: ...
  S3_ENDPOINT: https://..
  S3_REGION: ...
  S3_SECRET_ACCESS_KEY: ...

Then create a cronjob for each databaase

apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup-<DATABASE_NAME>
  namespace: postgres
spec:
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - env:
            - name: SLACK_WEBHOOK
              value: <a slack webhook for notifications, you may want to use a secret for it>
            - name: S3_PREFIX
              value: postgres/<DATABASE_NAME>
            - name: POSTGRES_DATABASE
              value: <DATABASE_NAME>
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_HOST
              value: <k8s svc service for postgres>
            - name: POSTGRES_PORT
              value: "5432"
            envFrom:
            - secretRef:
                name: s3-postgres-backup
            image: vitobotta/postgres-s3-backup:v345160c
            imagePullPolicy: IfNotPresent
            name: postgres-backup
          dnsPolicy: ClusterFirst
          restartPolicy: Never
          schedulerName: default-scheduler
          terminationGracePeriodSeconds: 30
  schedule: 0 5 * * *
  successfulJobsHistoryLimit: 3
  suspend: false

Hope this helps :)

[creib]

Thanks a lot!

[vitobotta]

You're welcome :)