Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Amazon S3 support to the backup/restore features #606

Merged
merged 7 commits into from
Jul 2, 2019
Merged

Add Amazon S3 support to the backup/restore features #606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ab33fbc
Add S3 backup support to tidb-backup helm chart
sokada1221 Jun 20, 2019
879fd76
Add S3 backup support to tidb-cluster helm chart
sokada1221 Jun 20, 2019
bb91cf0
Add S3 backup support to deploy templates
sokada1221 Jun 20, 2019
6269926
Use the latest tidb-cloud-backup image for S3
sokada1221 Jun 24, 2019
ecd672c
Update the doc on backup to Amazon S3
sokada1221 Jun 25, 2019
899c122
Merge branch 'master' into support-s3-backup
gregwebs Jul 1, 2019
00f0848
Merge branch 'master' into support-s3-backup
weekface Jul 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions charts/tidb-backup/templates/backup-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,16 @@ spec:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /gcp/credentials.json
{{- end }}
{{- if .Values.ceph }}
{{- if or .Values.ceph .Values.s3 }}
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: {{ .Values.ceph.secretName }}
name: {{ or .Values.ceph.secretName .Values.s3.secretName }}
key: access_key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ .Values.ceph.secretName }}
name: {{ or .Values.ceph.secretName .Values.s3.secretName }}
key: secret_key
{{- end }}
- name: TIDB_USER
Expand Down
6 changes: 3 additions & 3 deletions charts/tidb-backup/templates/restore-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,16 +44,16 @@ spec:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /gcp/credentials.json
{{- end }}
{{- if .Values.ceph }}
{{- if or .Values.ceph .Values.s3 }}
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: {{ .Values.ceph.secretName }}
name: {{ or .Values.ceph.secretName .Values.s3.secretName }}
key: access_key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ .Values.ceph.secretName }}
name: {{ or .Values.ceph.secretName .Values.s3.secretName }}
key: secret_key
{{- end }}
- name: TIDB_USER
Expand Down
8 changes: 8 additions & 0 deletions charts/tidb-backup/templates/scripts/_start_backup.sh.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,11 @@ uploader \
--endpoint={{ .Values.ceph.endpoint }} \
--backup-dir=${dirname}
{{- end }}

{{- if .Values.s3 }}
uploader \
--cloud=aws \
--region={{ .Values.s3.region }} \
--bucket={{ .Values.s3.bucket }} \
--backup-dir=${dirname}
{{- end }}
9 changes: 9 additions & 0 deletions charts/tidb-backup/templates/scripts/_start_restore.sh.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,15 @@ downloader \
--destDir=/data
{{- end }}

{{- if .Values.s3 }}
downloader \
--cloud=aws \
--region={{ .Values.s3.region }} \
--bucket={{ .Values.s3.bucket }} \
--srcDir=${BACKUP_NAME} \
--destDir=/data
{{- end }}

count=1
while ! mysql -u ${TIDB_USER} -h `eval echo '${'$host'}'` -P 4000 -p${TIDB_PASSWORD} -e 'select version();'
do
Expand Down
13 changes: 11 additions & 2 deletions charts/tidb-backup/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ initialCommitTs: ""
restoreOptions: "-t 16"

# By default, the backup/restore uses PV to store/load backup data
# You can choose to store/load backup data to/from gcp or ceph bucket by enabling the following corresponding section:
# You can choose to store/load backup data to/from gcp, ceph or s3 bucket by enabling the following corresponding section:

# backup to or restore from gcp bucket, the backup path is in the form of <clusterName>-<name>
gcp: {}
Expand All @@ -57,11 +57,20 @@ gcp: {}
# kubectl create secret generic gcp-backup-secret --namespace=<namespace> --from-file=./credentials.json
# secretName: gcp-backup-secret

# backup to or restore from ceph bucket, the backup path is in the form of <clusterName>-<name>
# backup to or restore from ceph bucket, the backup path is in the form of <clusterName>-<name>
ceph: {}
# endpoint: ""
# bucket: ""
# secretName is the name of the secret which stores ceph object store access key and secret key
# You can create the secret by:
# kubectl create secret generic ceph-backup-secret --namespace=<namespace> --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: ceph-backup-secret

# backup to or restore from s3 bucket, the backup path is in the form of <clusterName>-<name>
s3: {}
# region: ""
# bucket: ""
# secretName is the name of the secret which stores s3 object store access key and secret key
# You can create the secret by:
# kubectl create secret generic s3-backup-secret --namespace=<namespace> --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: s3-backup-secret
6 changes: 3 additions & 3 deletions charts/tidb-cluster/templates/scheduled-backup-cronjob.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,16 @@ spec:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /gcp/credentials.json
{{- end }}
{{- if .Values.scheduledBackup.ceph }}
{{- if or .Values.scheduledBackup.ceph .Values.scheduledBackup.s3 }}
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: {{ .Values.scheduledBackup.ceph.secretName }}
name: {{ or .Values.scheduledBackup.ceph.secretName .Values.scheduledBackup.s3.secretName }}
key: access_key
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ .Values.scheduledBackup.ceph.secretName }}
name: {{ or .Values.scheduledBackup.ceph.secretName .Values.scheduledBackup.s3.secretName }}
key: secret_key
{{- end }}
- name: TIDB_USER
Expand Down
8 changes: 8 additions & 0 deletions charts/tidb-cluster/templates/scripts/_start_scheduled_backup.sh.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,11 @@ uploader \
--endpoint={{ .Values.scheduledBackup.ceph.endpoint }} \
--backup-dir=${backupPath}
{{- end }}

{{- if .Values.scheduledBackup.s3 }}
uploader \
--cloud=aws \
--region={{ .Values.scheduledBackup.s3.region }} \
--bucket={{ .Values.scheduledBackup.s3.bucket }} \
--backup-dir=${backupPath}
{{- end }}
9 changes: 9 additions & 0 deletions charts/tidb-cluster/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,6 +481,15 @@ scheduledBackup:
# kubectl create secret generic ceph-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: ceph-backup-secret

# backup to s3
s3: {}
# region: ""
# bucket: ""
# secretName is the name of the secret which stores s3 object store access key and secret key
# You can create the secret by:
# kubectl create secret generic s3-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: s3-backup-secret

metaInstance: "{{ $labels.instance }}"
metaType: "{{ $labels.type }}"
metaValue: "{{ $value }}"
11 changes: 10 additions & 1 deletion deploy/aliyun/templates/tidb-cluster-values.yaml.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,7 @@ scheduledBackup:
binlogImage: "pingcap/tidb-binlog:${cluster_version}"
binlogImagePullPolicy: IfNotPresent
# https://github.com/tennix/tidb-cloud-backup
mydumperImage: pingcap/tidb-cloud-backup:latest
mydumperImage: pingcap/tidb-cloud-backup:20190610
mydumperImagePullPolicy: IfNotPresent
# storageClassName is a StorageClass provides a way for administrators to describe the "classes" of storage they offer.
# different classes might map to quality-of-service levels, or to backup policies,
Expand Down Expand Up @@ -402,6 +402,15 @@ scheduledBackup:
# kubectl create secret generic ceph-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: ceph-backup-secret

# backup to s3
s3: {}
# region: ""
# bucket: ""
# secretName is the name of the secret which stores s3 object store access key and secret key
# You can create the secret by:
# kubectl create secret generic s3-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: s3-backup-secret

metaInstance: "{{ $labels.instance }}"
metaType: "{{ $labels.type }}"
metaValue: "{{ $value }}"
11 changes: 10 additions & 1 deletion deploy/aws/templates/tidb-cluster-values.yaml.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ scheduledBackup:
binlogImage: "pingcap/tidb-binlog:${cluster_version}"
binlogImagePullPolicy: IfNotPresent
# https://github.com/tennix/tidb-cloud-backup
mydumperImage: pingcap/tidb-cloud-backup:latest
mydumperImage: pingcap/tidb-cloud-backup:20190610
mydumperImagePullPolicy: IfNotPresent
# storageClassName is a StorageClass provides a way for administrators to describe the "classes" of storage they offer.
# different classes might map to quality-of-service levels, or to backup policies,
Expand Down Expand Up @@ -403,6 +403,15 @@ scheduledBackup:
# kubectl create secret generic ceph-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: ceph-backup-secret

# backup to s3
s3: {}
# region: ""
# bucket: ""
# secretName is the name of the secret which stores s3 object store access key and secret key
# You can create the secret by:
# kubectl create secret generic s3-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: s3-backup-secret

metaInstance: "{{ $labels.instance }}"
metaType: "{{ $labels.type }}"
metaValue: "{{ $value }}"
11 changes: 10 additions & 1 deletion deploy/gcp/templates/tidb-cluster-values.yaml.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,7 @@ scheduledBackup:
binlogImage: "pingcap/tidb-binlog:${cluster_version}"
binlogImagePullPolicy: IfNotPresent
# https://github.com/tennix/tidb-cloud-backup
mydumperImage: pingcap/tidb-cloud-backup:latest
mydumperImage: pingcap/tidb-cloud-backup:20190610
mydumperImagePullPolicy: IfNotPresent
# storageClassName is a StorageClass provides a way for administrators to describe the "classes" of storage they offer.
# different classes might map to quality-of-service levels, or to backup policies,
Expand Down Expand Up @@ -402,6 +402,15 @@ scheduledBackup:
# kubectl create secret generic ceph-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: ceph-backup-secret

# backup to s3
s3: {}
# region: ""
# bucket: ""
# secretName is the name of the secret which stores s3 object store access key and secret key
# You can create the secret by:
# kubectl create secret generic s3-backup-secret --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: s3-backup-secret

metaInstance: "{{ $labels.instance }}"
metaType: "{{ $labels.type }}"
metaValue: "{{ $value }}"
6 changes: 3 additions & 3 deletions docs/backup-restore.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Currently, TiDB Operator only supports automatic [restore operation](#restore) f

Full backup uses `mydumper` to make a logical backup of TiDB cluster. The backup job will create a PVC([PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims), the same below) to store backup data.

By default, the backup uses PV ([Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes)) to store the backup data. You can also store the backup data to [Google Cloud Storage](https://cloud.google.com/storage/) bucket or [Ceph Object Storage](https://ceph.com/ceph-storage/object-storage/) by changing the configuration. This way the PV temporarily stores backup data before it is placed in object storage. Refer to [TiDB cluster Backup configuration](./references/tidb-backup-configuration.md) for full configuration guide of backup and restore.
By default, the backup uses PV ([Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistent-volumes)) to store the backup data. You can also store the backup data to [Google Cloud Storage](https://cloud.google.com/storage/) bucket, [Ceph Object Storage](https://ceph.com/ceph-storage/object-storage/) or [Amazon S3](https://aws.amazon.com/s3/) by changing the configuration. This way the PV temporarily stores backup data before it is placed in object storage. Refer to [TiDB cluster Backup configuration](./references/tidb-backup-configuration.md) for full configuration guide of backup and restore.

You can either set up a scheduled full backup or take a full backup in an ad-hoc manner.

Expand Down Expand Up @@ -77,15 +77,15 @@ For backups stored in PV, you can view the PVs by using the following command:
$ kubectl get pvc -n ${namespace} -l app.kubernetes.io/component=backup,pingcap.com/backup-cluster-name=${cluster_name}
```

If you store your backup data to [Google Cloud Storage](https://cloud.google.com/storage/) or [Ceph Object Storage](https://ceph.com/ceph-storage/object-storage/), you can view the backups by using the related GUI or CLI tool.
If you store your backup data to [Google Cloud Storage](https://cloud.google.com/storage/), [Ceph Object Storage](https://ceph.com/ceph-storage/object-storage/) or [Amazon S3](https://aws.amazon.com/s3/), you can view the backups by using the related GUI or CLI tool.

## Restore

The helm chart `charts/tidb-backup` helps restore a TiDB cluster using backup data. To perform a restore operation, modify the `charts/tidb-backup/values.yaml` file:

* Set `clusterName` to the target TiDB cluster name
* Set `mode` to `restore`
* Set `name` to the backup name you want to restore([view backups](#view-backups) helps you view all the backups available). If the backup is stored in `Google Cloud Storage` or `Ceph Object Storage`, you must configure the corresponding section too (you might continue to use the same configuration you set in the [adhoc full backup](#ad-hoc-full-backup)).
* Set `name` to the backup name you want to restore([view backups](#view-backups) helps you view all the backups available). If the backup is stored in `Google Cloud Storage`, `Ceph Object Storage` or `Amazon S3`, you must configure the corresponding section too (you might continue to use the same configuration you set in the [adhoc full backup](#ad-hoc-full-backup)).

Create a Kubernetes [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) containing the user and password that has the privilege to restore the database (skip this if you have already created one in the [adhoc full backup](#ad-hoc-full-backup) section):

Expand Down
2 changes: 1 addition & 1 deletion images/tidb-operator-e2e/tidb-cluster-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ fullbackup:
binlogImage: pingcap/tidb-binlog:v3.0.0-rc.1
binlogImagePullPolicy: IfNotPresent
# https://github.com/tennix/tidb-cloud-backup
mydumperImage: pingcap/tidb-cloud-backup:latest
mydumperImage: pingcap/tidb-cloud-backup:20190610
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this change was made for testing and shouldn't be committed? Perhaps we need to improve our e2e testing workflow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay to revert the changes to e2e testing. Just thought it'd make more sense to test with the same version of tidb-cloud-backup as the main code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thing is - is this tag 20190610 appropriate to be used in tidb-operator? The only reason why I replaced this is because the latest tag is outdated, and doesn't support Amazon S3.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the 20190610 is the latest of tidb-cloud-backup image. use it instead of latest.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shinnosuke-okada Do we have to update the tidb-cloud-backup code to support S3 backup/restore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tennix Yes, the latest tag is outdated, and cannot be used to upload to/download from Amazon S3.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tennix My bad! Just realized that I misread your message here last time. No need to change the code. The latest code is good enough to support S3 backup/restore, and it's available via 20190610. If you plan to rebuild the latest tag or some other tag, we can use that as well.

mydumperImagePullPolicy: IfNotPresent
# storageClassName is a StorageClass provides a way for administrators to describe the "classes" of storage they offer.
# different classes might map to quality-of-service levels, or to backup policies,
Expand Down
2 changes: 1 addition & 1 deletion tests/actions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2032,7 +2032,7 @@ func (oa *operatorActions) getBackupDir(info *TidbClusterConfig) ([]string, erro
Containers: []corev1.Container{
{
Name: getBackupDirPodName,
Image: "pingcap/tidb-cloud-backup:latest",
Image: "pingcap/tidb-cloud-backup:20190610",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this change was made for testing and shouldn't be committed?

Command: []string{"sleep", "3000"},
VolumeMounts: []corev1.VolumeMount{
{
Expand Down