Add Amazon S3 support to the backup/restore features #606
Conversation
|
Hi contributor, thanks for your PR. This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically. |
|
|
|
/ok-to-test |
| @@ -224,7 +224,7 @@ fullbackup: | |||
| binlogImage: pingcap/tidb-binlog:v3.0.0-rc.1 | |||
| binlogImagePullPolicy: IfNotPresent | |||
| # https://github.com/tennix/tidb-cloud-backup | |||
| mydumperImage: pingcap/tidb-cloud-backup:latest | |||
| mydumperImage: pingcap/tidb-cloud-backup:20190610 | |||
There was a problem hiding this comment.
It looks like this change was made for testing and shouldn't be committed? Perhaps we need to improve our e2e testing workflow.
There was a problem hiding this comment.
Okay to revert the changes to e2e testing. Just thought it'd make more sense to test with the same version of tidb-cloud-backup as the main code.
There was a problem hiding this comment.
Another thing is - is this tag 20190610 appropriate to be used in tidb-operator? The only reason why I replaced this is because the latest tag is outdated, and doesn't support Amazon S3.
There was a problem hiding this comment.
the 20190610 is the latest of tidb-cloud-backup image. use it instead of latest.
There was a problem hiding this comment.
@shinnosuke-okada Do we have to update the tidb-cloud-backup code to support S3 backup/restore?
There was a problem hiding this comment.
@tennix Yes, the latest tag is outdated, and cannot be used to upload to/download from Amazon S3.
There was a problem hiding this comment.
@tennix My bad! Just realized that I misread your message here last time. No need to change the code. The latest code is good enough to support S3 backup/restore, and it's available via 20190610. If you plan to rebuild the latest tag or some other tag, we can use that as well.
| @@ -2032,7 +2032,7 @@ func (oa *operatorActions) getBackupDir(info *TidbClusterConfig) ([]string, erro | |||
| Containers: []corev1.Container{ | |||
| { | |||
| Name: getBackupDirPodName, | |||
| Image: "pingcap/tidb-cloud-backup:latest", | |||
| Image: "pingcap/tidb-cloud-backup:20190610", | |||
There was a problem hiding this comment.
It looks like this change was made for testing and shouldn't be committed?
|
@tennix PTAL |
|
/run-e2e-tests |
|
/run-e2e-tests |
|
The e2e was broken now, I am working on it. |
|
/run-e2e-tests |
1 similar comment
|
/run-e2e-tests |
What problem does this PR solve?
#604
What is changed and how it works?
Respective commit comment should explain what's being changed. Added Amazon S3 support to the backup/restore feature.
The problem with the current design is that Ceph and S3 has to share the same set of environment variables (
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY) and thus both cannot be enabled at the same time unless we make changes touploaderanddownloader. Given that they're planned to be deprecated withrclone, I wouldn't address this limitation in this PR.Check List
Tests
scheduledBackup, and upload to Amazon S3 everyday - uploaded successfully to the specified S3 bucketscheduledBackup, and upload to Amazon S3 every 15 minutes - uploaded successfully to the specified S3 bucketCode changes
Side effects
Related changes
Does this PR introduce a user-facing change?: