Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support rclone options in backup and restore cr #2318

Merged
merged 5 commits into from
Apr 28, 2020

Conversation

DanielZhangQD
Copy link
Contributor

@DanielZhangQD DanielZhangQD commented Apr 28, 2020

What problem does this PR solve?

Fix #2270

What is changed and how does it work?

Support specifying rclone options in the backup and restore CR which can be used during backup and restore with mydumper and lightning.
For the issue in #2270, we need to set --ignore-checksum option for rclone, for detail, please check the doc.
CR example:

apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
  name: test31-lt-07
  namespace: test31
spec:
  from:
    host: test31-tidb
    port: 4000
    user: root
    secretName: backup-secret
  s3:
    provider: aws
    region: us-west-2
    secretName: s3-secret-aws
    bucket: danz-backup10/lt07
    options:
    - --ignore-checksum
  storageClassName: local-storage
  storageSize: 1Gi

Check List

Tests

  • Manual test (add detailed scripts or steps below)
  • backup, restore, scheduled backup, clean with s3 encryption disabled
  • backup, restore, scheduled backup, clean with s3 aes-256 encryption
  • backup, restore, scheduled backup, clean with s3 aws-kms encryption

Code changes

  • Has Go code change

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Does this PR introduce a user-facing change?:

Support `rclone` options in Backup and Restore CR

@@ -843,6 +843,8 @@ type S3StorageProvider struct {
Prefix string `json:"prefix,omitempty"`
// SSE Sever-Side Encryption.
SSE string `json:"sse,omitempty"`
// Options Rclone options for backup and restore with mydumper and lightning.
Options string `json:"options,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use string slice?

Copy link
Contributor

@Yisaer Yisaer Apr 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether expose args by a string is ok to users. Can we import the rclone configFlag as rcloneConfig in the Backup and Restore Spec like tidb/tikv/pd config

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be too much config fields. Since mydumper+lightning is not our focus already, I think it's OK to support like this.
I will change it to string slice.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't have to set all the configuration of rclone, just use the configuration we need.

onlymellb
onlymellb previously approved these changes Apr 28, 2020
Copy link
Contributor

@onlymellb onlymellb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@DanielZhangQD
Copy link
Contributor Author

@cofyc comments addressed, PTAL.

@DanielZhangQD
Copy link
Contributor Author

/merge

@sre-bot
Copy link
Contributor

sre-bot commented Apr 28, 2020

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented Apr 28, 2020

@DanielZhangQD merge failed.

@DanielZhangQD
Copy link
Contributor Author

/merge

@sre-bot
Copy link
Contributor

sre-bot commented Apr 28, 2020

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented Apr 28, 2020

cherry pick to release-1.1 in PR #2326

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mydumper CRD based backup doesn't work with encrypted S3 buckets
5 participants