Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

charts: backup: flexible credentials #1248

Merged
merged 6 commits into from
Dec 4, 2019
Merged

charts: backup: flexible credentials #1248

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
84ec6a8
charts: backup: flexible credentials
jlerche Jul 12, 2019
513bc72
Apply suggestions from code review
gregwebs Dec 3, 2019
09b740d
code review suggestions
gregwebs Dec 3, 2019
be2db9f
Merge branch 'master' into charts/backup-fix-credentials
gregwebs Dec 3, 2019
c13a6bc
Merge branch 'master' into charts/backup-fix-credentials
gregwebs Dec 4, 2019
cdb2b98
Merge branch 'master' into charts/backup-fix-credentials
gregwebs Dec 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions charts/tidb-backup/templates/backup-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,11 @@
apiVersion: batch/v1
kind: Job
metadata:
{{- if .Values.jobName }}
name: {{ tpl .Values.jobName . }}
{{- else }}
name: {{ .Values.clusterName }}-{{ tpl .Values.name . }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "chart.name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
Expand All @@ -22,6 +26,9 @@ spec:
{{ toYaml .Values.extraLabels | indent 8 }}
{{- end }}
spec:
{{- if .Values.serviceAccount }}
serviceAccount: {{ .Values.serviceAccount }}
{{- end }}
containers:
- name: backup
image: {{ .Values.image.backup }}
Expand All @@ -38,7 +45,7 @@ spec:
volumeMounts:
- name: data
mountPath: "/data"
{{- if .Values.gcp }}
{{- if .Values.gcp.secretName }}
- name: gcp-credentials
mountPath: "/gcp"
readOnly: true
Expand All @@ -50,11 +57,11 @@ spec:
- name: TZ
value: {{ .Values.timezone | default "UTC" }}
{{- end }}
{{- if .Values.gcp }}
{{- if .Values.gcp.secretName }}
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /gcp/credentials.json
{{- end }}
{{- if or .Values.ceph .Values.s3 }}
{{- if or .Values.ceph.secretName .Values.s3.secretName }}
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
Expand All @@ -81,7 +88,7 @@ spec:
- name: data
persistentVolumeClaim:
claimName: {{ tpl .Values.name . }}
{{- if .Values.gcp }}
{{- if .Values.gcp.secretName }}
- name: gcp-credentials
secret:
secretName: {{ .Values.gcp.secretName }}
Expand Down
38 changes: 31 additions & 7 deletions charts/tidb-backup/templates/scripts/_start_backup.sh.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
set -euo pipefail

{{- if .Values.host }}
host={{ .Values.host }}
{{- else }}
host=$(getent hosts {{ .Values.clusterName }}-tidb | head | awk '{print $1}')
{{- end }}

dirname=/data/${BACKUP_NAME}
echo "making dir ${dirname}"
Expand All @@ -15,6 +19,13 @@ fi
gc_life_time=`/usr/bin/mysql -h${host} -P4000 -u${TIDB_USER} ${password_str} -Nse "select variable_value from mysql.tidb where variable_name='tikv_gc_life_time';"`
echo "Old TiKV GC life time is ${gc_life_time}"

function reset_gc_lifetime() {
echo "Reset TiKV GC life time to ${gc_life_time}"
/usr/bin/mysql -h${host} -P4000 -u${TIDB_USER} ${password_str} -Nse "update mysql.tidb set variable_value='${gc_life_time}' where variable_name='tikv_gc_life_time';"
/usr/bin/mysql -h${host} -P4000 -u${TIDB_USER} ${password_str} -Nse "select variable_name,variable_value from mysql.tidb where variable_name='tikv_gc_life_time';"
}
trap "reset_gc_lifetime" EXIT

echo "Increase TiKV GC life time to {{ .Values.tikvGCLifeTime | default "720h" }}"
/usr/bin/mysql -h${host} -P4000 -u${TIDB_USER} ${password_str} -Nse "update mysql.tidb set variable_value='{{ .Values.tikvGCLifeTime | default "720h" }}' where variable_name='tikv_gc_life_time';"
/usr/bin/mysql -h${host} -P4000 -u${TIDB_USER} ${password_str} -Nse "select variable_name,variable_value from mysql.tidb where variable_name='tikv_gc_life_time';"
Expand All @@ -36,15 +47,28 @@ fi
--tidb-force-priority=LOW_PRIORITY \
{{ .Values.backupOptions }} ${snapshot_args:-}

echo "Reset TiKV GC life time to ${gc_life_time}"
/usr/bin/mysql -h${host} -P4000 -u${TIDB_USER} ${password_str} -Nse "update mysql.tidb set variable_value='${gc_life_time}' where variable_name='tikv_gc_life_time';"
/usr/bin/mysql -h${host} -P4000 -u${TIDB_USER} ${password_str} -Nse "select variable_name,variable_value from mysql.tidb where variable_name='tikv_gc_life_time';"

backup_name="$(basename "${dirname}")"
backup_base_dir="$(dirname "${dirname}")"
{{- if .Values.gcp }}
uploader \
--cloud=gcp \
--bucket={{ .Values.gcp.bucket }} \
--backup-dir=${dirname}
# Once we know there are no more credentials that will be logged we can run with -x
set -x
bucket={{ .Values.gcp.bucket }}
creds=${GOOGLE_APPLICATION_CREDENTIALS:-""}
if ! [[ -z $creds ]] ; then
creds="service_account_file = ${creds}"
fi

cat <<EOF > /tmp/rclone.conf
[gcp]
type = google cloud storage
bucket_policy_only = true
$creds
EOF

cd "${backup_base_dir}"
tar -cf - "${backup_name}" | pigz -p 16 \
| rclone --config /tmp/rclone.conf rcat gcp:${bucket}/${backup_name}/${backup_name}.tgz
gregwebs marked this conversation as resolved.
Show resolved Hide resolved
{{- end }}

{{- if .Values.ceph }}
Expand Down
15 changes: 14 additions & 1 deletion charts/tidb-backup/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ mode: backup # backup | restore | scheduled-restore
# name is the backup dir name and pvc name for ad-hoc backup and restore
name: fullbackup-{{ date "200601021504" .Release.Time }}

# The default jobName is clusterName-name
# jobName:

# The TiDB host to connect to
# By default this will match a normal tidb-operator deploy
# host:

# scheduledBackupName is the name of a scheduled backup directory,
# used to restore the tidb cluster from scheduled backup.
# scheduledBackupName: scheduled-backup-20190822-041004
Expand Down Expand Up @@ -40,6 +47,9 @@ resources:
cpu: 2000m
memory: 4Gi

# Kubernetes service account to use
# serviceAccount:

storage:
className: local-storage
size: 100Gi
Expand All @@ -48,7 +58,8 @@ storage:
# -F is the chunk size, a big table is partitioned into many chunks.
# Other useful options are -B for database, and -T for tables.
# See https://github.com/maxbube/mydumper/blob/master/docs/mydumper_usage.rst#options for more options.
backupOptions: "-t 16 -F 256 --skip-tz-utc --verbose=3"
backupOptions: "--compress-protocol -t 16 -F 256 --skip-tz-utc --verbose=3"

# Set the tidb_snapshot to be used for the backup
# Use `show master status` to get the ts:
# MySQL [(none)]> show master status;
Expand All @@ -72,6 +83,7 @@ tikvGCLifeTime: 720h
# backup to or restore from gcp bucket, the backup path is in the form of <clusterName>-<name>
gcp: {}
# bucket: ""
# secretName is not necessary on GKE if you use the workload identity feature
gregwebs marked this conversation as resolved.
Show resolved Hide resolved
# secretName is the name of the secret which stores the gcp service account credentials json file
# The service account must have read/write permission to the above bucket.
# Read the following document to create the service account and download the credentials file as credentials.json:
Expand All @@ -94,6 +106,7 @@ s3: {}
# region: ""
# bucket: ""
# secretName is the name of the secret which stores s3 object store access key and secret key
# This is not necessary on AWS. Instead you should be able to get the credentials from the EKS service IAM role.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then how to use it in the job container?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AWS SDK will find the credentials, it just needs to run with a service account. You have to be using EKS or properly running something like iam-authenticator in the k8s cluster.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just need to attach the service account, no need to install the SDK or call any command to run with it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does iam-authenticator need to be packaged in the Docker image?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, these changes are tested as working. Just set the service account.
On GCP the association with the k8s service account to the cloud service account is done with

kubectl -n operations annotate serviceaccount ${svcacct} iam.gke.io/gcp-service-account=${svcacct}@${PROJECT}.iam.gserviceaccount.com

On EKS I believe one just needs to attach a policy. If not using EKS you run an authenticator program for the entire cluster. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

# You can create the secret by:
# kubectl create secret generic s3-backup-secret --namespace=<namespace> --from-literal=access_key=<access-key> --from-literal=secret_key=<secret-key>
# secretName: s3-backup-secret