Skip to content

Commit

Permalink
en: backup-to-s3, deploy-on-aws-eks: various small changes (#1210)
Browse files Browse the repository at this point in the history
  • Loading branch information
dveeden authored May 24, 2021
1 parent 0f67a9c commit 4f64540
Show file tree
Hide file tree
Showing 2 changed files with 55 additions and 21 deletions.
70 changes: 51 additions & 19 deletions en/backup-to-s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,16 @@ The backup method described in this document is implemented based on CustomResou

Ad-hoc full backup describes the backup by creating a `Backup` custom resource (CR) object. TiDB Operator performs the specific backup operation based on this `Backup` object. If an error occurs during the backup process, TiDB Operator does not retry and you need to handle this error manually.

For the current S3-compatible storage types, Ceph and Amazon S3 work normally as tested. Therefore, this document shows examples in which the data of the `demo1` TiDB cluster in the `test1` Kubernetes namespace is backed up to Ceph and Amazon S3 respectively.
For the current S3-compatible storage types, Ceph and Amazon S3 work normally as tested. Therefore, this document shows examples in which the data of the `demo1` TiDB cluster in the `tidb-cluster` Kubernetes namespace is backed up to Ceph and Amazon S3 respectively.

### Prerequisites for ad-hoc full backup

1. Download [backup-rbac.yaml](https://github.com/pingcap/tidb-operator/blob/master/manifests/backup/backup-rbac.yaml), and execute the following command to create the role-based access control (RBAC) resources in the `test1` namespace:
1. Execute the following command to create the role-based access control (RBAC) resources in the `tidb-cluster` namespace based on [backup-rbac.yaml](https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/backup/backup-rbac.yaml):

{{< copyable "shell-regular" >}}

```shell
kubectl apply -f backup-rbac.yaml -n test1
kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/backup/backup-rbac.yaml -n tidb-cluster
```

2. Grant permissions to the remote storage.
Expand All @@ -37,16 +37,27 @@ For the current S3-compatible storage types, Ceph and Amazon S3 work normally as
{{< copyable "shell-regular" >}}

```shell
kubectl create secret generic backup-demo1-tidb-secret --from-literal=password=${password} --namespace=test1
kubectl create secret generic backup-demo1-tidb-secret --from-literal=password=${password} --namespace=tidb-cluster
```

### Required database account privileges

* The `SELECT` and `UPDATE` privileges of the `mysql.tidb` table: Before and after the backup, the `Backup` CR needs a database account with these privileges to adjust the GC time.
* SELECT
* RELOAD
* LOCK TABLES
* REPLICATION CLIENT
* The global privileges: `SELECT`, `RELOAD`, `LOCK TABLES` and `REPLICATION CLIENT`

An example for creating a backup user:

```sql
CREATE USER 'backup'@'%' IDENTIFIED BY '...';
GRANT
SELECT, RELOAD, LOCK TABLES, REPLICATION CLIENT
ON *.*
TO 'backup'@'%';
GRANT
UPDATE, SELECT
ON mysql.tidb
TO 'backup'@'%';
```

### Ad-hoc backup process

Expand All @@ -63,6 +74,17 @@ For the current S3-compatible storage types, Ceph and Amazon S3 work normally as
> - --ignore-checksum
> ```

> **Note**
>
> This section lists multiple storage access methods. Only follow the method that matches your situation.
>
> The methods are as follows:
>
> - Amazon S3 by importing AccessKey and SecretKey
> - Ceph by importing AccessKey and SecretKey
> - Amazon S3 by binding IAM with Pod
> - Amazon S3 by binding IAM with ServiceAccount

+ Create the `Backup` CR, and back up cluster data to Amazon S3 by importing AccessKey and SecretKey to grant permissions:

{{< copyable "shell-regular" >}}
Expand All @@ -81,7 +103,7 @@ For the current S3-compatible storage types, Ceph and Amazon S3 work normally as
kind: Backup
metadata:
name: demo1-backup-s3
namespace: test1
namespace: tidb-cluster
spec:
from:
host: ${tidb_host}
Expand Down Expand Up @@ -125,7 +147,7 @@ For the current S3-compatible storage types, Ceph and Amazon S3 work normally as
kind: Backup
metadata:
name: demo1-backup-s3
namespace: test1
namespace: tidb-cluster
spec:
from:
host: ${tidb_host}
Expand Down Expand Up @@ -166,7 +188,7 @@ For the current S3-compatible storage types, Ceph and Amazon S3 work normally as
kind: Backup
metadata:
name: demo1-backup-s3
namespace: test1
namespace: tidb-cluster
annotations:
iam.amazonaws.com/role: arn:aws:iam::123456789012:role/user
spec:
Expand Down Expand Up @@ -212,7 +234,7 @@ For the current S3-compatible storage types, Ceph and Amazon S3 work normally as
kind: Backup
metadata:
name: demo1-backup-s3
namespace: test1
namespace: tidb-cluster
spec:
backupType: full
serviceAccount: tidb-backup-manager
Expand Down Expand Up @@ -256,9 +278,19 @@ After creating the `Backup` CR, use the following command to check the backup st
{{< copyable "shell-regular" >}}
```shell
kubectl get bk -n test1 -owide
kubectl get bk -n tidb-cluster -owide
```
To get detailed information on a backup job, use the following command. For `$backup_job_name` in the command, use the name from the output of the previous command.
{{< copyable "shell-regular" >}}
```shell
kubectl describe bk -n tidb-cluster $backup_job_name
```
To run ad-hoc backup again, you need to [delete the backup CR](backup-restore-overview.md#delete-the-backup-cr) and create it again.
## Scheduled full backup to S3-compatible storage
You can set a backup policy to perform scheduled backups of the TiDB cluster, and set a backup retention policy to avoid excessive backup items. A scheduled full backup is described by a custom `BackupSchedule` CR object. A full backup is triggered at each backup time point. Its underlying implementation is the ad-hoc full backup.
Expand Down Expand Up @@ -302,7 +334,7 @@ The prerequisites for the scheduled backup is the same as the [prerequisites for
kind: BackupSchedule
metadata:
name: demo1-backup-schedule-s3
namespace: test1
namespace: tidb-cluster
spec:
#maxBackups: 5
#pause: true
Expand Down Expand Up @@ -351,7 +383,7 @@ The prerequisites for the scheduled backup is the same as the [prerequisites for
kind: BackupSchedule
metadata:
name: demo1-backup-schedule-ceph
namespace: test1
namespace: tidb-cluster
spec:
#maxBackups: 5
#pause: true
Expand Down Expand Up @@ -397,7 +429,7 @@ The prerequisites for the scheduled backup is the same as the [prerequisites for
kind: BackupSchedule
metadata:
name: demo1-backup-schedule-s3
namespace: test1
namespace: tidb-cluster
annotations:
iam.amazonaws.com/role: arn:aws:iam::123456789012:role/user
spec:
Expand Down Expand Up @@ -447,7 +479,7 @@ The prerequisites for the scheduled backup is the same as the [prerequisites for
kind: BackupSchedule
metadata:
name: demo1-backup-schedule-s3
namespace: test1
namespace: tidb-cluster
spec:
#maxBackups: 5
#pause: true
Expand Down Expand Up @@ -483,15 +515,15 @@ After creating the scheduled full backup, you can use the following command to c
{{< copyable "shell-regular" >}}
```shell
kubectl get bks -n test1 -owide
kubectl get bks -n tidb-cluster -owide
```
You can use the following command to check all the backup items:
{{< copyable "shell-regular" >}}
```shell
kubectl get bk -l tidb.pingcap.com/backup-schedule=demo1-backup-schedule-s3 -n test1
kubectl get bk -l tidb.pingcap.com/backup-schedule=demo1-backup-schedule-s3 -n tidb-cluster
```
From the example above, you can see that the `backupSchedule` configuration consists of two parts. One is the unique configuration of `backupSchedule`, and the other is `backupTemplate`.
Expand Down
6 changes: 4 additions & 2 deletions en/deploy-on-aws-eks.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ Before deploying a TiDB cluster on AWS EKS, make sure the following requirements
* Install and configure `eksctl` used for creating Kubernetes clusters.
* Install `kubectl`.

To verify whether AWS CLI is configured correctly, run the `aws configure list` command. If the output shows the values for `access_key` and `secret_key`, AWS CLI is configured correctly. Otherwise, you need to re-configure AWS CLI.

> **Note:**
>
> The operations described in this document requires at least the [minimum privileges needed by `eksctl`](https://eksctl.io/usage/minimum-iam-policies/) and the [service privileges needed to create a Linux bastion host](https://docs.aws.amazon.com/quickstart/latest/linux-bastion/architecture.html#aws-services).
Expand All @@ -31,7 +33,7 @@ Before deploying a TiDB cluster on AWS EKS, make sure the following requirements

According to AWS [Official Blog](https://aws.amazon.com/blogs/containers/amazon-eks-cluster-multi-zone-auto-scaling-groups/) recommendation and EKS [Best Practice Document](https://aws.github.io/aws-eks-best-practices/reliability/docs/dataplane/#ensure-capacity-in-each-az-when-using-ebs-volumes), since most of the TiDB cluster components use EBS volumes as storage, it is recommended to create a node pool in each availability zone (at least 3 in total) for each component when creating an EKS.

Save the following configuration as the `cluster.yaml` file. Replace `${clusterName}` with your desired cluster name.
Save the following configuration as the `cluster.yaml` file. Replace `${clusterName}` with your desired cluster name. The cluster and node group names should match the regular expression `[a-zA-Z][-a-zA-Z0-9]*`, so avoid names that contain `_`.

{{< copyable "" >}}

Expand Down Expand Up @@ -135,7 +137,7 @@ Execute the following command to create the cluster:
eksctl create cluster -f cluster.yaml
```

After executing the command above, you need to wait until the EKS cluster is successfully created and the node group is created and added in the EKS cluster. This process might take 5 to 10 minutes. For more cluster configuration, refer to [`eksctl` documentation](https://eksctl.io/usage/creating-and-managing-clusters/#using-config-files).
After executing the command above, you need to wait until the EKS cluster is successfully created and the node group is created and added in the EKS cluster. This process might take 5 to 20 minutes. For more cluster configuration, refer to [`eksctl` documentation](https://eksctl.io/usage/creating-and-managing-clusters/#using-config-files).

> **Warning:**
>
Expand Down

0 comments on commit 4f64540

Please sign in to comment.