Support TiKV auto scaling-out by Storage #2884

Yisaer · 2020-07-08T07:11:06Z

What problem does this PR solve?

Support auto scaling-out TiKV by Storage. The Storage Auto-scaling Strategy is similar to the AWS RDS Storage Auto-scaling

The tikv would be auto-scaling out in these rules:

If the available storage size have already been less than the 10%(could be customized) of the capacity storage size, the cluster status will be marked as Storage Pressure.
If the status Storage Pressure has been existed for 5min(could be customized), the tikv would be scaled-out.

Note that the Max-Replicas and ScaleOut Cool-down is still worked in this case.

Note that in v1.1.x, auto-scaling by storage and cpu couldn't be both work for tikv. We will support this feature in v1.2.x

If the rule based by cpu and based by storage are both set, only rule based by storage would be considered.

Here is one example for the tidbclusterautoscaler to create auto-scaling by storage :

apiVersion: pingcap.com/v1alpha1
kind: TidbClusterAutoScaler
metadata:
  name: basic
spec:
  cluster:
    name: basic
    namespace: tidb-admin
  monitor:
    name: monitor
    namespace: tidb-admin
  tikv:
    maxReplicas: 3
    scaleOutIntervalSeconds: 500
    metrics:
      - type: "Resource"
        leastStoragePressurePeriodSeconds: 300
        leastRemainAvailableStoragePercent: 10
        resource:
          name: "storage"

The Status would be like below (normal):

status:
  tikv:
    currentReplicas: 1
    lastAutoScalingTimestamp: "2020-07-08T11:30:38Z"
    metrics:
    - availableStorage: 100.1 GB
      baselineAvailableStorage: 15.6 GB
      capacityStorage: 156.4 GB
      name: storage
      storagePressure: false

During Storage Pressure:

status:
  tikv:
    currentReplicas: 1
    lastAutoScalingTimestamp: "2020-07-08T11:33:44Z"
    metrics:
    - availableStorage: 100.1 GB
      baselineAvailableStorage: 140.8 GB
      capacityStorage: 156.4 GB
      name: storage
      storagePressure: true
      storagePressureStartTime: "2020-07-08T11:33:44Z"

What is changed and how does it work?

Check List

Tests

Unit test
E2E test

Related changes

Need to cherry-pick to the release branch

Does this PR introduce a user-facing change?:

Support Auto-scaling by Storage for TiKV in TidbClusterAutoScaler

fix fix fix Update tikv_autoscaler.go fix fix fix lint add doc fix error fix error fix unit test fix error remove useless code fux unit test fix error Update docs.md fix api fix status

…-operator into support_storage_auto

Yisaer · 2020-07-09T04:25:51Z

/pull-e2e-kind-serial

Yisaer · 2020-07-09T05:04:57Z

/pull-e2e-kind-serial

Yisaer · 2020-07-09T05:07:12Z

/test pull-e2e-kind-serial

Yisaer · 2020-07-09T06:01:30Z

/test pull-e2e-kind-serial

…-operator into support_storage_auto

cofyc · 2020-07-13T03:18:10Z

I would suggest using the phrase storage-based auto-scaling instead of auto-scaling by storage:

Support storage-based auto-scaling for TiKV in TidbClusterAutoScaler

ti-srebot · 2020-07-13T03:19:13Z

@cofyc,Thanks for your review.

ti-srebot · 2020-07-13T06:18:04Z

@weekface,Thanks for your review.

Yisaer · 2020-07-13T06:18:21Z

/merge

ti-srebot · 2020-07-13T06:20:57Z

/run-all-tests

Yisaer · 2020-07-13T09:13:26Z

/run-all-tests

Yisaer · 2020-07-13T11:04:46Z

/test pull-e2e-kind-asts

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2020-07-13T11:19:35Z

cherry pick to release-1.1 in PR #2917

Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Song Gao <disxiaofei@163.com>

Yisaer marked this pull request as draft July 8, 2020 07:11

Yisaer force-pushed the support_storage_auto branch from 8d34c50 to e39a4e4 Compare July 8, 2020 07:12

Yisaer added area/auto-scaling related to auto-scaling needs-cherry-pick-1.1 labels Jul 8, 2020

Yisaer force-pushed the support_storage_auto branch from d67aeeb to d650d70 Compare July 8, 2020 08:53

Yisaer changed the title ~~Support TiKV Auto-scaling by Storage~~ Support TiKV auto scaling-out by Storage Jul 8, 2020

add storage api

75ead5d

fix fix fix Update tikv_autoscaler.go fix fix fix lint add doc fix error fix error fix unit test fix error remove useless code fux unit test fix error Update docs.md fix api fix status

Yisaer force-pushed the support_storage_auto branch from efe3d76 to 75ead5d Compare July 8, 2020 11:39

Yisaer added 2 commits July 8, 2020 20:38

add e2e case

a843fe7

Merge branch 'master' into support_storage_auto

797cf82

Yisaer requested review from DanielZhangQD, cofyc, weekface, zjj2wry and lichunzhu July 8, 2020 12:39

Yisaer marked this pull request as ready for review July 8, 2020 12:39

Yisaer added the status/PTAL PR needs to be reviewed label Jul 8, 2020

Yisaer added 7 commits July 8, 2020 21:06

fix lint

675203a

fmt file

dfbf77b

fix e2e

3633979

Merge branch 'master' into support_storage_auto

e3710ab

fix update

b6e7bc1

Merge branch 'support_storage_auto' of https://github.com/Yisaer/tidb…

746d054

…-operator into support_storage_auto

fix e2e

2004065

This comment has been minimized.

Sign in to view

Merge branch 'support_storage_auto' of https://github.com/Yisaer/tidb…

7192741

…-operator into support_storage_auto

cofyc previously approved these changes Jul 13, 2020

View reviewed changes

ti-srebot added the status/LGT1 label Jul 13, 2020

update notes

6f057ba

Yisaer dismissed cofyc’s stale review via 6f057ba July 13, 2020 03:45

Yisaer requested a review from cofyc July 13, 2020 04:58

cofyc approved these changes Jul 13, 2020

View reviewed changes

weekface approved these changes Jul 13, 2020

View reviewed changes

ti-srebot removed the status/LGT1 label Jul 13, 2020

ti-srebot approved these changes Jul 13, 2020

View reviewed changes

ti-srebot added the status/LGT2 label Jul 13, 2020

ti-srebot added the CanMerge label Jul 13, 2020

Merge branch 'master' into support_storage_auto

691a03a

This comment has been minimized.

Sign in to view

Yisaer merged commit e0a5137 into pingcap:master Jul 13, 2020

ti-srebot pushed a commit to ti-srebot/tidb-operator that referenced this pull request Jul 13, 2020

cherry pick pingcap#2884 to release-1.1

fd0fbf8

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jul 13, 2020

Support TiKV auto scaling-out by Storage (#2884) #2917

Merged

Yisaer added a commit that referenced this pull request Jul 14, 2020

cherry pick #2884 to release-1.1 (#2917)

d48cc1e

Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Song Gao <disxiaofei@163.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support TiKV auto scaling-out by Storage #2884

Support TiKV auto scaling-out by Storage #2884

Yisaer commented Jul 8, 2020 •

edited

Loading

Yisaer commented Jul 9, 2020

Yisaer commented Jul 9, 2020

This comment has been minimized.

This comment has been minimized.

Yisaer commented Jul 9, 2020

Yisaer commented Jul 9, 2020

cofyc commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

Yisaer commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

Yisaer commented Jul 13, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Yisaer commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

Support TiKV auto scaling-out by Storage #2884

Support TiKV auto scaling-out by Storage #2884

Conversation

Yisaer commented Jul 8, 2020 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Does this PR introduce a user-facing change?:

Yisaer commented Jul 9, 2020

Yisaer commented Jul 9, 2020

This comment has been minimized.

This comment has been minimized.

Yisaer commented Jul 9, 2020

Yisaer commented Jul 9, 2020

cofyc commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

Yisaer commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

Yisaer commented Jul 13, 2020

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Yisaer commented Jul 13, 2020

ti-srebot commented Jul 13, 2020

Yisaer commented Jul 8, 2020 •

edited

Loading