Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UCP: Support the noise reduction for auto-scaling #2241

Closed
Yisaer opened this issue Apr 21, 2020 · 9 comments
Closed

UCP: Support the noise reduction for auto-scaling #2241

Yisaer opened this issue Apr 21, 2020 · 9 comments
Assignees
Labels
area/auto-scaling related to auto-scaling challenge-program-2 difficulty/medium enhancement New feature or request status/help-wanted Extra attention is needed
Milestone

Comments

@Yisaer
Copy link
Contributor

Yisaer commented Apr 21, 2020

Description

Currently, the auto-scaling doesn't have the noise reduction design. It means in each reconciliation in auto-scaling controller, the result would be updated into tidbcluster immediately when the result indicates to scale-out or scale-in.

Now we want to support the noise reduction for auto-scaling and design detailed is designed in following:

For TidbClusterAutoScaler, the status would have 3 choices: Normal / ReadyToScaleOut / ReadyToScaleIn.

  1. The initialial status is Normal
  2. If the result in the reconciliation in auto-scaling controller is lager/smaller than the current replicas, the status would become ReadyToScaleOut/ReadyToScaleIn and record the timestamp in annotation.
  3. If the status is remained as ReadyToScaleOut / ReadyToScaleIn for a certain time (which could also be configured in the TidbClusterAutoScaler spec), the auto-scaling controller would finally updated its result in the tidbcluster.
  4. When auto-scaling updated the tidbcluster or the result in the reconciliation in auto-scaling controller is equal to the current replicas, the status should become Normal again.

I think the PR for this issue not only contains the logic of the noise reduction but also should have a e2e case to cover it. You could look the previous logic for auto-scaling in e2e here:

ginkgo.It("auto-scaling TidbCluster", func() {

Score

  • 2500

Mentor(s)

Contact the mentors: #tidb-challenge-program channel in TiDB-Operator Community Slack Workspace
As the logic is not simple and the detail is still needed to be discussed, we are welcome for every challenger to contact the mentor if you have any questions.

Recommended Skills

  • Golang
  • Kubernetes
@Yisaer Yisaer added enhancement New feature or request status/help-wanted Extra attention is needed labels Apr 21, 2020
@TennyZhuang
Copy link

/pick-up-challenge

@sre-bot
Copy link
Contributor

sre-bot commented Apr 21, 2020

@TennyZhuang already had picked pingcap/tidb-dashboard#303, pingcap/tidb#15008, tikv/tikv#7223, finish one before pick up a new one.

@zhoudian64
Copy link

/pick-up-challenge

@sre-bot
Copy link
Contributor

sre-bot commented Apr 21, 2020

@zhoudian64 don't have enough score, pick up failed

Progress 0/200
You may pick up some easy issues first

@vincent178
Copy link
Contributor

/pick-up-challenge

@sre-bot
Copy link
Contributor

sre-bot commented Apr 21, 2020

@vincent178 pick up issue success

@Yisaer
Copy link
Contributor Author

Yisaer commented Apr 26, 2020

@vincent178 Hi, Is there any progress now?

@vincent178
Copy link
Contributor

@Yisaer yeah, I'm still reading the code, will reach out to you maybe tomorrow or the day after to discuss some details, thanks.

@Yisaer
Copy link
Contributor Author

Yisaer commented Apr 26, 2020

@vincent178 okay, please contact me through slack personal message if you have any problem or question. I would help you.

vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* ensure cluster not auto-scale when upgrade
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn
for user configured time
vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* ensure cluster not auto-scale when upgrade
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn for user configured time
vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* ensure cluster not auto-scale when upgrade
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn for user configured time
vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* ensure cluster not auto-scale when upgrade
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn for user configured time
vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* ensure cluster not auto-scale when upgrade
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn for user configured time
vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* ensure cluster not auto-scale when upgrade
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn for user configured time
vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn for user configured time
vincent178 added a commit to vincent178/tidb-operator that referenced this issue Apr 27, 2020
* add ReadyToScaleOut, ReadyToScaleIn status
* record ReadyToScaleOut, ReadyToScaleIn timestamp
* ensure auto scale happens after cluster status remains ReadyToScaleOut, ReadyToScaleIn for user configured time
@DanielZhangQD DanielZhangQD added this to the v1.1.0 milestone May 6, 2020
@DanielZhangQD DanielZhangQD modified the milestones: v1.1.0, v1.1.1 May 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/auto-scaling related to auto-scaling challenge-program-2 difficulty/medium enhancement New feature or request status/help-wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

6 participants