TiUP should serially scale in and scale out PD

[HunDunDM]

Feature Request

Is your feature request related to a problem? Please describe:

• Changing the number of PDs needs to follow the raft process, and only one can be changed at a time. Otherwise, the expanded PD may fail to start.

Describe the feature you'd like:

• TiUP should serially scale in and scale out PD

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

• Go

[9547]

@HunDunDM Could you please provide the instruction and failed logs in detail?

[9547]

I've reproduced this issue. My originally Topology has 3 PDs(all nodes are healthy), and the scale-out topology as below:

pd_servers:
  - host: n6
  - host: n7
  - host: n8
  - host: n9
  - host: n10
  - host: n11
  - host: n12
  - host: n13
  - host: n14
  - host: n15
  - host: n16
  - host: n17
  - host: n18
  - host: n19
  - host: n20

The scale-out failed, message as below:

the error log in n9:2379's pd.log:

[2021/03/27 06:42:07.492 +00:00] [FATAL] [main.go:93] ["join meet error"] [error="there is a member that has not joined successfully"] [stack="github.com/pingcap/log.Fatal\n\t/home/jenkins/agent/workspace/build_pd_multi_branch_v4.0.4/go/pkg/mod/github.com/pingcap/log@v0.0.0-20200511115504-543df19646ad/global.go:59\nmain.main\n\t/home/jenkins/agent/workspace/build_pd_multi_branch_v4.0.4/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:93\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]

The cluster display:

2/15 nodes scale-out succeed.

[seiya-annie]

/found customer