Detect leader health and automatically do failover

[nolouch]

Feature Request

Describe your feature request related problem

PD will bind the PD leader to the etcd leader to reduce the burden of understanding for users. based on this behavior, here meet a problem that the PD leader lease lost but etcd leader doesn't lose. and then the previous PD cannot elect as leader again because of some problems with the leader election. we can simulate the problem like dropping all pockets coming from this connection.

sudo iptables -A INPUT -p tcp --sport 44438 -j DROP

While etcd raft heartbeat to keep etcd leadership goes to other nodes, the PD leader lease keepalive goes directly to the local peer advertise address, so completely different connections. In this case, the PD leader lost but other followers cannot elect a new leader due to the etcd leader still being in the old one, then PD cannot serve, and the cluster is unavailable for a long time until etcd leader be changed.

Describe the feature you'd like

Reduce the unavailable time of the cluster.

Describe alternatives solutions you've considered

PD Leader health detect

Because all followers watch the pd leader's key, so actually all members know who is the leader. we can store the leader-member id and the update time in the memory of all members. once the leader key lease is lost, the leader key will be deleted because the lease expired, then all members should know it by watching the key, then clear the leader id and record the upated time and reset it until the new leader is elected.

the leader record struct like:

type leaderEvent struct {
   leader  *pdpb.Member
   updatedTime time.Time
}

and members can watch the leader key and do relatively handle for it here:

pd/server/server.go

Lines 1428 to 1430 in 46fdd96

	log.Info("start to watch pd leader", zap.Stringer("pd-leader", leader))
	// WatchLeader will keep looping and never return unless the PD leader has changed.
	leader.Watch(s.serverLoopCtx)

Resign etcd leader if no pd leader for a long time

After knowing the pd leader and updated time, pd members can decide to do a failover with let etcd to do a new election base on the lost time. we can do this logic on this:

pd/server/server.go

Lines 1567 to 1582 in 46fdd96

	func (s *Server) etcdLeaderLoop() {
	defer logutil.LogPanic()
	defer s.serverLoopWg.Done()
	ctx, cancel := context.WithCancel(s.serverLoopCtx)
	defer cancel()
	for {
	select {
	case <-time.After(s.cfg.LeaderPriorityCheckInterval.Duration):
	s.member.CheckPriority(ctx)
	case <-ctx.Done():
	log.Info("server is closed, exit etcd leader loop")
	return
	}
	}
	}

once we detect there has etcd leader but no pd leader for a long time(such as 10 * etcdElectionTimeout), we can let the first follower member, with sorted by member id, to do an etcd re-election, the interface can use

pd/pkg/member/member.go

Line 282 in e15b211

func (m *EmbeddedEtcdMember) MoveEtcdLeader(ctx context.Context, old, new uint64) error {

ETA

a week fix on master

[nolouch]

ref #6403, ref #6556