[feature request] NodePool Governance Capability: yurthub are delegated to report heartbeats for nodes that disconnected with cloud

[Peeknut]

What would you like to be added:
Yurthub needs to add a leader election mechanism and a lease proxy mechanism.

Leader election mechanism can be described as:

• Yurthub senses the existence of pool-coordinator and participates in leader election if it can connect to the cloud.
• Leader Yurthub should list/watch the lease resources in the pool-coordinator.

Lease proxy mechanism can be described as:

• every node(yurthub component) will send node lease to pool-coordinator at interval(10s), and add delegate annotation in node lease when node disconnected from the cloud.
• Leader Yurthub list/watch node lease from pool-coordinator, and it will proxy the node lease to cloud kube-apiserver when node lease has delegation annotation. If the lease of pool-coordinator node is also need to be forwarded, the leader YurtHub will give priority to forwarding Node leases of its node.

by the way, yurthub should upload node scope data(like pods, node) to pool-coordinator when pool-coordinator is detected in the nodePool.

Notes: For details, please refer to the proposal: #772

Why is this needed:
As mentioned in the proposal(#772), in the node pool governance, the yurthub connected to the cloud will elect a leader, and the leader will forward the heartbeat of the node disconnected from the cloud to the cloud, so that the cloud can distinguish whether the disconnection of the node is caused by a node failure or a network disconnection. Therefore, yurthub needs to add a leader election mechanism and a heartbeat information reporting mechanism.
others
/kind feature

[rambohe-ch]

/assign @rambohe-ch

[rambohe-ch]

Progress update:

1. pool-coordinator existence checker: in progress
2. leader election for yurthub: almost finished
3. a new client certificate for pool-coordinator in yurthub: in progress
4. lease forwarding: todo

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rambohe-ch]

State machine for leader election of Yurthub:

[rambohe-ch]

workflow for leader election of Yurthub:

[rambohe-ch]

workflow for Yurthub coordinator:

[rambohe-ch]

/close

[openyurt-bot]

@rambohe-ch: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.