Optimize rke to handle creation of more than 1000 nodes

[galal-hussein]

Currently rke can be used with any number of nodes, however running on large number of nodes raises some issues including too many open socket files and slow of the process.

The following is a list of optimizations that can be implemented to speed up the process and handle this number of nodes:

• Optimize the Docker dialer to connect concurrently in parallel. use errgroup for tunneling hosts #966
• Cluster state deployer scale issue Potential performance bottleneck starting/removing cluster-state-deployer on large environments #958
• Fix the sync for node labels and taints to handle large number of nodes. Potential performance bottleneck when syncing node labels and taints on large environments #957
• Control the number of threads used when creating k8s planes. Switch concurrent tasks to use worker pool #969
• [ ] Add the ability to only reconcile added/deleted nodes instead of run rke on all nodes. Moved to Refactor reconcile to be done on added/deleted nodes instead of all nodes #974

This should be part of the rke optimization and refactoring issue rancher/rancher#15975

[moelsayed]

Fixes done for this issue are mostly performance tweaks to handle large number of nodes. Several areas are changed internally, so the common use cases should be verified with the latest build.

Notable changes in UX:

• The tunnel up phase is much faster now.
• Sync labels and taints phase is faster and more stable with large number of nodes.
• State save phase is much faster now.

[dnoland1]

@moelsayed Will these improvements also help rke remove? On a ~800 node cluster, it takes over 30 minutes.

[dnoland1]

My observation is rke remove is slow on large clusters and running tasks sequentially. Filed another issue to address - #981

[dnoland1]

Tested yesterday with a master server build - b324973 on an environment with 793 nodes. Total time for rke up to run was 13 minutes, 24 seconds. When using rke v 0.1.10, it took 35 minutes, 5 seconds on the same environment.