Data mover backup node black list - Don't run in specified node

[Lyndon-Li]

The data mover backup exposer generally has the ability to select node for running the data movement. This enables us to fulfill below user requirement:
Sometimes, if a node is running very critical workloads, users don't want data movements to run in the node

We can develop a black list of node mechanism, the nodes in the list will not host data movements.

[Lyndon-Li]

Another user case on this requirement #7185

[Lyndon-Li]

Another use case on this requirement #7243

[balbiv]

@Lyndon-Li thank you for adding this to the milestones. Because the helm deployment allows to set nodeSelectors for the node-agent, I created a dedicated node pool for it so data movement (CSI Snapshot) does have no impact on critical production applications. However, I figured that the pod responsible for mounting the backup PVC (running /velero-helper pause) is allowed to schedule on every node. If the backup pod can take over the nodeSelectors/tolerations from the node-agent daemonset, it will already be a big improvement. This also fixes that the backup pod cannot be started because the image pull policy is set to never.

[Lyndon-Li]

@balbiv
Thanks for the suggestion, as the current plan, we will create a dedicate loadAffinity configMap, instead of coupling node-agent pods scheduling to data mover backup pods scheduling, for below reasons:

• The node-agent not only runs snapshot data movement, but for many other purpose, e.g., it also runs fs-backup/restore. So users may still not want to run data mover backups in all the nodes where node-agent pods resides in
• The other plan is to automatically inherit node-agent pods' node-selection configurations to backupPods'. However, we are clear of some of the configurations, i.e., nodeSelectors, but we are not confident to some others. Specifically, we are not confident that simply inheriting all the configurations from node-agent would bring the same result for backupPod schedule as the node-agent pod schedule since we think the daemonset scheduler behaves differently from plain pod scheduler

Finally, we decided not to bring node-agent's node selection into consideration, if there are some of these configurations, users must apply them to loadAffinity configMap in an appropriate way, see the design PR #7383.

This is the initial plan, we may make changes according to comments of the PR, so for any ideas, you can comment in the PR.