feat: scheduler (12/): add more scheduler logic

[michaelawyu]

Description of your changes

This PR is part of the PRs that implement the Fleet workload scheduling.

It features more scheduling logic for PickAll type CRPs.

I have:

• Run make reviewable to ensure this PR is ready for review.

How has this code been tested

• Unit tests

Special notes for your reviewer

To control the size of the PR, certain unit tests are not checked in; they will be sent in a separate PR.

[michaelawyu]

A special note:

Currently the API has a limit of 100 for cluster decisions, but it also dictates that we need to keep all decisions from selected clusters. Considering that we do not have a schema-side range set for numOfClusters, do we drop decisions when there are over 100 selected clusters or do we set a limit of 100 to numOfClusters as well?

[ryanzhang-oss]

A special note:

Currently the API has a limit of 100 for cluster decisions, but it also dictates that we need to keep all decisions from selected clusters. Considering that we do not have a schema-side range set for numOfClusters, do we drop decisions when there are over 100 selected clusters or do we set a limit of 100 to numOfClusters as well?

I wonder where is the 100 limit come from? I think k8s scheduler has a parameter about how many nodes to consider. Do we need to add that?

[zhiying-lin]

A special note:

Currently the API has a limit of 100 for cluster decisions, but it also dictates that we need to keep all decisions from selected clusters. Considering that we do not have a schema-side range set for numOfClusters, do we drop decisions when there are over 100 selected clusters or do we set a limit of 100 to numOfClusters as well?

we cannot limit the numberOfClusters for the selectAll type and still need to handle the 100 limit size.

A special note:
Currently the API has a limit of 100 for cluster decisions, but it also dictates that we need to keep all decisions from selected clusters. Considering that we do not have a schema-side range set for numOfClusters, do we drop decisions when there are over 100 selected clusters or do we set a limit of 100 to numOfClusters as well?

I wonder where is the 100 limit come from? I think k8s scheduler has a parameter about how many nodes to consider. Do we need to add that?

100 limit comes from https://github.com/Azure/fleet/blob/main/apis/placement/v1beta1/policysnapshot_types.go#L77

[michaelawyu]

A special note:
Currently the API has a limit of 100 for cluster decisions, but it also dictates that we need to keep all decisions from selected clusters. Considering that we do not have a schema-side range set for numOfClusters, do we drop decisions when there are over 100 selected clusters or do we set a limit of 100 to numOfClusters as well?

I wonder where is the 100 limit come from? I think k8s scheduler has a parameter about how many nodes to consider. Do we need to add that?

Hi Ryan! The API has a maxItems = 100 limit on ClusterDecisions, so it may be in conflict with the status updating process if the customer picks over 100 clusters. This does not concern the node threshold much I think; the threshold does not kick unless the cluster has over 100 nodes IIRC. I guess it's kind of safe right now for us to assume that most fleets will not have over 100 member clusters (the design target was 500?)

[zhiying-lin]

LGTM
nit, i feel maxClusterDecisionCount is better than the current one. leaving the decision to you :)

[michaelawyu]

Rebased to solve conflicts.

I will keep the old name for now and see if there's a better name.