RemovePodsViolatingTopologySpreadConstraints different setting for constraints

[a7i]

Is your feature request related to a problem? Please describe.

We would to define a different behavior for TopologySpreadConstraints when it's DoNotSchedule vs ScheduleAnyway.

This is because DoNotSchedule triggers a scale-up with cluster autoscaler, but ScheduleAnyway does not.
As a result, I do not want to do a nodeFit for DoNotSchedule but want it enabled for ScheduleAnyway.

Describe the solution you'd like

Ideally we change the Descheduler config API to allow for granular constraints filter.

profiles:
  - name: Soft
    pluginConfig:
    - name: "DefaultEvictor"
    - name: "RemovePodsViolatingTopologySpreadConstraint"
      args:
        constraints:
          - "ScheduleAnyway"
        # make sure there is capacity before evicting, otherwise scheduler will just put it back on the same node
        nodeFit: true 
  - name: Hard
    pluginConfig:
    - name: "DefaultEvictor"
    - name: "RemovePodsViolatingTopologySpreadConstraint"
      args:
        constraints:
          - "DoNotSchedule"
        # do not perform a nodeFit to ensure that cluster-autoscaler does a scale-up when it sees failedscheduling
        nodeFit: false

We can also deprecate includeSoftConstraints arg, enable it when args.constraints includes ScheduleAnyway and eventually remove it in the next API.

Describe alternatives you've considered

I cannot run two separate profiles or two separate descheduler instances, because there is no way to disable the hard constraint.

What version of descheduler are you using?

descheduler version: 0.24.x but plan on upgrading to 0.26.0

Additional context

[damemi]

Makes sense to me, the goal of the new config is better configurability.

We originally held off on supporting soft constraints because they are best-effort, so didn't want to confuse users (why is my pod ending up on the same node?). But now that we support them, there's no difference in supporting just them.

[a7i]

/assign