No documentation for multiple affinity for Spot/On-Demand

[nitrag]

I can't find any documentation to be able to support multiple autoscaling groups for a single pod with priority. The use case is to attempt scale up with spot instances (Spot ASG) and if that fails (lack of capacity), have a fallback ASG (On-Demand). I see the work at #2235 but can't conceptualize the relationship to the pod config and how cluster-autoscaler works.

Would having:

Two AWS AutoScaling Groups, with near-identical Launch templates except for the spot config.

• zeroscaling-gpu-accelerated-p2-xlarge-spot

 --kubelet-extra-args '--node-labels=nodepool=zeroscaling-gpu-accelerated-p2-xlarge-spot --register-with-taints=specialized=true:NoExecute

• zeroscaling-gpu-accelerated-p2-xlarge

 --kubelet-extra-args '--node-labels=nodepool=zeroscaling-gpu-accelerated-p2-xlarge --register-with-taints=specialized=true:NoExecute

and this Affinity spec work?

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nodepool
            operator: In
            values:
            - zeroscaling-gpu-accelerated-p2-xlarge-spot
            - zeroscaling-gpu-accelerated-p2-xlarge
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: nodepool
            operator: Equals
            value: zeroscaling-gpu-accelerated-p2-xlarge-spot
  tolerations:
    - key: specialized
      operator: Exists

[nitrag]

@piontec any thoughts/help here based upon your work for #2235 ? Or an example how you are handling it?

[nitrag]

Guess not, wish it would.

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#does-ca-respect-node-affinity-when-selecting-node-groups-to-scale-up