Stuck while scaling AWS spot instances in a "full" zone

[xrl]

We spent 45 minutes today waiting for spot instances to come online. The AZ was fully committed and would not give out spot reservations. I manually changed the scaling order to a different ASG (i.e., different AZ) and we got our servers right away.

Here is what it looked like when the ASG was trying to scale in a fully committed AZ:

I do not have any volume/taint restrictions for these workloads. It's safe to schedule them on any host. Could we make it so ASGs are knocked to the back of the list after a few failed attempts at scaling up?

[seh]

Does your version of the cluster autoscaler have #2235 included?

[Jeffwan]

@xrl Please provides CA version and it would be helpful for us to debug the issue

[xrl]

I just built master and deployed it. I'm running an EKS 1.13 but running master... hopefully that isn't too problematic!

I'm going to close this ticket while I wait for another failure situation in our eu-central-1 region.