ScaleOutFactor * number of jobs cannot be greater than MaxSize or no agents will be scaled out

[nitrocode]

I was wondering why I wasn't seeing any agents. I went to the lambda's cloudwatch logs and saw this.

2020-10-06T17:47:13.030-04:00 | 2020/10/06 21:47:13 👮‍️ Increasing scale-out of 8 by factor of 2.00
2020-10-06T17:47:13.030-04:00 | 2020/10/06 21:47:13 Scaling OUT 📈 to 16 instances (currently 0)
2020-10-06T17:47:13.142-04:00 | 2020/10/06 21:47:13 Scaling error: ValidationError: New SetDesiredCapacity value 16 is above max value 10 for the AutoScalingGroup.

This looks like an error in the lambda scaler. If this scaler hits a max, instead of failing to scale out any agents, it should scale to the MaxSize set on the asg.

buildkite-agent-scaler/scaler/scaler.go

Lines 226 to 229 in 12d9dfb

	log.Printf("Scaling OUT 📈 to %d instances (currently %d)", desired, current.DesiredCount)
	if err := s.setDesiredCapacity(desired); err != nil {

[yob]

Thanks for flagging this.

As of ~24 hours there's a bug in master that will leave an instance running but with no agent if the terminate-instance exits non-zero. We hope to have this fixed in the next day or two.

[yob]

I've transferred this over to the scaler repo, where the fix would be required.

We think the solution might be to clamp the scale up logic to min(MaxSize, newDesiredSize)

[nitrocode]

That sounds like it would work.

One other issue i noticed is that the lambda output returned a 0 return code even though it returned an error. I've created a separate ticket #36 for this.