Add scaling to include waiting jobs

[lox]

The agent metrics api now includes a metric for WaitingJobCount which tracks jobs that are behind a wait step. This allows for pre-emptive scaling so that instances are ready by the time that jobs are scheduled.

By default waiting jobs aren't included, but they can be enabled with INCLUDE_WAITING=1 in the lambda env.

[lox]

Any thoughts on this one @etaoins?

[etaoins]

Interesting, I hadn't considered including waiting jobs.

I'm assuming this is off by default because some of the running agents would be free by the time the waiting jobs were unblocked so this would overscale? I wonder if a better default-on heuristic would be:

count := metrics.ScheduledJobs + max(metrics.RunningJobs, metrics.WaitingJobs)

That way we would only scale up to account for waiting jobs if they exceeded the number of running jobs, i.e. once they're unblocked we won't have enough agents to run them.

I don't feel too strongly about this, just throwing it out there 😄

[lox]

Yeah, it's a reasonably new idea for me too. Basically we see lots of pipelines in the form of [upload] -> [long docker build] -> [tests with 40x parallelism]. My hope here was for pre-emptive scaling so that by the time the docker build is done there are some agents to run the tests.

It's a good point about some agents freeing up by the time waiting jobs become scheduled. I like the idea of max(running, waiting). 👌🏻

Really appreciate your feedback and insights on the last few PR's @etaoins.

[etaoins]

No worries, it's good to work with a company that takes open source seriously and I have a vested interest in my builds going faster 😉

Another pattern a see a lot at SEEK is

steps:
  - label: 'Build & Test'
    queue: prod

  - wait

  - label: 'Deploy Dev'
    queue: dev

  - wait

  - label: 'Deploy Prod'
    queue: prod

Because most of our e.g. branch builds happen in the prod queue it's usually already scaled up. I'm assuming including waiters would allow the dev queue to scale up preemptively for the deploy? That could be a big win for us.

[lox]

I'm assuming including waiters would allow the dev queue to scale up preemptively for the deploy? That could be a big win for us.

Yup!