Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perform separate queries per queue assigned to worker #46

Merged
merged 2 commits into from
Nov 22, 2023
Merged

Perform separate queries per queue assigned to worker #46

merged 2 commits into from
Nov 22, 2023

Commits on Nov 22, 2023

  1. Perform separate queries per queue assigned to worker 

    We do this in the order specified in the worker configuration, and
by priority within each queue. It's still possible to specify "*",
which means all queues, and in that case the order will be only by
priority. Paused queues aren't queried.

The way it works when there are more than one queue in the list is
as follows. Imagine the worker is configured to run
```
queue1, queue2, queue3
```
In that order. Then:

- Query as most N (N is the worker's capacity, or its number of idle threads)
  jobs from queue1. If we got at least N jobs, stop and assign these to the pool,
  and continue querying jobs from queue1.
- If we got fewer jobs than N, query jobs from queue2. If we reached N, stop and
  post those to the pool, and go back to queue1.
- If we still have space because we haven't filled N, go on to queue3 and do the
  same. Then go back to queue1.
    @rosa
    rosa committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    2b8f732 View commit details
    Browse the repository at this point in the history

  2. Order both by priority and job_id, to ensure deterministic ordering 

    When all priorities are the same.
    @rosa
    rosa committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    78ca4b8 View commit details
    Browse the repository at this point in the history