landing_job: fix ordering of jobs in queue (bug 1878666) #384
Conversation
zzzeid
commented
Mar 6, 2024
zzzeid
commented
- order first by priority, then by status and creation time
- decrease job priority by 1 when it is deferred
- order first by priority, then by status and creation time - decrease job priority by 1 when it is deferred
| if action == LandingJobAction.DEFER: | ||
| self.priority -= 1 |
There was a problem hiding this comment.
The way this is written now, any job which has its priority lowered will always be at the back of the queue, meaning the patch effectively needs to wait until the queue is empty before it will be considered for landing. This could cause a situation where someone queues their revision for landing, it doesn't land the first time due to some transient issue, then their landing is stuck in limbo because it has a lower priority than newer revisions that are added to the queue. If the queue is empty often enough it's probably not a big deal, but worth noting here.
As an alternative we could note the last attempt time in the landing job, and filter out revisions that have been deferred for a certain duration before attempting to land again. For example we defer a landing job, and then in the job query we filter out all deferred revisions with a last attempted time within the last 15-20 minutes.
| # `LandingJobStatus.SUBMITTED` jobs, higher priority items come first | ||
| # and then we order by creation time (older first). | ||
| q = q.order_by(cls.status.desc(), cls.priority.desc(), cls.created_at) | ||
| q = q.order_by(cls.priority.desc(), cls.status.desc(), cls.created_at) |
There was a problem hiding this comment.
Do we need this change? We already get the highest priority items with the highest priority statuses, right?
