refactor(job-queue-plugin): Use multiple BullMQ queues instead of one

[dlhck]

Description

Refactors the bullmq-job-queue-strategy to use a separate BullMQ queue for every Vendure job queue. This should improve the stability and issues with horizontally scaled worker instances. The current implementation pushes all jobs to a single BullMQ queue called vendure-job-queue, which is not best practice.

Resolves #2419 and #1726

Breaking changes

There is no breaking change.

Checklist

📌 Always:

• I have set a clear title
• My PR is small and contains a single feature

👍 Most of the time:

• I have added or updated test cases
• I have updated the README if needed

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	Oct 7, 2024 11:01am

[dlhck]

There is currently one major problem in the core implementation: Every job needs to have a unique ID. But with BullMQ the IDs are only unique within the sequence of a queue. This is the core issue that still needs to be resolved.

Edit: This has been solved by using a combination of the queue name and the BullMQ job ID as a unique ID

[dlhck]

@michaelbromley we are running into an issue, that it is not possible to display a list of all jobs from all queues in all states (aka the default list view). This is because of limitations of the BullMQ library and the way how the Redis keys are structured.

I see two options here:

• Not support the default list view at all
• Change the list view implementation to make it possible to disable the initial state (no filters) and automatically select the first queue (ergo only make it possible to view jobs filtered by queue)