DM-44606: Convert to arq #163
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rra
force-pushed
the
tickets/DM-44606
branch
9 times, most recently
from
2f169dc
to
d643e63
Compare
Switch the task queuing system from Dramatiq to arq. Restructure the way workers are set up and executed to match the way that arq thinks of the world instead of the way that Dramatiq thinks of the world. This allows task queuing to be async, allowing it to not block the event loop. Add new functions, build_worker and build_uws_worker, that construct the arq worker configuration for the main worker and the UWS database helper workers.
Support gs:// URLs in addition to s3:// URLs with the cutout backend. The hope is that this will add support for workload identity. This support requires installing some more libraries into the worker Docker image.
Exceptions need a bit of special handling to ensure that they can be pickled properly. Also add a bit more logging in the database worker.
Several tests have to manually push queued jobs through their lifecycle in the mock arq queue. Add a helper class that wraps the data structures required to do this and provides a simple interface. This can eventually move into Safir when we move the rest of the support code there.
Document the changes due to the arq conversion and its corresponding Phalanx chart changes. Add a pointer to the change log to README.md.
Until there is a new release of Safir with the accumulated fixes required by the backend (or the arq support has been split into a separate package), install the main branch of Safir into the cutout worker Docker image.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Switch the task queuing system from Dramatiq to arq. Restructure the way workers are set up and executed to match the way that arq thinks of the world instead of the way that Dramatiq thinks of the world. This allows task queuing to be async, allowing it to not block the event loop.
Add new functions,
build_workerandbuild_uws_worker, that construct the arq worker configuration for the main worker and the UWS database helper workers.Add support for
gsURLs so that the image cutout backend can use workload identity when running on Google Cloud.