Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamp recurring tasks configuration and management #338

Merged
merged 11 commits into from
Sep 11, 2024

Conversation

rosa
Copy link
Member

@rosa rosa commented Sep 9, 2024

This PR changes how recurring tasks are defined and how they're enqueued. They're no longer part of the dispatchers configuration or enqueued by the dispatchers. Instead, they have their own configuration file (config/recurring.yml, but it can be changed) and their own process, a new scheduler that's just in charge of enqueuing jobs for recurring tasks. Besides, the configuration for each task has been extended, and now it allows optionally specifying a queue and priority, and it also supports providing just a "command" (any Ruby code) instead of requiring a job class and arguments. The command will be eval'ed within a new SolidQueue::RecurringJob provided as well. This is how the new configuration looks like:

a_periodic_job:
  class: MyJob
  args: [ 42, { status: "custom_status" } ]
  schedule: every second
a_cleanup_task:
  command: "DeletedStuff.clear_all"
  queue: cleanup
  priority: 2
  schedule: every day at 9am

`config/recurring.yml`. Run them using the least busy dispatcher
(largest polling interval) or a new default dispatcher if none.

This also introduces some new options for the CLI, to allow for
`--dispatch_only` and `--work_only`, and to skip recurring tasks.
And also setting a description that we can use from Mission Control in
the future.
A Scheduler, instead of Dispatcher, that will be created only when there
are recurring tasks to run.
Calling `set` on the job class returns a `ConfiguredJob`, and calling
`perform_later` on a `ConfiguredJob` doesn't yield the job to a block
passed as parameter, so we can't check if the enqueuing was successful
or not. To work around this, the only way is to instantiate the job
ourselves with the arguments, then call `enqueue` on that, and rely on
the result, which would be what `job_class.perform_later` would yield
to the block.
No need to have a class, can specify this using just a command for which
we'd provide a default RecurringtJob class that will just eval the command.
@rosa rosa force-pushed the improve-recurring-tasks-config branch from 75b76ce to 8df600d Compare September 9, 2024 19:07
rosa added a commit to rails/mission_control-jobs that referenced this pull request Sep 9, 2024
rosa added a commit to rails/mission_control-jobs that referenced this pull request Sep 9, 2024
rosa added a commit to rails/mission_control-jobs that referenced this pull request Sep 9, 2024
rosa added a commit to rails/mission_control-jobs that referenced this pull request Sep 9, 2024
@rosa rosa merged commit cf656a9 into main Sep 11, 2024
8 checks passed
@rosa rosa deleted the improve-recurring-tasks-config branch September 11, 2024 11:37
rosa added a commit to rails/mission_control-jobs that referenced this pull request Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant