Skip to content

Commit

Permalink
Adding config block and transaction adapter configuration
Browse files Browse the repository at this point in the history
  • Loading branch information
hlascelles committed Jun 1, 2018
1 parent afd1034 commit 125b3ae
Show file tree
Hide file tree
Showing 14 changed files with 79 additions and 27 deletions.
3 changes: 2 additions & 1 deletion .reek
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
---
Attribute:
enabled: true
exclude: []
exclude:
- Que::Scheduler
BooleanParameter:
enabled: false
exclude: []
Expand Down
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
## Unreleased
## 3.1.0 (2018-06-01)

* Addition of a gem config initializer [#29](https://github.com/hlascelles/que-scheduler/pull/29)
* Allow configuration of the transaction block adapter [#29](https://github.com/hlascelles/que-scheduler/pull/29)
* Handling middle overriding enqueue that prevents a job from being enqueued [#28](https://github.com/hlascelles/que-scheduler/pull/28)

## 3.0.0 (2018-05-23)
Expand Down
32 changes: 22 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ needs to be run, enqueueing those jobs, then enqueueing itself to check again la
```ruby
gem 'que-scheduler'
```
1. Specify a schedule config in a yml file (see below). The default location that que-scheduler will
look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler config
1. Specify a schedule in a yml file (see below). The default location that que-scheduler will
look for it is `config/que_schedule.yml`. They are essentially the same as resque-scheduler
files, but with additional features.

1. Add a migration to start the job scheduler and prepare the audit table.
Expand All @@ -35,7 +35,7 @@ files, but with additional features.
## Schedule configuration

The schedule file is a list of que job classes with arguments and a schedule frequency (in crontab
syntax). The format is similar to the resque-scheduler config format, though priorities must be supplied as
syntax). The format is similar to the resque-scheduler format, though priorities must be supplied as
integers, and job classes must be migrated from Resque to Que. Cron syntax can be anything
understood by [fugit](https://github.com/floraison/fugit#fugitcron).

Expand Down Expand Up @@ -111,11 +111,22 @@ A job can have a `schedule_type` assigned to it. Valid values are:
This feature ensures that jobs which *must run* for a certain cron match will always eventually
execute, even after a total system crash, or even a DB backup restore.

## Environment Variables
## Gem configuration

You can configure some aspects of the gem with environment variables.
You can configure some aspects of the gem with an initializer. The default is given below.

* `QUE_SCHEDULER_CONFIG_LOCATION` - The location of the schedule configuration (default `config/que_schedule.yml`)
```ruby
Que::Scheduler.configure do |config|
# The location of the schedule yaml file.
config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
# Specify a transaction block adapter. By default, que-scheduler uses the one supplied by que.
# However, if, for example you rely on listeners to ActiveRecord's exact `transaction` method, or
# Sequel's DB.after_commit helper, then you can supply it here.
config.transaction_adapter = ::Que.method(:transaction)
end
```

## Scheduler Audit

Expand All @@ -134,7 +145,7 @@ in a coherent state with the rest of your database.

## Concurrent scheduler detection

No matter how many tasks you have defined in your config, you will only ever need one que-scheduler
No matter how many tasks you have defined in your schedule, you will only ever need one que-scheduler
job enqueued. que-scheduler knows this, and it will check before performing any operations that
there is only one of itself present.

Expand All @@ -144,15 +155,15 @@ it will rollback correctly and try again. It won't schedule jobs twice for a cro
## How it works
que-scheduler is a job that reads a config file, enqueues any jobs it determines that need to be run,
que-scheduler is a job that reads a schedule file, enqueues any jobs it determines that need to be run,
then reschedules itself. The flow is as follows:
1. The que-scheduler job runs for the very first time.
1. que-scheduler loads the schedule config file. It will not schedule any other jobs, except itself,
1. que-scheduler loads the schedule file. It will not schedule any other jobs, except itself,
as it has never run before.
1. Some time later it runs again. It knows what jobs it should be monitoring, and notices that some
have are due. It enqueues those jobs and then itself. Repeat.
1. After a deploy that changes the config, the job notices any new jobs to schedule, and knows which
1. After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which
ones to forget. It does not need to be re-enqueued or restarted.
## DB Migrations
Expand Down Expand Up @@ -182,6 +193,7 @@ Major feature changes are listed below. The full
the root of the project.
#### Versions 3.x
- Addition of a config initializer.
- Addition of numerous extra columns to the audit table.
- Required cumulative migration: `Que::Scheduler::Migrations.migrate!(version: 4)`
#### Versions 2.x
Expand Down
3 changes: 3 additions & 0 deletions lib/que-scheduler.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# rubocop:disable Naming/FileName
require 'que/scheduler'
# rubocop:enable Naming/FileName
1 change: 1 addition & 0 deletions lib/que/scheduler.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
require 'que/scheduler/version'
require 'que/scheduler/config'
require 'que/scheduler/scheduler_job'
require 'que/scheduler/db'
require 'que/scheduler/audit'
Expand Down
24 changes: 24 additions & 0 deletions lib/que/scheduler/config.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
require 'que'

module Que
module Scheduler
class << self
attr_accessor :configuration
end

def self.configure
self.configuration ||= Configuration.new
yield(configuration)
end

class Configuration
attr_accessor :schedule_location
attr_accessor :transaction_adapter
end
end
end

Que::Scheduler.configure do |config|
config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
config.transaction_adapter = ::Que.method(:transaction)
end
4 changes: 4 additions & 0 deletions lib/que/scheduler/db.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ def count_schedulers
def now
Que.execute(NOW_SQL).first.values.first
end

def transaction
Que::Scheduler.configuration.transaction_adapter.call { yield }
end
end
end
end
Expand Down
10 changes: 4 additions & 6 deletions lib/que/scheduler/defined_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,6 @@
module Que
module Scheduler
class DefinedJob < Hashie::Dash
QUE_SCHEDULER_CONFIG_LOCATION =
ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml')
ERROR_MESSAGE_SUFFIX = "in que-scheduler config #{QUE_SCHEDULER_CONFIG_LOCATION}".freeze

include Hashie::Extensions::Dash::PropertyTranslation

SCHEDULE_TYPES = [
Expand Down Expand Up @@ -55,7 +51,8 @@ def calculate_missed_runs(last_run_time, as_time)

class << self
def defined_jobs
@defined_jobs ||= YAML.safe_load(IO.read(QUE_SCHEDULER_CONFIG_LOCATION)).map do |k, v|
schedule_string = IO.read(Que::Scheduler.configuration.schedule_location)
@defined_jobs ||= YAML.safe_load(schedule_string).map do |k, v|
Que::Scheduler::DefinedJob.new(
{
name: k,
Expand All @@ -73,7 +70,8 @@ def defined_jobs
private

def err_field(field, value)
raise "Invalid #{field} '#{value}' #{ERROR_MESSAGE_SUFFIX}"
schedule = Que::Scheduler.configuration.schedule_location
raise "Invalid #{field} '#{value}' in que-scheduler schedule #{schedule}"
end
end

Expand Down
2 changes: 1 addition & 1 deletion lib/que/scheduler/migrations.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ module Migrations

class << self
def migrate!(version:)
::Que.transaction do
Que::Scheduler::Db.transaction do
current = db_version
if current < version
migrate_up(current, version)
Expand Down
2 changes: 1 addition & 1 deletion lib/que/scheduler/scheduler_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ class SchedulerJob < Que::Job
@priority = 0

def run(options = nil)
::Que.transaction do
Que::Scheduler::Db.transaction do
assert_one_scheduler_job
scheduler_job_args = SchedulerJobArgs.build(options)
logs = ["que-scheduler last ran at #{scheduler_job_args.last_run_time}."]
Expand Down
2 changes: 1 addition & 1 deletion lib/que/scheduler/version.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
module Que
module Scheduler
VERSION = '3.0.0'.freeze
VERSION = '3.1.0'.freeze
end
end
5 changes: 5 additions & 0 deletions spec/que/scheduler/db_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,10 @@ def check(str)
it 'Sequel is not used explicitly' do
check('Sequel')
end

# Check Que.transaction is not used, as the config transaction proc should be used instead
it 'Que.transaction is not used explicitly' do
check('Que.transaction')
end
end
end
8 changes: 4 additions & 4 deletions spec/que/scheduler/defined_job_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def expect_time(from, to, exp)
job_class: 'HalfHourlyTestJob',
cron: 'foo 17 * * *'
)
end.to raise_error(/Invalid cron 'foo 17 \* \* \*' in que-scheduler config/)
end.to raise_error(/Invalid cron 'foo 17 \* \* \*' in que-scheduler schedule/)
end

it 'allows crons with fugit compatible english words' do
Expand Down Expand Up @@ -77,7 +77,7 @@ def expect_time(from, to, exp)
job_class: 'HalfHourlyTestJob',
queue: 3_214_214
)
end.to raise_error(/Invalid queue '3214214' in que-scheduler config/)
end.to raise_error(/Invalid queue '3214214' in que-scheduler schedule/)
end

it 'checks the priority is an integer' do
Expand All @@ -87,7 +87,7 @@ def expect_time(from, to, exp)
job_class: 'HalfHourlyTestJob',
priority: 'foo'
)
end.to raise_error(/Invalid priority 'foo' in que-scheduler config/)
end.to raise_error(/Invalid priority 'foo' in que-scheduler schedule/)
end

it 'checks the job_class is a Que::Job from job_class' do
Expand All @@ -96,7 +96,7 @@ def expect_time(from, to, exp)
name: 'checking_job_types',
job_class: 'NotAQueJob'
)
end.to raise_error(/Invalid job_class 'NotAQueJob' in que-scheduler config/)
end.to raise_error(/Invalid job_class 'NotAQueJob' in que-scheduler schedule/)
end
end
end
6 changes: 4 additions & 2 deletions spec/spec_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@
require 'coveralls'
Coveralls.wear!

ENV['QUE_SCHEDULER_CONFIG_LOCATION'] = "#{__dir__}/config/que_schedule.yml"

# By default, que-scheduler specs run in different timezones with every execution, thanks to
# zonebie. If you want to force one particular timezone, you can use the following:
# ENV['ZONEBIE_TZ'] = 'International Date Line West'
Expand All @@ -17,6 +15,10 @@

Dir["#{__dir__}/../spec/support/**/*.rb"].each { |f| require f }

Que::Scheduler.configure do |config|
config.schedule_location = "#{__dir__}/config/que_schedule.yml"
end

RSpec.configure do |config|
config.example_status_persistence_file_path = '.rspec_status'
config.disable_monkey_patching!
Expand Down

0 comments on commit 125b3ae

Please sign in to comment.