Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide some configuration DSL for custom Strategies and Locks #383

Merged
merged 7 commits into from
Apr 15, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,14 @@
- [Until Timeout](#until-timeout)
- [Unique Until And While Executing](#unique-until-and-while-executing)
- [While Executing](#while-executing)
- [Custom Locks](#custom-locks)
- [Conflict Strategy](#conflict-strategy)
- [Log](#log)
- [Raise](#raise)
- [Reject](#reject)
- [Replace](#replace)
- [Reschedule](#reschedule)
- [Custom Strategies](#custom-strategies)
- [Usage](#usage)
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
- [After Unlock Callback](#after-unlock-callback)
Expand Down Expand Up @@ -233,6 +235,41 @@ In the console you should see something like:
10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
```

### Custom Locks

You may need to define some custom lock. You can define it in one project folder:

```ruby
# lib/locks/my_custom_lock.rb
module Locks
class MyCustomLock < SidekiqUniqueJobs::Lock::BaseLock
def execute
# Do something ...
end
end
end
```

You can refer on all the locks defined in `lib/sidekiq_unique_jobs/lock`.

In order to make it available, you should call in your project startup:

```ruby
# For rails application
# config/initializers/sidekiq_unique_jobs.rb
# For other projects, whenever you prefer

SidekiqUniqueJobs.configure do |config|
config.add_lock :my_custom_lock, Locks::MyCustomLock
end
```

And then you can use it in the jobs definition:

`sidekiq_options lock: :my_custom_lock, on_conflict: :log`

Please not that if you try to override a default lock, an `ArgumentError` will be raised.

## Conflict Strategy

Decides how we handle conflict. We can either reject the job to the dead queue or reschedule it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the sidekiq server process.
Expand Down Expand Up @@ -274,6 +311,41 @@ This strategy is intended to be used with `WhileExecuting` and will delay the jo

`sidekiq_options lock: :while_executing, on_conflict: :reschedule`

### Custom Strategies

You may need to define some custom strategy. You can define it in one project folder:

```ruby
# lib/strategies/my_custom_strategy.rb
module Strategies
class MyCustomStrategy < OnConflict::Strategy
def call
# Do something ...
end
end
end
```

You can refer on all the startegies defined in `lib/sidekiq_unique_jobs/on_conflict`.

In order to make it available, you should call in your project startup:

```ruby
# For rails application
# config/initializers/sidekiq_unique_jobs.rb
# For other projects, whenever you prefer

SidekiqUniqueJobs.configure do |config|
config.add_strategy :my_custom_strategy, Strategies::MyCustomStrategy
end
```

And then you can use it in the jobs definition:

`sidekiq_options lock: :while_executing, on_conflict: :my_custom_strategy`

Please not that if you try to override a default lock, an `ArgumentError` will be raised.

## Usage

All that is required is that you specifically set the sidekiq option for _unique_ to a valid value like below:
Expand Down
1 change: 1 addition & 0 deletions lib/sidekiq_unique_jobs.rb
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,5 @@
require "sidekiq_unique_jobs/sidekiq_unique_ext"
require "sidekiq_unique_jobs/on_conflict"

require "sidekiq_unique_jobs/config"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the move to a separate file now that it turned complex again! 👍

require "sidekiq_unique_jobs/sidekiq_unique_jobs"
69 changes: 69 additions & 0 deletions lib/sidekiq_unique_jobs/config.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# frozen_string_literal: true

module SidekiqUniqueJobs
# Shared class for dealing with gem configuration
#
# @author Mauro Berlanda <mauro.berlanda@gmail.com>
class Config < Concurrent::MutableStruct.new(
:default_lock_timeout,
:enabled,
:unique_prefix,
:logger,
:locks,
:strategies,
)
DEFAULT_LOCKS = {
until_and_while_executing: SidekiqUniqueJobs::Lock::UntilAndWhileExecuting,
until_executed: SidekiqUniqueJobs::Lock::UntilExecuted,
until_executing: SidekiqUniqueJobs::Lock::UntilExecuting,
until_expired: SidekiqUniqueJobs::Lock::UntilExpired,
until_timeout: SidekiqUniqueJobs::Lock::UntilExpired,
while_executing: SidekiqUniqueJobs::Lock::WhileExecuting,
while_executing_reject: SidekiqUniqueJobs::Lock::WhileExecutingReject,
}.freeze

DEFAULT_STRATEGIES = {
log: SidekiqUniqueJobs::OnConflict::Log,
raise: SidekiqUniqueJobs::OnConflict::Raise,
reject: SidekiqUniqueJobs::OnConflict::Reject,
replace: SidekiqUniqueJobs::OnConflict::Replace,
reschedule: SidekiqUniqueJobs::OnConflict::Reschedule,
}.freeze

# Returns a default configuration
# @return [Concurrent::MutableStruct] a representation of the configuration object
def self.default
new(
0,
true,
"uniquejobs",
Sidekiq.logger,
DEFAULT_LOCKS,
DEFAULT_STRATEGIES,
)
end

# Adds a lock type to the configuration. It will raise if the lock exists already
#
# @param [String] name the name of the lock
# @param [Class] klass the class describing the lock
def add_lock(name, klass)
raise ArgumentError, "Lock #{name} already defined, please use another name" if locks.key?(name.to_sym)

new_locks = locks.dup.merge(name.to_sym => klass).freeze
self.locks = new_locks
end

# Adds an on_conflict strategy to the configuration.
# It will raise if the strategy exists already
#
# @param [String] name the name of the custom strategy
# @param [Class] klass the class describing the strategy
def add_strategy(name, klass)
raise ArgumentError, "strategy #{name} already defined, please use another name" if strategies.key?(name.to_sym)

new_strategies = strategies.dup.merge(name.to_sym => klass).freeze
self.strategies = new_strategies
end
end
end
13 changes: 5 additions & 8 deletions lib/sidekiq_unique_jobs/on_conflict.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,17 +15,14 @@ module SidekiqUniqueJobs
# @author Mikael Henriksson <mikael@zoolutions.se>
#
module OnConflict
STRATEGIES = {
log: OnConflict::Log,
raise: OnConflict::Raise,
reject: OnConflict::Reject,
replace: OnConflict::Replace,
reschedule: OnConflict::Reschedule,
}.freeze
# A convenience method for using the configured strategies
def self.strategies
SidekiqUniqueJobs.strategies
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've used delegation a lot in many places but I honestly prefer a defined method and was planning on refactoring in this direction in some places 👍

end

# returns OnConflict::NullStrategy when no other could be found
def self.find_strategy(strategy)
STRATEGIES.fetch(strategy.to_s.to_sym) { OnConflict::NullStrategy }
strategies.fetch(strategy.to_s.to_sym) { OnConflict::NullStrategy }
end
end
end
17 changes: 6 additions & 11 deletions lib/sidekiq_unique_jobs/options_with_fallback.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,15 @@ module SidekiqUniqueJobs
# 3. worker_class (required, can be anything)
# @author Mikael Henriksson <mikael@zoolutions.se>
module OptionsWithFallback
LOCKS = {
until_and_while_executing: SidekiqUniqueJobs::Lock::UntilAndWhileExecuting,
until_executed: SidekiqUniqueJobs::Lock::UntilExecuted,
until_executing: SidekiqUniqueJobs::Lock::UntilExecuting,
until_expired: SidekiqUniqueJobs::Lock::UntilExpired,
until_timeout: SidekiqUniqueJobs::Lock::UntilExpired,
while_executing: SidekiqUniqueJobs::Lock::WhileExecuting,
while_executing_reject: SidekiqUniqueJobs::Lock::WhileExecutingReject,
}.freeze

def self.included(base)
base.send(:include, SidekiqUniqueJobs::SidekiqWorkerMethods)
end

# A convenience method for using the configured locks
def locks
SidekiqUniqueJobs.locks
end

# Check if unique has been enabled
# @return [true, false] indicate if the gem has been enabled
def unique_enabled?
Expand All @@ -49,7 +44,7 @@ def lock
# @return [SidekiqUniqueJobs::Lock::BaseLock] an instance of a child class
def lock_class
@lock_class ||= begin
LOCKS.fetch(lock_type.to_sym) do
locks.fetch(lock_type.to_sym) do
raise UnknownLock, "No implementation for `lock: :#{lock_type}`"
end
end
Expand Down
26 changes: 13 additions & 13 deletions lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,22 @@ module SidekiqUniqueJobs

module_function

Config = Concurrent::MutableStruct.new(
:default_lock_timeout,
:enabled,
:unique_prefix,
:logger,
)

# The current configuration (See: {.configure} on how to configure)
def config
# Arguments here need to match the definition of the new class (see above)
@config ||= Config.new(
0,
true,
"uniquejobs",
Sidekiq.logger,
)
@config ||= SidekiqUniqueJobs::Config.default
end

# The current strategies
# @return [Hash] the configured strategies
def strategies
config.strategies
end

# The current locks
# @return [Hash] the configured locks
def locks
config.locks
end

# The current logger
Expand Down
64 changes: 64 additions & 0 deletions spec/integration/sidekiq_unique_jobs/configuration_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# frozen_string_literal: true

require "spec_helper"

RSpec.describe SidekiqUniqueJobs do
describe "define custom lock strategies" do
class FoobarJob < MyJob
sidekiq_options lock: :foobar,
queue: :customqueue,
on_conflict: :raise
end

class CustomLock < SidekiqUniqueJobs::Lock::BaseLock
def lock
true
end
end

subject(:middleware_call) do
SidekiqUniqueJobs::Client::Middleware.new.call(worker_class, item, queue) do
true
end
end

let(:queue) { :customqueue }
let(:lock_type) { :foobar }
let(:digest) { "1234567890" }
let(:jid) { "randomjid" }
let(:ttl) { nil }
let(:item) do
{
SidekiqUniqueJobs::UNIQUE_DIGEST_KEY => digest,
SidekiqUniqueJobs::JID_KEY => jid,
SidekiqUniqueJobs::LOCK_EXPIRATION_KEY => ttl,
SidekiqUniqueJobs::LOCK_KEY => lock_type,
}
end
let(:worker_class) { FoobarJob }

context "when the lock is not defined" do
it "raises SidekiqUniqueJobs::UnknownLock" do
expect { middleware_call }.to raise_exception(
SidekiqUniqueJobs::UnknownLock, /No implementation for `lock: :foobar`/
)
end
end

context "when the lock is defined" do
let(:custom_config) do
SidekiqUniqueJobs::Config.default.tap do |cfg|
cfg.add_lock :foobar, CustomLock
end
end

before do
allow(described_class).to receive(:config).and_return(custom_config)
end

it "returns the block given" do
expect(middleware_call).to be(true)
end
end
end
end
Loading