Rename to unique: to lock: (#295)

* Update reek and rubocop configuration * Rename unique: to lock: To better match the lock_expiration and lock_timeout options * Use a strategy for conflict resolution This should scale better and be easier to understand/maintain * Updates README with the latest changes * Add missing exception and strategy method
mhenrixon · Jul 21, 2018 · 22cfbc8 · 22cfbc8
1 parent 11a29b8
commit 22cfbc8
Show file tree

Hide file tree

Showing 113 changed files with 876 additions and 609 deletions.
diff --git a/.codeclimate.yml b/.codeclimate.yml
@@ -8,22 +8,21 @@ plugins:
     config:
       languages:
         - ruby
-  # editorconfig:
-  #   enabled: true
-  #   channel: beta
-  #   config:
-  #     editorconfig: .editorconfig
   flog:
     enabled: true
   fixme:
     enabled: true
   markdownlint:
-      enabled: true
+    enabled: true
   reek:
     enabled: true
+    config:
+      file: .reek.yml
   rubocop:
     enabled: true
-    channel: rubocop-0-54
+    channel: rubocop-0-57
+    config:
+      file: .rubocop.yml
 exclude_patterns:
   - Gemfile
   - "*.gemspec"

diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -18,7 +18,7 @@ What happens instead of the expected behavior?
 ```ruby
 class MyWorker
   include Sidekiq::Worker
-  sidekiq_options unique: :until_executed, queue: :undefault
+  sidekiq_options lock: :until_executed, queue: :undefault
   def perform(args); end
 
   def self.unique_args(args)

diff --git a/.reek.yml b/.reek.yml
@@ -11,53 +11,15 @@ detectors:
     exclude:
     - Sidekiq#self.use_options
   IrresponsibleModule:
+    enabled: false
+  LongParameterList:
+    enabled: true
     exclude:
-    - Sidekiq::Simulator
-    - SidekiqUniqueJobs::Cli
-    - SidekiqUniqueJobs::Client::Middleware
-    - SidekiqUniqueJobs
-    - Array
-    - Hash
-    - SidekiqUniqueJobs::JidMissing
-    - SidekiqUniqueJobs::LockTimeout
-    - SidekiqUniqueJobs::MaxLockTimeMissing
-    - SidekiqUniqueJobs::RunLockFailed
-    - SidekiqUniqueJobs::ScriptError
-    - SidekiqUniqueJobs::UnexpectedValue
-    - SidekiqUniqueJobs::UniqueKeyMissing
-    - SidekiqUniqueJobs::UnknownLock
-    - SidekiqUniqueJobs::Lock::BaseLock
-    - SidekiqUniqueJobs::Lock::UntilAndWhileExecuting
-    - SidekiqUniqueJobs::Lock::UntilExecuted
-    - SidekiqUniqueJobs::Lock::UntilExecuting
-    - SidekiqUniqueJobs::Lock::UntilExpired
-    - SidekiqUniqueJobs::Lock::WhileExecuting
-    - SidekiqUniqueJobs::Lock::WhileExecutingReject
-    - SidekiqUniqueJobs::Lock::WhileExecutingRequeue
-    - SidekiqUniqueJobs::Locksmith
-    - SidekiqUniqueJobs::Middleware
-    - SidekiqUniqueJobs::Normalizer
-    - SidekiqUniqueJobs::Scripts
-    - SidekiqUniqueJobs::Server::Middleware
-    - Sidekiq::Job
-    - Sidekiq::Job::UniqueExtension
-    - Sidekiq::JobSet
-    - Sidekiq::JobSet::UniqueExtension
-    - Sidekiq::Queue
-    - Sidekiq::Queue::UniqueExtension
-    - Sidekiq::ScheduledSet
-    - Sidekiq::ScheduledSet::UniqueExtension
-    - Sidekiq::SortedEntry
-    - Sidekiq::SortedEntry::UniqueExtension
-    - Sidekiq
-    - Sidekiq::Worker
-    - Sidekiq::Worker::ClassMethods
-    - Sidekiq::Worker::Overrides
-    - Sidekiq::Worker::Overrides::Testing
-    - SidekiqUniqueJobs::Timeout::Calculator
-    - SidekiqUniqueJobs::Timeout
-    - SidekiqUniqueJobs::Unlockable
-    - SidekiqUniqueJobs::Util
+    - initialize
+    - Hash#slice
+    - SidekiqUniqueJobs::Locksmith#create_lock
+    - SidekiqUniqueJobs::Locksmith#expire_when_necessary
+    - SidekiqUniqueJobs::Client::Middleware#call
   TooManyStatements:
     exclude:
     - initialize
@@ -84,7 +46,7 @@ detectors:
     exclude:
     - Hash#slice
     - Hash#slice!
-    - SidekiqUniqueJobs::Lock::WhileExecutingReject#deadset_kill?
+    - SidekiqUniqueJobs::OnConflict::Reject#deadset_kill?
     - SidekiqUniqueJobs::SidekiqWorkerMethods#worker_method_defined?
   MissingSafeMethod:
     exclude:
@@ -93,7 +55,7 @@ detectors:
     enabled: false
   FeatureEnvy:
     exclude:
-    - SidekiqUniqueJobs::Lock::WhileExecutingReject#push_to_deadset
+    - SidekiqUniqueJobs::OnConflict::Reject#push_to_deadset
     - SidekiqUniqueJobs::Logging#debug_item
     - SidekiqUniqueJobs::Util#batch_delete
   NestedIterators:

diff --git a/.rubocop.yml b/.rubocop.yml
@@ -64,6 +64,9 @@ RSpec/ExampleLength:
 RSpec/ExpectActual:
   Enabled: false
 
+RSpec/ExpectChange:
+  Enabled: false
+
 RSpec/ExpectInHook:
   Enabled: false
 

diff --git a/README.md b/README.md
@@ -6,6 +6,7 @@
 * [Documentation](#documentation)
 * [Requirements](#requirements)
 * [Installation](#installation)
+* [Support Me](#support-me)
 * [General Information](#general-information)
 * [Options](#options)
    * [Lock Expiration](#lock-expiration)
@@ -18,6 +19,7 @@
    * [Until Timeout](#until-timeout)
    * [Unique Until And While Executing](#unique-until-and-while-executing)
    * [While Executing](#while-executing)
+* [Conflict Strategy](#conflict-strategy)
 * [Usage](#usage)
    * [Finer Control over Uniqueness](#finer-control-over-uniqueness)
    * [After Unlock Callback](#after-unlock-callback)
@@ -65,6 +67,11 @@ Or install it yourself as:
 
     $ gem install sidekiq-unique-jobs
 
+
+## Support Me
+
+Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link https://paypal.me/mhenrixon. Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
+
 ## General Information
 
 See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
@@ -150,31 +157,31 @@ Locks from when the client pushes the job to the queue. Will be unlocked before
 **NOTE** this is probably not so good for jobs that shouldn't be running simultaneously (aka slow jobs).
 
 ```ruby
-sidekiq_options unique: :until_executing
+sidekiq_options lock: :until_executing
 ```
 
 ### Until Executed
 
 Locks from when the client pushes the job to the queue. Will be unlocked when the server has successfully processed the job.
 
 ```ruby
-sidekiq_options unique: :until_executed
+sidekiq_options lock: :until_executed
 ```
 
 ### Until Timeout
 
 Locks from when the client pushes the job to the queue. Will be unlocked when the specified timeout has been reached.
 
 ```ruby
-sidekiq_options unique: :until_expired
+sidekiq_options lock: :until_expired
 ```
 
 ### Unique Until And While Executing
 
 Locks when the client pushes the job to the queue. The queue will be unlocked when the server starts processing the job. The server then goes on to creating a runtime lock for the job to prevent simultaneous jobs from being executed. As soon as the server starts processing a job, the client can push the same job to the queue.
 
 ```ruby
-sidekiq_options unique: :until_and_while_executing
+sidekiq_options lock: :until_and_while_executing
 ```
 
 ### While Executing
@@ -184,7 +191,7 @@ With this lock type it is possible to put any number of these jobs on the queue,
 **NOTE** Unless this job is configured with a `lock_timeout: nil` or `lock_timeout: > 0` then all jobs that are attempted to be executed will just be dropped without waiting.
 
 ```ruby
-sidekiq_options unique: :while_executing, lock_timeout: nil
+sidekiq_options lock: :while_executing, lock_timeout: nil
 ```
 
 There is an example of this to try it out in the `rails_example` application. Run `foreman start` in the root of the directory and open the url: `localhost:5000/work/duplicate_while_executing`.
@@ -206,12 +213,42 @@ In the console you should see something like:
 10:33:04 worker.1 | 2017-04-23T08:33:04.973Z 84404 TID-ougq8cs8s WhileExecutingWorker JID-9e197460c067b22eb1b5d07f INFO: done: 40.014 sec
 ```
 
+## Conflict Strategy
+
+Decides how we handle conflict. We can either reject the job to the dead queue or reschedule it. Both are useful for jobs that absolutely need to run and have been configured to use the lock `WhileExecuting` that is used only by the sidekiq server process. 
+
+The last one is log which can be be used with the lock `UntilExecuted` and `UntilExpired`. Now we write a log entry saying the job could not be pushed because it is a duplicate of another job with the same arguments
+
+### Raise
+
+This strategy is intended to be used with `WhileExecuting`. Basically it will allow us to let the server process crash with a specific error message and be retried without messing up the Sidekiq stats.
+
+`sidekiq_options lock: :while_executing, on_conflict: :raise, retry: 10`
+
+### Reject
+
+This strategy is intended to be used with `WhileExecuting` and will push the job to the dead queue on conflict.
+
+`sidekiq_options lock: :while_executing, on_conflict: :reject`
+
+### Reschedule
+
+This strategy is intended to be used with `WhileExecuting` and will delay the job to be tried again in 5 seconds. This will mess up the sidekiq stats but will prevent exceptions from being logged and confuse your sysadmins.
+
+`sidekiq_options lock: :while_executing, on_conflict: :reschedule`
+
+### Log
+
+This strategy is intended to be used with `UntilExecuted` and `UntilExpired`. It will log a line about that this is job is a duplicate of another.
+
+`sidekiq_options lock: :until_executed, on_conflict: :log`
+
 ## Usage
 
 All that is required is that you specifically set the sidekiq option for _unique_ to a valid value like below:
 
 ```ruby
-sidekiq_options unique: :while_executing
+sidekiq_options lock: :while_executing
 ```
 
 Requiring the gem in your gemfile should be sufficient to enable unique jobs.
@@ -227,7 +264,7 @@ The method or the proc can return a modified version of args without the transie
 ```ruby
 class UniqueJobWithFilterMethod
   include Sidekiq::Worker
-  sidekiq_options unique: :until_and_while_executing,
+  sidekiq_options lock: :until_and_while_executing,
                   unique_args: :unique_args # this is default and will be used if such a method is defined
 
   def self.unique_args(args)
@@ -240,7 +277,7 @@ end
 
 class UniqueJobWithFilterProc
   include Sidekiq::Worker
-  sidekiq_options unique: :until_executed,
+  sidekiq_options lock: :until_executed,
                   unique_args: ->(args) { [ args.first ] }
 
   ...
@@ -253,7 +290,7 @@ It is also quite possible to ensure different types of unique args based on cont
 ```ruby
 class UniqueJobWithFilterMethod
   include Sidekiq::Worker
-  sidekiq_options unique: :until_and_while_executing, unique_args: :unique_args
+  sidekiq_options lock: :until_and_while_executing, unique_args: :unique_args
 
   def self.unique_args(args)
     if Sidekiq::ProcessSet.new.size > 1
@@ -277,7 +314,7 @@ If you need to perform any additional work after the lock has been released you
 ```ruby
 class UniqueJobWithFilterMethod
   include Sidekiq::Worker
-  sidekiq_options unique: :while_executing,
+  sidekiq_options lock: :while_executing,
 
   def after_unlock
    # block has yielded and lock is released
@@ -293,7 +330,7 @@ To see logging in sidekiq when duplicate payload has been filtered out you can e
 ```ruby
 class UniqueJobWithFilterMethod
   include Sidekiq::Worker
-  sidekiq_options unique: :while_executing,
+  sidekiq_options lock: :while_executing,
                   log_duplicate_payload: true
 
   ...

diff --git a/examples/another_unique_job.rb b/examples/another_unique_job.rb
@@ -4,8 +4,10 @@
 
 class AnotherUniqueJob
   include Sidekiq::Worker
-  sidekiq_options queue: :working2, retry: 1, backtrace: 10,
-                  unique: :until_executed
+  sidekiq_options backtrace: 10,
+                  lock: :until_executed,
+                  queue: :working2,
+                  retry: 1
 
   def perform(args)
     args

diff --git a/examples/custom_queue_job_with_filter_method.rb b/examples/custom_queue_job_with_filter_method.rb
@@ -5,7 +5,7 @@
 require_relative 'custom_queue_job'
 
 class CustomQueueJobWithFilterMethod < CustomQueueJob
-  sidekiq_options unique: :until_executed, unique_args: :args_filter
+  sidekiq_options lock: :until_executed, unique_args: :args_filter
 
   def self.args_filter(args)
     args.first

diff --git a/examples/custom_queue_job_with_filter_proc.rb b/examples/custom_queue_job_with_filter_proc.rb
@@ -7,7 +7,7 @@
 class CustomQueueJobWithFilterProc < CustomQueueJob
   # slightly contrived example of munging args to the
   # worker and removing a random bit.
-  sidekiq_options unique: :until_expired,
+  sidekiq_options lock: :until_expired,
                   unique_args: (lambda do |args|
                     options = args.extract_options!
                     options.delete('random')

diff --git a/examples/expiring_job.rb b/examples/expiring_job.rb
@@ -4,7 +4,7 @@
 
 class ExpiringJob
   include Sidekiq::Worker
-  sidekiq_options unique: :until_executed, lock_expiration: 10 * 60
+  sidekiq_options lock: :until_executed, lock_expiration: 10 * 60
 
   def perform(one, two)
     [one, two]

diff --git a/examples/inline_worker.rb b/examples/inline_worker.rb
@@ -4,7 +4,7 @@
 
 class InlineWorker
   include Sidekiq::Worker
-  sidekiq_options unique: :while_executing, lock_timeout: 5
+  sidekiq_options lock: :while_executing, lock_timeout: 5
 
   def perform(one)
     TestClass.run(one)

diff --git a/examples/just_a_worker.rb b/examples/just_a_worker.rb
@@ -5,7 +5,7 @@
 class JustAWorker
   include Sidekiq::Worker
 
-  sidekiq_options unique: :until_executed, queue: :testqueue
+  sidekiq_options lock: :until_executed, queue: :testqueue
 
   def perform(options = {})
     options

diff --git a/examples/long_running_job.rb b/examples/long_running_job.rb
@@ -4,8 +4,10 @@
 
 class LongRunningJob
   include Sidekiq::Worker
-  sidekiq_options queue: :customqueue, retry: true, unique: :until_and_while_executing,
-                  lock_expiration: 7_200, retry_count: 10
+  sidekiq_options lock: :until_and_while_executing,
+                  lock_expiration: 7_200,
+                  queue: :customqueue,
+                  retry: 10
   def perform(one, two)
     [one, two]
   end

diff --git a/examples/main_job.rb b/examples/main_job.rb
@@ -4,8 +4,9 @@
 
 class MainJob
   include Sidekiq::Worker
-  sidekiq_options queue: :customqueue, unique: :until_executed,
-                  log_duplicate_payload: true
+  sidekiq_options lock: :until_executed,
+                  log_duplicate_payload: true,
+                  queue: :customqueue
 
   def perform(arg)
     [arg]

diff --git a/examples/my_unique_job.rb b/examples/my_unique_job.rb
@@ -4,11 +4,10 @@
 
 class MyUniqueJob
   include Sidekiq::Worker
-  sidekiq_options queue: :customqueue,
-                  retry: true,
-                  retry_count: 10,
-                  unique: :until_executed,
-                  lock_expiration: 7_200
+  sidekiq_options lock: :until_executed,
+                  lock_expiration: 7_200,
+                  queue: :customqueue,
+                  retry: 10
 
   def perform(one, two)
     [one, two]