Add RequeueWhileExecuting strategy

[pnomolos]

Similar to the WhileExecuting strategy, but rather than sleep 0.1 while another job is executing it will do a sleep some_interval then push the job back on the queue. I realize this will mess up Sidekiq's counts, but it's a much better strategy when dealing with long-running jobs where there exists the potential for many non-unique jobs to be pushed on at the same time.

I've already written the majority of the worker - will submit a PR with tests if this would be a good idea 👍

[mensfeld]

@pnomolos I would love to use that one as well. Strategy like that is much much better.

[mensfeld]

@mhenrixon this does the job:

module SidekiqUniqueJobs
  module Lock
    class WhileExecutingReschedule < WhileExecuting
      MUTEX = Mutex.new

      def synchronize
        MUTEX.lock
        @reschedule = true

        unless locked?
          sleep 0.1
          @item['class'].constantize.perform_async(*@item['args'])
          @reschedule = false
          return
        end

        yield
      rescue Sidekiq::Shutdown
        logger.fatal { "the unique_key: #{@unique_digest} needs to be unlocked manually" }
        raise
      ensure
        if @reschedule
          SidekiqUniqueJobs.connection(@redis_pool) { |conn| conn.del @unique_digest }
        end

        MUTEX.unlock
      end
    end
  end
end

[avinasha]

A variant of this strategy, WhileExecutingReject, is also useful.

Use case

We use sidekiq scheduler to schedule periodic jobs. If a job from the previously triggered periodic job is still executing, then we don't want to schedule a new one. We don't want to reschedule for later either as the job will be triggered in the next period.

[mhenrixon]

I’ll get to these two later today

[mhenrixon]

@avinasha try the latest master. It will push jobs to the queue but drop jobs that are trying to execute at the same time.

[mhenrixon]

Since v6.0.0.rc7 there are two ways of achieving this:

class MyWorker
  # Alternative 1 - will mess up the sidekiq counts
  sidekiq_options lock: :while_executing, lock_timeout: 5, on_conflict: :reschedule
  # Alternative 2 - will raise exceptions
  sidekiq_options lock: :while_executing, lock_timeout: 5, on_conflict: :raise
end

Both of these will get you the same result, one of them will push back the job 5 seconds and the other one will put the job on the retry queue. Let me know how that works for you guys.