Deadlock using :while_executing?

[orlando]

Hello,

I'm using this library to allow to enqueue multiple jobs but only process 1 job of a time for a certain parameter (in our case a model id).

If I enqueue multiple jobs with no workers running, when I start a worker it deadlocks trying to run multiple jobs of the same parameter at the same time.

Any suggestions?

[mhenrixon]

Well first of all you'd have to provide some more details for me to be able to make any suggestions.

Have you tried turning on debug logging? What does your worker class look like? How have you configured the gem? What sidekiq version? Are you using Activejob? etc.

[orlando]

@mhenrixon Hey thanks for jumping in

1. No, I'm logging with puts in the code (sidekiq-unique-jobs code), I saw the debugging and logging section but I don't know how to turn in on
2. Looks like this

class UniqueAchievementCardScoreWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'unique_achievement_card_scores',
    unique: :while_executing,
    unique_args: ->(args) { [args.last] }

  def perform(challenge_attempt_id, child_id)
    challenge_attempt = ChallengeAttempt.find(challenge_attempt_id) # Locks Here
    child_id = challenge_attempt.child_id
    achievement_cards = challenge_attempt.achievement_cards

    # init a Calculator and run update_score
    calculator = UniqueAchievementCardScoreCalculator.new(challenge_attempt)
    calculator.update_score

    # get all parent achievement cards we need to update (remove duplicates and nils)
    parent_achievement_card_ids = achievement_cards.collect(&:parent_id).uniq.compact

    # create one job for each parent
    parent_achievement_card_ids.each do |achievement_card_id|
      CompositeAchievementCardScoreWorker.perform_async(achievement_card_id, child_id)
    end
  end
end

All the code here and the classes don't have anything fancy, just Ruby and ActiveRecord

3. Sidekiq 5.0.4 / UniqueJobs 5.0.9

4. No, I'm not using ActiveJob

Looks like it locks when calling ActiveRecord. I added more connections to the pool but still hangs there. Looking at the puts I placed at while_executing.rb, seems like one job goes through but still hangs when reaching ActiveRecord

[orlando]

Oh one more thing, I'm using postgres with the pg gem and redis 3.2.9

[mhenrixon]

Related to #230

[orlando]

@mhenrixon if you want to replicate this just:

1. enqueue more than 1 job with unique: :while_executing
2. create a worker that uses ActiveRecord
3. bundle exec sidekiq

I just removed the library and works as expected and I'll take a look next week (we are just making a release today and I had to disable unique-jobs to make it happen).

If you find something please let me know

[mhenrixon]

@orlando if you wanted to help test the branch better-runtime-locks that would be amazing. I am reworking the while executing lock but before I optimise anything I want to see if it works better.

[JoelESvensson]

I also get this issue. All jobs lock for exactly 1-3 minutes. I'm using Sidekiq 5.0.4 and Unique Jobs 5.0.10. My use case is very similar to orlando's

[StefanH]

@mhenrixon using unique jobs 5.0.9 I just ran into the same issue probably. It seems that, on 1 sidekiq worker, there can be only 1 uniq while_executing job running at the time. The rest just blocks until the running one is completed.

Looking at the code it seems that 1 global mutex is used around all while_executing locks. This explains why only 1 while_executing job runs on a particular worker.

[mhenrixon]

Please try the master branch if possible. Would be great to hear if that refactoring does anything to improve the situation.