Job considered as duplicate after completion only in production

[wojtha]

We are using Sidekiq Unique Jobs for a several months successfully. Recently we've updated Sidekiq from 3.3.4 to 3.5.0 and Sidekiq Unique Jobs from 3.0.14 to 3.0.15. After upgrade we find that background jobs are not working as expected in production: Jobs are now being marked as duplicate even when they are completed. They can be repeated after some significant amount of time (minutes). To make things even more stranger and complicated the unique jobs is working ok in development but failing in production and staging environments.

We have tried to downgrade back to 3.3.4 and 3.0.14. But it doesn't help so the issue was present there maybe for weeks. Unfortunately we don't have automated test for this. So we just know that it worked several weeks back when we were checking it manually.

In staging and production:

Booting Sidekiq 3.3.4 with redis options {:url=>nil}
2015-09-11 11:33 EmbedPlayerBundleJob for site 755, options: {}, topics: 2
done: 0.744 sec
payload is not unique {"class"=>"EmbedPlayerBundleJob", "args"=>[755], "retry"=>false, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"f303ac5c00514b7fb3268f94", "enqueued_at"=>1441965574.2390435, "unique_hash"=>"sidekiq_unique:36921ee9dba3ac68b6dbeced4ddb9d47"}
payload is not unique {"class"=>"EmbedPlayerBundleJob", "args"=>[755], "retry"=>false, "queue"=>"default", "unique"=>true, "log_duplicate_payload"=>true, "jid"=>"42f087d7315d084eafc2a041", "enqueued_at"=>1441965592.0896568, "unique_hash"=>"sidekiq_unique:36921ee9dba3ac68b6dbeced4ddb9d47"}

In development it is always ok:

Booting Sidekiq 3.5.0 with redis options {:url=>nil}
start
2015-09-11 10:57 EmbedPlayerBundleJob for site 755, options: {}, topics: 2
done: 1.388 sec
start
2015-09-11 10:58 EmbedPlayerBundleJob for site 755, options: {}, topics: 2
done: 0.252 sec
start
2015-09-11 10:58 EmbedPlayerBundleJob for site 755, options: {}, topics: 2
done: 0.432 sec

Job definition:

class EmbedPlayerBundleJob
  include Sidekiq::Worker
  sidekiq_options retry: false , unique: true, log_duplicate_payload: true

  def perform(site_id, options = {})
    ActiveRecord::Base.connection_pool.with_connection do
      # processing
    end
  end
end

Environment

Ruby 2.1.4
Rails 4.1.13
Unicorn 4.9.0
Sidekiq 3.5.0/3.3.4
Sidekiq Unique Jobs 3.0.15/3.0.14
Redis server v=3.0.3 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=3997b92a93e9adb8

Any clue what could be causing this or how can I debug it?

[mhenrixon]

try version 4.0.0 and report back if still an issue