Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow worker to specify which arguments to include in uniquing hash #12

Merged
merged 3 commits into from
Mar 9, 2013

Conversation

sax
Copy link

@sax sax commented Mar 6, 2013

We're running into an issue where we want to add non-unique arguments to a sidekiq worker, but we'd like the job to still be counted as unique. For instance (pseudocode):

MyWorker.enqueue_delayed(product.id, Time.now, 1.hour.from_now)
sleep 10
MyWorker.enqueue_delayed(product.id, Time.now, 1.hour.from_now)

Time.now is obviously not going to be unique, and so a second job will be scheduled. What I'd really like is to be able to specify which arguments should be included in the hashing algorithm, so that the second job would not be enqueued.

I'm happy to implement this, but wanted to ensure that it was something that would be accepted as a pull request before doing so.

Eric Saxby & Konstantin Gredeskoul added 3 commits March 4, 2013 18:56
Addresses a use case where only certain arguments should be used
in determining if the job is unique or not. The default to hash
all arguments is not always preferred.
- eval is evil
- we realized that Sidekiq adds core extensions to String
  if active_support is not loaded
@sax
Copy link
Author

sax commented Mar 6, 2013

We started using this in production yesterday, with the use case shown above. We run a lot of sidekiq jobs that are either computationally expensive or expensive on our databases. For these types of jobs, we like to schedule them some amount of time in the future, with payload uniquing.

The problem is that sometimes we need to include data in the job payload that makes the job unique even when we'd like it to fold into a previously queued job. For instance, when queuing a change to a user feed, the timestamp of the original job may be needed in order to correctly do work (by doing work on the delta between timestamp and now). New changes to the same user feed queued before the first job runs may be identical to the first job, except for the timestamp. Only the first job needs to actually run in order to correctly update the feed.

This can also help to keep queue counts from exploding when workers need to be stopped for some reason, or when due to various issues queues back up.

mhenrixon added a commit that referenced this pull request Mar 9, 2013
Allow worker to specify which arguments to include in uniquing hash
@mhenrixon mhenrixon merged commit 1cf978f into mhenrixon:master Mar 9, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants