Creating a Worker

The Sneakers worker DSL was made to be remarkably similar to existing DSL such as Sidekiq or Resque, however where appropriate we deviated from the path dictated by those.

Here is an example of a worker which does simple scraping of Web page titles as they come.

require 'sneakers'
require 'open-uri'
require 'nokogiri'


class TitleScraper
  include Sneakers::Worker
  from_queue 'downloads'

  def work(msg)
    doc = Nokogiri::HTML(open(msg))
    worker_trace "FOUND <#{doc.css('title').text}>"
    ack!
  end
end

Quick overview

You should expect composition all over. This is why a sneaker worker is actually mixed in and not derived from.

In more advanced scenarios you would be able to mix in components of a worker into your own worker class - but this may be too soon to discuss.

Declaring a queue is simply done with from_queue. What you won't see here is that Sneakers will be smart enough to namespace your environment to this. For example under the development environment your queue will really be downloads_development on RabbitMQ. You can always opt-out or opt-in to this, and it's detected automatically from your existing RACK_ENV.

Message

A work unit, or message, will be what ever you pushed to the queue.

A good idea would be to push serialized JSON. This will allow you the flexibility of moving away from Sneakers (gasp!) if you chose to.

Job control

You can signal back what you've decided to do about a specific work item.

• ack! - signal back to RabbitMQ that this is really done. Take the item off the queue forever.
• requeue! - sends the item back to the top of the queue. Good for timeout failures and temporary failures. This is also a dangerous primitive to use (can create an endless loop) so only use if you're certain that it will eventually resolve.
• reject! - special feedback that indicates the message should move to a different queue or dead-letter-box.

Please mind that you have to return this signal from work method. Either make it a last line of work method or use explicit return statement. This will not work:

# DO NOT do it
def work(raw_event)
  Event.create!(JSON.parse(raw_event))
  ack! # this is not returned from method
  logger.info "Good job"
end

# DO NOT do it
def work(raw_event)
  Event.create!(JSON.parse(raw_event))
  logger.info "Good job"
ensure
  ack! # this is not returned from method
end

Concerns

• Logging - assume logger exists within your work body.
• Metrics - assume metrics#increment and metrics#timing(&block) exist in your work body.
• Trace - you can say worker_trace "my msg" in order to get a highly detailed log message which includes thread ID, worker ID, and more fine-grained detail which will help you debug problems should they arise.

Workflow

A worker can also publish messages. This enables a workflow-like situation where a worker may finish its unit of work and declare a new work item to a different queue of a friend worker.

class TitleScraper
  include Sneakers::Worker
  from_queue 'downloads'

  def work(msg)
    doc = Nokogiri::HTML(open(msg))
    worker_trace "FOUND <#{doc.css('title').text}>"
    publish(doc.css('title').text, :to_queue => 'title_classification')
    ack!
  end
end