Honeykiq

Sidekiq → Honeycomb 🐝

Send Sidekiq-related events to Honeycomb.

Installation

Add this line to your application's Gemfile:

gem 'honeykiq'

Usage

The library provides three use cases:

• Honeykiq::ClientMiddleware
• Honeykiq::ServerMiddleware
• Honeykiq::PeriodicReporter

Honeykiq::ClientMiddleware

Add Honeykiq to your Sidekiq client middleware chain. It will propagate tracing fields when used in combination with Honeykiq::ServerMiddleware.

This middleware is only configured to work with the Honeycomb beeline gem, not with a custom Libhoney client.

# Configure Honeycomb beeline
Honeycomb.configure do |config|
  config.write_key = ENV.fetch('HONEYCOMB_WRITE_KEY')
  config.dataset = ENV.fetch('HONEYCOMB_DATASET')
end

# Add the middleware to Sidekiq chain
Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add Honeykiq::ClientMiddleware
  end
end

# Also add it to the server client chain
Sidekiq.configure_server do |config|
  # Configure the server client, used when a worker enqueues a job itself.
  config.client_middleware do |chain|
    chain.add Honeykiq::ClientMiddleware
  end

  # Configure ServerMiddleware with a tracing mode
  config.server_middleware do |chain|
    # tracing_mode: options are nil (default), :child, :child_trace, :link
    #
    # - nil (default) starts a span and trace when the job executes.
    #
    # - :child starts a span as a child of the enqueuing trace,
    #   requires ClientMiddleware to be configured when enqueuing the job.
    #
    # - :child_trace (experimental) starts a span and a trace
    #   with trace.parent_id set to the enqueuing trace id,
    #   requires ClientMiddleware to be configured when enqueuing the job.
    #
    # - :link (experimental) starts a span and trace
    #   with a link event that points to the enqueuing trace,
    #   requires ClientMiddleware to be configured when enqueuing the job.
    #   https://docs.honeycomb.io/getting-data-in/tracing/send-trace-data/#links
    chain.add Honeykiq::ServerMiddleware, tracing_mode: :child
  end
end

Honeykiq::ServerMiddleware

Add Honeykiq to your Sidekiq server middleware chain. It will send an event to Honeycomb once a job finishes or fails. Have a look at server_middleware.rb to see what kind of information it sends.

# Configure Honeycomb beeline
Honeycomb.configure do |config|
  config.write_key = ENV.fetch('HONEYCOMB_WRITE_KEY')
  config.dataset = ENV.fetch('HONEYCOMB_DATASET')
end

# Add the middleware to Sidekiq chain
Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Honeykiq::ServerMiddleware
  end
end

# Or pass the libhoney client to the middleware
Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Honeykiq::ServerMiddleware,
      libhoney: Libhoney::Client.new(
        write_key: ENV.fetch('HONEYCOMB_WRITE_KEY'),
        dataset: ENV.fetch('HONEYCOMB_DATASET')
      )
  end
end

You can add your own data or functions to the Honeycomb event by subclassing Honeykiq::ServerMiddleware, and overriding the extra_fields method with your own hash. The contents will be serialized into individual items in the event:

class MyServerMiddleware < Honeykiq::ServerMiddleware
  def extra_fields(job) # Sidekiq::Job instance
    {
      my_data: 'evaluated and added to the event after the job has finished/errored',
      my_function: -> { Time.now }
    }
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add MyServerMiddleware
  end
end

Note: If you have long running jobs, an event is only sent to Honeycomb when the job finishes. Therefore, it may appear as though no jobs are currently running. Additionally, if the process receives a SIGKILL then no event is sent about that job, and the job may keep retrying without appearing in Honeycomb. The PeriodicReporter provides visibility for these cases.

Honeykiq::PeriodicReporter

The periodic reporter should be scheduled to report every few seconds, depending on your use case. Every time the #report method is called it will send a total of 1 + P + Q events to Honeycomb where P and Q are the number of processes and queues respectively.

It sends three types of events: instance, process, and queue. Have a look at periodic_reporter.rb to see what kind of information we send for each type.

A setup using clockwork to report every 30 seconds would look like this:

require 'honeycomb-beeline'
require 'clockwork'
require 'honeykiq'

Honeycomb.configure do |config|
  config.write_key = ENV.fetch('HONEYCOMB_WRITE_KEY')
  config.dataset = ENV.fetch('HONEYCOMB_DATASET')
end

module Clockwork
  every(30, 'Honeykiq::PeriodicReporter') do
    Honeykiq::PeriodicReporter.new.report
  end
end

Contributing

Pull requests are very welcome!

Please report bugs in a new issue.

Everyone is expected to follow the code of conduct.

License

The gem is available as open source under the terms of the MIT License.