How to properly handle errors on consumers?

[geovanisouza92]

My current setup is:

• Rails 4 publisher
• JRuby 9k consumer

hutch.yml (configuration):

mq_host: ...
mq_username: guest
mq_password: guest
mq_api_host: ...
channel_prefetch: 1
heartbeat: 150
require_paths:
  - foo.rb
error_handlers:
  - Logger
  - Sentry
error_acknowledgements:
  - Requeuer

Command: hutch --config config/hutch.yaml

I have a consumer executing a process that could raise some errors:

class Foo
  include Hutch::Consumer
  consume 'foo'

  def process(message)
    # process that handle message and could raise errors
  end
end

class Requeuer < Hutch::Acknowledgements::Base
  def handle(delivery_info, properties, broker, ex)
    broker.requeue delivery_info.delivery_tag
    true
  end
end

Hunting down the issue, I:

• Started not handling the errors, but as discussed in Handling Exceptions (not StandardError) #195, this could be dangerous considering the Thread pool nature of AMQP client.
• Then I tried just capturing and requeue'ing the message, but this clashes with @broker.ack on Hutch::Worker#handle_message, sending an requeue and ack messages to RabbitMQ, broking the protocol.
• Then I tried the code above, that requeue and raise the error, but ended with hutch shut down gracefully in the middle of processing.

Considering #165, acking manually seems to be the best option, taking into account that as said in #195, the library could not decided whether if the worker/consumer should crashes due an unhandled exception.

This issue is more a doubt about what is the correct way to handle the errors on consumers?

[michaelklishin]

Consumers should handle their exceptions. I'm not sure if this answers your question but there isn't much Hutch-specific. If you need to inspect protocol method flow or message flow, use Wireshark and RabbitMQ tracing, respectively.

[geovanisouza92]

@michaelklishin Thanks for reply. My issue was not related to protocol itself, but the aspect of handling the errors without stopping worker.

[michaelklishin]

Then perhaps I don't understand the question. Consumers are generally supposed to handle exceptions they can do something about.

[jnarowski]

I am also wondering how to best address this. If there is a syntax error in my code, I want the job to go back into the queue.

Consider this flow:

1. message is received by consumer (and currently acked)
2. There is a syntax error in the consumer code, we push a hotfix for this bug
3. I'd like the message to go back into the queue to be tried again, and be reprocessed once the deploy is pushed and the consumers are reconnected.

How would we handle this logic so failed jobs aren't lost forever?

[michaelklishin]

#177 introduced configurable acknowledgement behavior.