Producer does not retry messages when shutting down

[eapache]

I almost filed this as an enhancement, since this was a design trade-off/simplification I explicitly made when building the current producer, but I guess it qualifies as sufficiently unexpected behaviour (for people other than me) to be considered a bug.

If you call Close or AsyncClose on a producer instance while messages are inflight, and those messages fail to produce (e.g. because leadership has moved), they will not be retried even if they have retries remaining. They will get fed back to the topicDispatcher, which will promptly return them with the confusing error ErrShuttingDown.

[eapache]

There are actually several variants of this problem that are real bugs as well: we can send chaser messages out the errors channel incorrectly, which can cause unexpected behaviour in userspace code which is expecting all messages to conform (especially w.r.t. metadata). This can also cause the internal retry logic to hang itself.