From 8e40b15aeb7431cc4914d69be4668dfc664fa913 Mon Sep 17 00:00:00 2001 From: Christopher Grainger Date: Thu, 18 Jan 2024 15:41:11 +0100 Subject: [PATCH 1/2] Provide more clarity on the length of list --- lib/broadway.ex | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/broadway.ex b/lib/broadway.ex index ef65a74..49fe28d 100644 --- a/lib/broadway.ex +++ b/lib/broadway.ex @@ -815,9 +815,14 @@ defmodule Broadway do instead of doing it once per message, you can do it on this callback. The length of the list of messages received by this callback is based on - the `min_demand`/`max_demand` configuration in the processor. This callback - must always return all messages it receives, as `c:handle_message/3` is still - called individually for each message afterwards. + the `min_demand`/`max_demand` configuration in the processor. That is, this + will determine how frequently more items are requested. The length of the + list of messages ultimately depends on the producer. For example, the + `BroadwayRabbitMQ.Producer` is a streaming producer and it is more likely for + messages to appear as they are pushed. + + This callback must always return all messages it receives, as + `c:handle_message/3` is still called individually for each message afterwards. """ @callback prepare_messages(messages :: [Message.t()], context :: term) :: [Message.t()] From ad708915e48cf81e2141aa7d15ef2a0a697ec04e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jos=C3=A9=20Valim?= Date: Thu, 18 Jan 2024 15:58:24 +0100 Subject: [PATCH 2/2] Update broadway.ex --- lib/broadway.ex | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/broadway.ex b/lib/broadway.ex index 49fe28d..3b2ab0b 100644 --- a/lib/broadway.ex +++ b/lib/broadway.ex @@ -814,12 +814,12 @@ defmodule Broadway do by `c:handle_message/3`. For example, if you need to query the database, instead of doing it once per message, you can do it on this callback. - The length of the list of messages received by this callback is based on - the `min_demand`/`max_demand` configuration in the processor. That is, this - will determine how frequently more items are requested. The length of the - list of messages ultimately depends on the producer. For example, the - `BroadwayRabbitMQ.Producer` is a streaming producer and it is more likely for - messages to appear as they are pushed. + The length of the list of messages received by this callback is often based + on the `min_demand`/`max_demand` configuration in the processor but ultimately + it depends on the producer and the frequency data arrives. A pipeline that + receives one message per minute will most likely emit single element lists. + Producers which are push-based, rather than pull-based, such as + `BroadwayRabbitMQ.Producer`, are more likely to send messages as they appear. This callback must always return all messages it receives, as `c:handle_message/3` is still called individually for each message afterwards.