v1.5.0
librdkafka v1.5.0
The v1.5.0 release brings usability improvements, enhancements and fixes to
librdkafka.
Enhancements
- Improved broker connection error reporting with more useful information and
hints on the cause of the problem. - Consumer: Propagate errors when subscribing to unavailable topics (#1540)
- Producer: Add
batch.size
producer configuration property (#638) - Add
topic.metadata.propagation.max.ms
to allow newly manually created
topics to be propagated throughout the cluster before reporting them
as non-existent. This fixes race issues where CreateTopics() is
quickly followed by produce(). - Prefer least idle connection for periodic metadata refreshes, et.al.,
to allow truly idle connections to time out and to avoid load-balancer-killed
idle connection errors (#2845) - Added
rd_kafka_event_debug_contexts()
to get the debug contexts for
a debug log line (by @wolfchimneyrock). - Added Test scenarios which define the cluster configuration.
- Added MinGW-w64 builds (@ed-alertedh, #2553)
./configure --enable-XYZ
now requires the XYZ check to pass,
and--disable-XYZ
disables the feature altogether (@benesch)- Added
rd_kafka_produceva()
which takes an array of produce arguments
for situations where the existingrd_kafka_producev()
va-arg approach
can't be used. - Added
rd_kafka_message_broker_id()
to see the broker that a message
was produced or fetched from, or an error was associated with. - Added RTT/delay simulation to mock brokers.
Upgrade considerations
- Subscribing to non-existent and unauthorized topics will now propagate
errorsRD_KAFKA_RESP_ERR_UNKNOWN_TOPIC_OR_PART
and
RD_KAFKA_RESP_ERR_TOPIC_AUTHORIZATION_FAILED
to the application through
the standard consumer error (the err field in the message object). - Consumer will no longer trigger auto creation of topics,
allow.auto.create.topics=true
may be used to re-enable the old deprecated
functionality. - The default consumer pre-fetch queue threshold
queued.max.messages.kbytes
has been decreased from 1GB to 64MB to avoid excessive network usage for low
and medium throughput consumer applications. High throughput consumer
applications may need to manually set this property to a higher value. - The default consumer Fetch wait time has been increased from 100ms to 500ms
to avoid excessive network usage for low throughput topics. - If OpenSSL is linked statically, or
ssl.ca.location=probe
is configured,
librdkafka will probe known CA certificate paths and automatically use the
first one found. This should alleviate the need to configure
ssl.ca.location
when the statically linked OpenSSL's OPENSSLDIR differs
from the system's CA certificate path. - The heuristics for handling Apache Kafka < 0.10 brokers has been removed to
improve connection error handling for modern Kafka versions.
Users on Brokers 0.9.x or older should already be configuring
api.version.request=false
andbroker.version.fallback=...
so there
should be no functional change. - The default producer batch accumulation time,
linger.ms
, has been changed
from 0.5ms to 5ms to improve batch sizes and throughput while reducing
the per-message protocol overhead.
Applications that require lower produce latency than 5ms will need to
manually setlinger.ms
to a lower value. - librdkafka's build tooling now requires Python 3.x (python3 interpreter).
Fixes
General fixes
- The client could crash in rare circumstances on ApiVersion or
SaslHandshake request timeouts (#2326) ./configure --LDFLAGS='a=b, c=d'
with arguments containing = are now
supported (by @sky92zwq)../configure
arguments now take precedence over cachedconfigure
variables
from previous invocation.- Fix theoretical crash on coord request failure.
- Unknown partition error could be triggered for existing partitions when
additional partitions were added to a topic (@benesch, #2915) - Quickly refresh topic metadata for desired but non-existent partitions.
This will speed up the initial discovery delay when new partitions are added
to an existing topic (#2917).
Consumer fixes
- The roundrobin partition assignor could crash if subscriptions
where asymmetrical (different sets from different members of the group).
Thanks to @ankon and @wilmai for identifying the root cause (#2121). - The consumer assignors could ignore some topics if there were more subscribed
topics than consumers in taking part in the assignment. - The consumer would connect to all partition leaders of a topic even
for partitions that were not being consumed (#2826). - Initial consumer group joins should now be a couple of seconds quicker
thanks expedited query intervals (@benesch). - Fix crash and/or inconsistent subscriptions when using multiple consumers
(in the same process) with wildcard topics on Windows. - Don't propagate temporary offset lookup errors to application.
- Immediately refresh topic metadata when partitions are reassigned to other
brokers, avoiding a fetch stall of up totopic.metadata.refresh.interval.ms
. (#2955) - Memory for batches containing control messages would not be freed when
using the batch consume APIs (@pf-qiu, #2990).
Producer fixes
- Proper locking for transaction state in EndTxn handler.
Checksums
Release asset checksums:
- v1.5.0.zip SHA256
76a1e83d643405dd1c0e3e62c7872b74e3a96c52be910233e8ec02d501fa33c8
- v1.5.0.tar.gz SHA256
f7fee59fdbf1286ec23ef0b35b2dfb41031c8727c90ced6435b8cf576f23a656