v1.6.5

New/Fix
Clarity added for config param for topic/consumer/producer creation (see docs/reference.md). Previously could only use symbol type for config settings, can now also use string/char as described in reference.

v1.6.4

New

• Examples test_producer.q and test_consumer.q have improved formatting/explaination of meta info retrieved & print librdkafka version info
• Improved reference.md (api docs) including clarity around offsets, callbacks, etc
• makefile picks up KFK_ROOT environment variable for setting of location contains include/lib directory of librdkafka
• Support for building with 2.3.0 librdkafka

Fixes

• Example test_offsetc.q consumer program should have added one to manually committed offset
• An error from the following could have the error string corrupted & produced a small memory leak:
  • .kfk.Consumer due to bad config or unable to create
  • .kfk.Producer due to bad config or unable to create
  • .kfk.Topic due to bad config
• Fixed small memory leak on each successful call to .kfk.Consumer,.kfk.Producer,.kfk.Topic.

v1.6.3

Fix:

• Bad config that passes local validation (e.g. depreciated config provided to brokers) could prevent callbacks from firing (e.g. consumer would receive no messages)
• Removed consumer config (fetch.wait.max.ms) from publisher examples

v1.6.2

New:

Can now query broker for low (oldest/beginning) and high (newest/end) offsets for partition.

.kfk.queryWatermark returns 2 element long for low/high watermark.
Takes 4 params :

1. cid is client ID (integer)
2. Topic (sym)
3. Partition (long)
4. Timeout ms (long)
   e.g. .kfk.queryWatermark[client;`test1;0;1000]

Ref: https://docs.confluent.io/3.3.1/clients/librdkafka/rdkafka_8h.html#a4550ff7d014f08406666124573f70495

v1.6.1

Consumer/Producer config can now be provided with a sym list as a configuration value. As as example, see metadata.broker.list in the following config

kfk_cfg:(!) . flip(
    (`group.id;`0);
    (`metadata.broker.list;`localhost:9092`localhost:9091);
    (`fetch.wait.max.ms;`10);
    (`auto.offset.reset;`earliest);
    (`enable.auto.commit;`false);
    (`statistics.interval.ms;`10000)
    );

Previous releases could give an erroneous error when provided with this type.

v1.6.0

Additions:

• '.kfk.Sub' & '.kfk.Subscribe' can now be given a list of symbols/topics to subscribe to in place of the single symbol. The function can also be used as before, where a single topic/symbol can be provided. 'partition' param no longer tries to set partitions/offsets (ignored by kafka). Examples changed to show subscriptions to multiple symbols. e.g.
  q) .kfk.Sub[client;(topic1;topic2);enlist .kfk.PARTITION_UA]
• '.kfk.Assign' can now be used to subscribe to multiple symbol partitions and their offsets. Can be used as before (Symbol!Long dictionary mapping the name of a topic to an associated partition), but now can also be given a Symbol!Dictionary mapping of topic to partitions/offset (dictionary mapping of integer partition to long offset location) e.g. subscribe to test1 (partition 0 from offset 10) and test2 (partition 0 from offset 10)
  q).kfk.Assign[client;`test1`test2!(((1#0i)!1#10);((1#0i)!1#10))]

Fixes:

• 'test_offsetc.q' example should not be using both '.kfk.AssignOffsets' & '.kfk.Sub' calls. '.kfk.AssignOffsets' removed. A client should not be using both manual and dynamically assigned partitions.

v1.5.0

Additions:

• Assignment API support
• Mechanism to allow publishing with headers (Fixes #48)
• Batch publishing of messages (Fixes #30)
• Exposed callback functions for throttling and error events (Fixes #33)

Fixes:

• Previously consumption logic would result in thread starvation if consuming a large group of messages. exposure of MaxMsgsPerPoll and changes to underlying function calling mechanism have removed this issue. (Fixes #37)

Subscriptions to multiple topics

• Addition of support for handling of multiple calls to .kfk.Sub to allow for subscriptions to multiple topics for a single client.
• This functionality has not changed the historical behaviour of any of the functions within the interface.
• Examples have been updated to coincide with the addition of the multiple subscription functionality.

v1.3.0

Additions

• .kfk.Flush function allows user to ensure all 'produce requests' completed, ensuring queued and in-flight requests completed prior to producer destruction
• .kfk.VersionSym return version of librdkafka used in human readable format
• .kfk.ThreadCount returns how many threads are currently in use by librdkafka
• .kfk.SetLoggerLevel specify the maximum logging level produced by internal kafka logging and debugging

Modification

• Both .kfk.CommittedOffsets and .kfk.PositionOffsets now support partition list of int/long/short and/or a dictionary as final parameter.
• Enhanced type checking at C level to ensure types to dictionaries passed in offset functionality is correct

v1.2.0

Build Changes

• Addition of Windows build instructions.
• Modifications to kfk.c to account for windows build.
• Modification to general structure of repository.
• Change to Makefile structure for linux/mac.

Fixes

• log callbacks now on main thread (prevent thread issues that could lead to crash/corruption)

Improvement

• subscription msg callback now includes client id