npm install --save kafka-streams
const {KafkaStreams} = require("kafka-streams");
const config = require("./config.json");
const factory = new KafkaStreams(config);
const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(/* .. */);
kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");
CHANGES: The latest version brings a lot of changes, please check here before updating.
- Quick Start
- Message 'to' and 'from' Apache Kafka
- API Info
- Documentation
- Operator descriptions
- Examples
- Native Client | SSL, SASL, Kerberos
- Prerequisites
- Aim of this Library
- Description
- Port Progress Overview
- Operator Implementations
- Additional Operators
- Stream Action Implementations
- Join Operators Status
- Window Operations
- FAQ - More
- kafka broker should be version
>= 0.11.x
- Node.js should be version
>= 8.x.x
- this is not a 1:1 port of the official JAVA kafka-streams
- the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
- stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example
kafka-streams ๐ equivalent for nodejs โจ๐ข๐โจ build on super fast ๐ฅ observables using most.js ๐ค
ships with sinek ๐ for backpressure
comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features
the lib also comes with a few window
operations that are more similar to Apache Flink,
yet they still feel natural in this api :squirrel:
overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..
async (Promises) and sync stream operators e.g. stream$.map()
or stream$.asyncMap()
super easy API
the lib is based on sinek
, which is based on kafka-node's ConsumerGroups
- core structure
- KStream base - stream as a changelog
- KTable base - stream as a database
- KStream & KTable cloning
- complex stream join structure
- advanced joins see
- windows (for joins) see
- flink like window operations
- word-count example
- more examples
- local-storage for etl actions
- local-storage factory (one per action)
- KStorage example for any DB that supports atomic actions
- backing-up local-storage via kafka
- kafka client implementation
- KTable replay to Kafka (produce)
- stream for topic message production only
- sinek implementation
- backpressure mode for KafkaClient
- auto-json payloads (read-map/write-map)
- auto producer partition and keyed-message handling
- documentation
- API description
- higher join & combine examples
- embed native client
librdkafka
for more performance - SSL
- SASL
- Kerberos
- map
- asyncMap
- constant
- scan
- timestamp
- tap
- filter
- skipRepeats
- skipRepeatsWith
- slice
- take
- skip
- takeWhile
- skipWhile
- until
- since
- reduce
- chainReduce
- forEach (observe)
- chainForEach
- drain
- _zip
- _merge
- _join
- _combine
- _sample
- throttle
- debounce
- delay
- multicast
- A description of the operators can be found here
- Missing an operator? Feel free to open an issue ๐ฎ
- mapStringToArray
- mapArrayToKV
- mapStringToKV
- mapParse
- mapStringify
- atThroughput
- mapWrapKafkaPayload
- mapToFormat
- mapFromFormat
- Want more? Feel free to open an issue ๐ฎ
- countByKey
- sumByKey
- min
- max
- Want more? Feel free to open an issue ๐ฎ
- merge
- outerJoin
- innerJoin
- leftJoin
- branch
- merge
- outerJoin
- innerJoin
- leftJoin
- merge
- outerJoin
- innerJoin
- leftJoin
- window
- advanced window
- rolling window
Yes.
Probably, yes. ๐
Forks or Stars give motivation