Messaging Gateway

[sijie]

Motivation

There are already many messaging applications written based on other messaging protocols, such as Kafka, MQTT and other customized messaging protocol. For those messaging applications to leverage the features/advantages of Pulsar, people have to rewrite their applications to use Pulsar API. That creates barriers for people to tryout.

As Pulsar's messaging model is a flexible and unified messaging model, which already unifies queuing and streaming into one messaging model and API. So it should be able to adopt to many other messaging models. Pulsar's tiered architecture makes it self a great messaging backend for different messaging protocols, so it would be great that Pulsar can provide a gateway/proxy to allow pluginning different protocols. So Pulsar can naturally support the messaging applications that already written in other messaging protocols/apis, such as Kafka, MQTT.

This is the master issue for tracking the idea for providing a messaging gateway in Pulsar to support different type of messaging protocols. Also it would be great to collect the feedback from Pulsar users or people who is looking into using Pulsar.

[Inilien]

It would be great to be able to just plug in Pulsar in existing network of corporation messages where Kafka is currently in use. Right now, the only solution I have found is to manually recompile all other applications connected to Kafka with Kafka Wrapper which is only available for Java applications by the way.

With that feature implemented I could use for example Elastic Beats with Pulsar as well, since they added Kafka as supported output awhile ago.

I sincerely believe this is the must have feature at least until Pulsar will not be popular enough that every application out there would have to implement support for Pulsar in order to satisfy community's demand.

[haphut]

It would be great if Pulsar could act as an MQTT broker while storing the received messages.

We at HSL have sources sending us data over MQTT and we publish open APIs over MQTT. For example, our fleet produces High-frequency positioning (HFP) MQTT messages with QoS 1. Including the confidential messages, we get roughly 1000-2000 messages per second into the HFP topic tree. We'd like to store those messages in Pulsar.

We'd also like to filter out the repeated messages that occur due to QoS 1. As Pulsar doesn't offer compaction that would retain only the first message for the key instead of the last, we'd have to build that kind of compaction in a Pulsar client.

Additionally, we'd like to avoid losing data due to MQTT subscriber crashes and this problem: "Effectively-once publishing in practice only makes sense when the messages are coming from a replayable source as opposed to a non-replayable source (for example online HTTP requests). For non-replayable sources, there’s no way to re-send the previous pending messages after a crash."

An approach for not losing data is to deploy several MQTT subscribers into different availability zones forwarding messages into the same Pulsar topic. The key of each Pulsar message would contain at least the digest of the payload. Again the variant of topic compaction that would retain the first message for each key would allow deduplicating the repeated data. This feels overly complicated, though, just for storing the messages.

MQTT seems to be on the rise in public transport in the Nordics. Off the top of my head I can see several reasons:

1. The protocol standard and several existing broker products avoids vendor lock-in.
2. It's fairly easy to understand the basics of MQTT.
3. Safely mixing public and confidential uses on the same MQTT broker is not a problem.
4. Easy demarcation of responsibility between data providers and consumers aids in demanding MQTT in contracts.

Unfortunately, storing the delivered data is often ignored in the contract phase but almost all of the data needs to be stored. There could be a market for an MQTT broker that acked messaged only after storing them, not after receiving them.

[sijie]

For anyone who is interested in this topic, we have just started a new PIP - PIP-41 for introducing pluggable protocol handlers to support different types of messaging protocols.

[sijie]

For people who are interested in this feature, we have introduced a general framework in 2.5.0 to support pluggable protocol handlers. We also developed a Kafka protocol handler (Kafka-on-Pulsar, #KoP) using this framework - https://github.com/streamnative/kop. If you are interested in the Kafka compatibility layer, please check it out.

We are also using the same framework to develop the protocol support for AMQP and MQTT. Please stay tuned.

[sijie]

Close this issue since the protocol handler mechanism has been introduced in Pulsar.

Currently Kafka and AMQP protocols are supported from this framework.

KoP: https://github.com/streamnative/kop
AoP: https://github.com/streamnative/aop