PIP-52: [pulsar-sever] Add support of dispatch throttling relative to publish-rate

[rdhabalia]

Motivation

With PIP-3 , Pulsar broker already supports to configure dispatch rate-limiting for a given topic. Dispatch-throttling feature allows user to configure absolute dispatch rate based on current publish-rate for a given topic or subscriber, and broker will make sure to dispatch only configured number of messages to the consumers regardless current publish-rate or backlog on that topic.

Current dispatch-rate limiting doesn't consider change in publish-rate so, increasing publish-rate on the topic might be larger than configured dispatch-rate which will cause backlog on the topic and consumers will never be able to catch up the backlog unless user again reconfigured the dispatch-rate based on current publish-rate. Reconfiguring dispatch-rate based on publish-rate requires human interaction and monitoring. Therefore, we need a mechanism to configure dispatch rate relative to the current publish-rate on the topic.

Modification

set-dispatch-rate cli have a flag --relative-to-publish-rate to enable relative dispatch throttling.

pulsar-admin namespaces <property/cluster/namespace> set-dispatch-rate --msg-dispatch-rate 1000 --relative-to-publish-rate

Note:

I will add broker-level configuration and documentation into separate PR.

[sijie]

@rdhabalia do we need this in 2.5.0? Can we move it to 2.6.0 or 2.5.1?

[rdhabalia]

yes, can we do 2.5.0 because we are spending lot of manual monitoring for dispatch throttling and we need this feature for multiple users to make life easy.

and quick question regarding PR build: do you know how can we trigger continuous-integration/appveyor/pr individually ? it's newly added and I am not aware about it.

[aahmed-se]

appveyor is going away don't add job for it. We are discussing plans to move to gihub actions.

[sijie]

@rdhabalia sure

[jiazhai]

@rdhabalia Thanks a lot for the work. looking forward for the following work.