Skip to content
This repository has been archived by the owner on Jan 24, 2024. It is now read-only.

[BUG] KoP can not support FetchRequest for topics with lots of partitions #318

Closed
aloyszhang opened this issue Jan 14, 2021 · 1 comment
Closed

Comments

@aloyszhang
Copy link
Contributor

aloyszhang commented Jan 14, 2021

Describe the bug
Currently, KoP handles FetchRequest by reading all requsted partition in one round. If topic has lots of partitions, e.g. 1000, the response will be very huge. Which will makes the channel un-writablle.

To Reproduce
Steps to reproduce the behavior:

  1. Start pulsr with kop enabled
  2. Create a partitioned topic with lots of partitions, e.g. 1000
  3. Send messages to this topic from kafka client
  4. Consume message from this topic

Expected behavior
KoP should handle this circumstances, even with many partitions, consume should be work normal.

Screenshots
When consumer we can see channel un-writable
image

From the view of client, consume hangs for a period of time, consume may be continue or not.
image

Additional context
Even we disable the logic of configure channel auto read when channelWritabilityChanged like this

 @Override
    public void channelWritabilityChanged(ChannelHandlerContext ctx) throws Exception {
        if (log.isDebugEnabled()) {
            log.debug("Channel writability has changed to: {}", ctx.channel().isWritable());
        }
      /*  if (ctx.channel().isWritable()) {
            // set auto read to true if channel is writable.
            ctx.channel().config().setAutoRead(true);
        } else {
            log.debug("channel is not writable, disable auto reading for back pressing");
            ctx.channel().config().setAutoRead(false);
            ctx.flush();
        }
        ctx.fireChannelWritabilityChanged();*/
    }

the problem still happens

@dockerzhang
Copy link
Contributor

this PR is useless.
9a4be21

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants