Prefetch while consuming from Stream

[vivek-singh2]

Is there a way to specify Prefetch (Qos) while consuming from stream (super stream)? By default it is showing 0 as Prefetch on Rabbit UI Dashboard against consumer, which I guess is saying unlimited messages. I am running load and wanted to control how many messages to be read from stream. Or if there is other way unlike classic queues then kindly suggest.

[Gsantomaggio]

Hi @vivek-singh2,
There is no public way to control the consumer's message flow.
To receive more messages from the server, the client uses client.Credit(EntityId, 1) command.

rabbitmq-stream-dotnet-client/RabbitMQ.Stream.Client/RawConsumer.cs

Lines 414 to 424 in fa8f66b

	while (_chunksBuffer.Reader.TryRead(out var chunk))
	{
	// We send the credit to the server to allow the server to send more messages
	// we request the credit before process the check to keep the network busy
	try
	{
	if (Token.IsCancellationRequested)
	break;
	await _client.Credit(EntityId, 1).ConfigureAwait(false);
	}
	catch (InvalidOperationException)

At the moment is hidden from the user. You can only configure the Initial credits in this way:

var c = await system.CreateRawConsumer(new RawConsumerConfig(streamsList[0])
                        {
                            OffsetSpec = new OffsetTypeFirst(),
                            InitialCredits = 1, // be careful to use high values here.
                        

The client asks for more credits only when a chuck is terminated.

About:

I am running load and wanted to control how many messages to be read from stream.

Can you please elaborate more on the use case?

[vivek-singh2]

Thanks @Gsantomaggio for inputs.

Initial credit is just used while consumer starts or gets used later as well?

My use case is- I need to consume messages from super stream near real time (offset lag should remain close to 0 or may be few hundreds max) Message rate which is expected to be consumed is 11K per second. I am using three partitions of stream as of now and trying to consume 900 messages per second. But I always see offset lag is increasing at fast pace and reaches to 100K within few mins. Processing at consume end, for each message, is just a DB hit, which is taking 3-4 milliseconds. My aim is to increase consumer throughput and reach 11K consumption rate.

[vivek-singh2]

Ok, Does it mean that consumer will also start getting messages in batch and it will increase messages delivery throughput at consumer end?

[Gsantomaggio]

@vivek-singh2 Ok let me explain.

When you use the standard send(message) the client aggregates the messages and sends them in a single TCP send.
These messages will end up in a chunk. More messages you have in a chunk, and fewer chunks the consumer needs to download to deliver the messages.

The standard Send(message) decides automatically how to aggregate the messages.
With the Send(messages) is up to you to decide the aggregation.

The Send(messages,Compression.Gzip) enables the sub-entry batch. so each group of messages will be compressed ( if you want) and put in a chunk in a consistent way.

So what is the faster way?

• It depends on your use case. We provide different ways to deal with different use cases.

I was doing some tests on the super stream, and I have seen that some parts can be improved. ATM I can't work on it btw.

Hope it is clear. Let me know.

[vivek-singh2]

Thanks @Gsantomaggio, got it now.

[vivek-singh2]

@Gsantomaggio In Continuation to my analysis, would appreciate if you could provide your inputs on below points-

1. As of now, I am manually updating offset count. Is there a concept of auto-commit where client don't need to call update offset.
2. How many partitions are recommended for 11K messages per second? As of now I am following the rule of partitions count= number of service instance.

[Gsantomaggio]

As of now, I am manually updating offset count. Is there a concept of auto-commit where client don't need to call update offset.

There is not at the moment. We can consider to add as we did in Java and Go

How many partitions are recommended for 11K messages per second?

11K It is a low rate. If you have a performance problem, we'd need to understand where the problem is.
The consumer should be able to download these numbers.

You can find me on discord if you want to discuss about that.