Backlog quota must greater than retention policy, why?

[Shawyeok]

For messages in the backlog (messages that the consumer has not yet acknowledged), I aim to impose a restriction based solely on size (mainly due to considerations regarding the use of disk space), without any time-based limitations, as I cannot assume how users will employ Pulsar.

pulsarctl namespaces set-backlog-quota -l 20G -p consumer_backlog_eviction public/default

For messages in retention (messages that the consumer has already acknowledged), I intend to retain them for a certain period for traceback purposes but not indefinitely. Additionally, a safety size limit is necessary to prevent the occupation of disk space. However, when I set it up this way, I encounter the error code: 412 reason: Retention Quota must exceed the configured backlog quota for the namespace.

pulsarctl namespaces set-retention --size 2G --time 7d public/default
[✖]  code: 412 reason: Retention Quota must exceed configured backlog quota for namespace.

After code blame, the size-based check has been part of Pulsar since its initial commit, whereas the time-based check was introduced in #10401.

My question is the reason behind this limitation, it seem unreasonable for my situation. I'm looking forward to a solution to this and thank you in advance.

[Shawyeok]

Link the code behind this limitation check:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java

Lines 374 to 393 in 7315aeb

	protected boolean checkBacklogQuota(BacklogQuota quota, RetentionPolicies retention) {
	if (retention == null
	|| (retention.getRetentionSizeInMB() <= 0 && retention.getRetentionTimeInMinutes() <= 0)) {
	return true;
	}
	if (quota == null) {
	quota = pulsar().getBrokerService().getBacklogQuotaManager().getDefaultQuota();
	}
	if (retention.getRetentionSizeInMB() > 0
	&& quota.getLimitSize() >= (retention.getRetentionSizeInMB() * 1024 * 1024)) {
	return false;
	}
	// time based quota is in second
	if (retention.getRetentionTimeInMinutes() > 0
	&& quota.getLimitTime() >= retention.getRetentionTimeInMinutes() * 60) {
	return false;
	}
	return true;
	}

[lhotari]

It seems like a bug. -1 isn't properly handled in the backlog quota check.

[lhotari]

Well it might be ok.

[Shawyeok]

Maybe the key problem is why we need this compare check at all?

[Shawyeok]

I’m curious what’s the potential problem with this combination? Now it is rejected at the beginning.

• backlog quota: size(20G) time(30d)
• retention policy: size(2G) time(1w)

[Shawyeok]

I've start a discuss thread on mail list, feel free to take a look, thanks. @lhotari

https://lists.apache.org/thread/ojh23zs11tktgprx7z69ncxog8p8jtxh