How to set Aggregation replication?

[YanshuoH]

Hi,

As described in https://pulsar.apache.org/docs/next/concepts-replication/#aggregation-replication, there is an aggregation replication mode.

Yet in the doc of https://pulsar.apache.org/docs/next/administration-geo/, there is no instruction of setting the aggregation mode.

So my question is how to set the aggregation replication mode ?

Thank you.

[lhotari]

Summarizing the answer from Slack:

There is no special "aggregration replication mode". The documentation is not very clear and doesn't provide an example of such configuration.

The "aggregation replication" pattern referred in the docs is about configuring the replication in a hub-spoke way where instead of having point-to-point connections (full mesh) between each cluster, you have a hub cluster and you connect all other clusters to this hub.

I have been using geo-replication without a shared metadata store. At least in that case, replication is configured point-to-point for each one-way direction.
Yes, there are some gotchas with partitioned topics. I think one solution is to disable topic auto creation. I believe that the replicator will attempt to create the topic in the remote cluster when replication starts the first time. In addition, setting up authorization and TLS isn't very straightforward.

Side note:
For subscription replication with multiple clusters, it is usually necessary to increase the replicatedSubscriptionsSnapshotTimeoutSeconds configuration setting for example to value 60 (comment).

Other geo-replication discussions:
#22315
#21612

[nareshv]

@lhotari , how is this point done ?
I have been using geo-replication without a shared metadata store. At least in that case, replication is configured point-to-point for each one-way direction Will this help answer #22438 ?

[lhotari]

In the configuration, same commands are used to configure the point-to-point replication and hub-spoke replication.
I haven't been using a shared global metadata store either. I'm not sure if using a global metadata store configuration support hub-spoke replication at all.

The configuration of hub-spoke replication is very similar to point-to-point replication. You simply choose a hub cluster and configure replication between each spoke and the hub. I think having a public sample configuration would be necessary to validate the details of the configuration.

It's fairly simple to have a local k8s configuration with multiple clusters deployed with Apache Pulsar Helm chart and validate the configuration that way. However, I don't have an updated configuration available right now. I do have an outdated dual cluster example in a public repo. The scripts might be cryptic since I've made it just for myself (example of deploying 2 clusters in separate namespaces, configuring georep, including some workarounds).