Question regarding ZkBookieRackAffinityMapping

[estebangarcia]

Hi. I couldn't find any information on the docs on how to tell pulsar to write on different bookies depending on which rack/region/az they are.

Searching the code I found the class ZkBookieRackAffinityMapping, that (correct me if I'm wrong), expects a json with the 'topology' on a znode called /bookie that will use to determine on which bookie write.

So we went ahead and created a little java process that checks the available bookies on /ledgers/available and writes that json, the output is similar to the one I've attached.

My main question here is, are we doing it right ? 😛
And how can we check if it's working?
Should we tell pulsar to use this class ?

Any help will be much appreciated. Thanks

bookie.txt

[merlimat]

Yes, this is another area where we need to add more documentation.

We use the BookKeeper rack-aware placement policy. The "rack" concept can be anything, (rack/region/availability zone). The base idea is: when forming the ensemble for a new ledger, pick bookies from different racks, to reduce the chances of data unavailability.

If bookies from different racks are not available, the policy falls back to chose randomly across available bookies.

The ZkBookieRackAffinityMapping is a way to feed the rack information into the rack-aware policy. The JSON z-node content you've attached looks correct. The first level is the group-name, and can be used to provide bookie isolation (strictly picking from a subset of bookies).

You only need to update the /bookies z-node when you add new bookies. There is no need to update it when processes are up or down, that is controlled by the list in /ledgers/available.

If the /bookies z-node is not there, the BK client will choose randomly. Same thing if all bookies are in the same "rack" (whatever that means).

So, in AWS, you'd probably want to create instances in different AZs and update the /bookies z-node to reflect that.

My main question here is, are we doing it right ? 😛

Yes, just make sure you have bookies from different "racks" in the configuration.

And how can we check if it's working?

You should be able to verify that all ledgers should have the ensemble containing bookies from different AZs :

bin/bookkeeper shell ledgermetadata -ledgerid $MY_LEDGER_ID

Should we tell pulsar to use this class ?

It's automatically used when bookkeeperClientRackawarePolicyEnabled=true is set on conf/broker.conf.

[estebangarcia]

@merlimat We're getting lots of this messages:

2017-01-30 19:44:40,500 - WARN - [pulsar-io-39-5:RackawareEnsemblePlacementPolicy@526] - Failed to choose a bookie from /default-rack : excluded [], fallback to choose bookie randomly from the cluster.

Is this normal? It looks like our topolopgy is not being used. Contents of our znode /bookie

{"us-east-1":{"10.64.103.105:3181":{"rack":"us-east-1d","hostname":"ip-10-64-103-105.ec2.internal"},"10.64.102.228:3181":{"rack":"us-east-1c","hostname":"ip-10-64-102-228.ec2.internal"},"10.64.102.146:3181":{"rack":"us-east-1c","hostname":"ip-10-64-102-146.ec2.internal"},"10.64.103.213:3181":{"rack":"us-east-1e","hostname":"ip-10-64-103-213.ec2.internal"},"10.64.102.121:3181":{"rack":"us-east-1a","hostname":"ip-10-64-102-121.ec2.internal"},"10.64.102.43:3181":{"rack":"us-east-1a","hostname":"ip-10-64-102-43.ec2.internal"}}}

[estebangarcia]

Looking at a random ledger metadata, we can see the ensemble members are all from different AZs, but we don't know if it's just a coincidence or if the affinity is really working and that WARN is expected.