Store metadata in ZK with binary protobuf format

[merlimat]

In Pulsar we are storing a lot of metadata in ZooKeeper using different formats:

• BookKeeper ledgers: Protobuf Text
• Managed Ledgers and cursors: Protobuf Text
• Broker and namespace bundles load reports: JSON

Using text formats has been good for quick debugging sessions without special tools but has drawbacks:

• Size of data stored in ZK can be significant when many topics (>1M) are active in a cluster. Protobuf text format is like json and needs to repeat all the field names each time.
• Speed of serializing/deserializing (binary formats are always faster to parse)
• Garbage generated (with binary format we could switch to the custom protobuf code generator to generate reusable objects)
• Backward compatibility. Text protobuf is not backward compatible (unlike the binary parser), it will fail to parse unknown fields (and there's no way to change that). This makes very difficult to change the format (typically we would do 1 release that can understand the new format but still writes the old one, then next release to write new format). Backward compatibility is key to ensure we can rollback a release if some issue is detected during deployment.

Of the 3 categories listed above, I don't think we should bother about load reports, because they're not where the bulk of metadata is.

My proposal would be:

• 1.17 release:

  1. Add the code to read both formats
  2. A config switch to enable writing binary format for ML and cursors data in ZK, with default to text format.
  3. Add tools to dump the content of a ML for human consumption

• 1.18 release:

  1. Make binary default
  2. Remove config switch for text/binary

Once the change has been implemented it would be easy to pre-verify the size difference and eventually think of storing even BK ledgers in binary format.

cc: @saandrews @rdhabalia @msb-at-yahoo @sschepens

[msb-at-yahoo]

👍 We'll need a CLI that knows how to decode into protobuf text so that we can still examine the system: maybe a wrapper around zoosh.

How do you propose to rewrite the data in ZK into the new format? We obviously won't want to do it too quickly.

[merlimat]

For the re-writing I was thinking to just do that each of them individually, at the first write occourence.

For z-nodes that are not rewritten, we can leave the code that fallback to deserislize the text format indefinitely.

[merlimat]

Added PR with first part of needed changes. About the CLI tool, I'm leaning towards having a REST API that read from ZK and returns the json. The topic won't need to be loaded, just any broker could answer that request.