Feature request: Scatter-gather support on Query APIs

[jeqo]

Rational
One idea that comes from the development of https://github.com/jeqo/zipkin-storage-kafka and hope it fits under this issue:

The storage layer is based on Kafka Streams local store, that is aligned with partitioning. Currently we have specified that our implementation supports running only a standalone instance for storage, because if we scale the Zipkin instances, storage will get partitioned between servers.

In order to cope with this scenario I'd like to propose a scatter-gather support that allows storage layer to query other instances to build a response.

Example Scenario

Given a partitioned back-end with 3 zipkin servers (a,b,c) running as a cluster, if we receive a query from client-side, zipkin-a receive the request, and forward the same query to zipkin-b and zipkin-c with an additional query param (e.g. peer=true) so b and c don't propagate the query. zipkin-a receives responses and build response.

Prior Art

Kafka Streams already supports a metadata API to register peers URLs. https://kafka.apache.org/documentation/streams/developer-guide/interactive-queries.html#adding-an-rpc-layer-to-your-application

[codefromthecrypt]

main question is why does this need to affect other storage implementation? ex what part of the query prevents you from coordinating underneath with a private api? usually we let patterns repeat prior to generalizing. so this is more a request for what is needed to facilitate this.. maybe we can brainstorm about how to get one working before affecting the main api.

[jeqo]

@adriancole agree :) and probably moving it to the kafka-storage repo would be a better place to discuss.

[jeqo]

Moved to openzipkin-contrib/zipkin-storage-kafka#36