You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The storage layer is based on Kafka Streams local store, that is aligned with partitioning. Currently we have specified that our implementation supports running only a standalone instance for storage, because if we scale the Zipkin instances, storage will get partitioned between servers.
In order to cope with this scenario I'd like to propose a scatter-gather support that allows storage layer to query other instances to build a response.
Example Scenario
Given a partitioned back-end with 3 zipkin servers (a,b,c) running as a cluster, if we receive a query from client-side, zipkin-a receive the request, and forward the same query to zipkin-b and zipkin-c with an additional query param (e.g. peer=true) so b and c don't propagate the query. zipkin-a receives responses and build response.
main question is why does this need to affect other storage implementation?
ex what part of the query prevents you from coordinating underneath with a
private api?
usually we let patterns repeat prior to generalizing. so this is more a
request for what is needed to facilitate this.. maybe we can brainstorm
about how to get one working before affecting the main api.
Rational
One idea that comes from the development of https://github.com/jeqo/zipkin-storage-kafka and hope it fits under this issue:
The storage layer is based on Kafka Streams local store, that is aligned with partitioning. Currently we have specified that our implementation supports running only a standalone instance for storage, because if we scale the Zipkin instances, storage will get partitioned between servers.
In order to cope with this scenario I'd like to propose a scatter-gather support that allows storage layer to query other instances to build a response.
Example Scenario
Given a partitioned back-end with 3 zipkin servers (a,b,c) running as a cluster, if we receive a query from client-side, zipkin-a receive the request, and forward the same query to zipkin-b and zipkin-c with an additional query param (e.g. peer=true) so b and c don't propagate the query. zipkin-a receives responses and build response.
Prior Art
Kafka Streams already supports a metadata API to register peers URLs. https://kafka.apache.org/documentation/streams/developer-guide/interactive-queries.html#adding-an-rpc-layer-to-your-application
The text was updated successfully, but these errors were encountered: