Partition-mapping routing in the gateway to avoid selecting a wrong replica in a multi-replica scenario

[DawidNiezgodka]

When we have a single mirror, then the situation is clear. All partitions are located on this particular mirror, and when there is a request for the value of a given key, it is retrieved seamlessly.
The problem arises when we have more than one replica of a mirror:

Scenario: Mirror1 stores partitions 1 and 4, and Mirror2 stores partitions 2 and 3. Let's say that we want to get data for the key="x". Our hashing function h says that h("x") = 1 which means that the value for the key "x" is stored in the first partition, which is located in Mirror1.
Currently, a request from gateway goes to the (Kubernetes?) service that chooses a replica in a round-robin fashion. This means that it is statistically half of the time wrong. When this happens, the request has to be redirected to the other replica.

Because of the fact that a mirror knows which partitions it stores (it has a mapping between a partitions and a host), we can use this information to introduce routing based on the partition mapping.

Questions:

1. How do we expose this info to the gateway?
2. In which part of the gateway should we put this functionality?

Extra ideas:

1. Caching
2. Mirror could announce itself by the gateway (observer pattern?)