gRPC Load Balancing between http-adapter and things

[nwest1]

BUG REPORT

1. What were you trying to achieve?
   Targeted TPS load testing against http-adapter and a failure scenario

2. What are the expected results?
   balanced, self-healing connections between mainflux components

3. What are the received results?
   gRPC not being load balanced, and when a pod is killed, these connections are not rebalanced

4. What are the steps to reproduce the issue?

• Run a load test in kubernetes with multiple replicas (http-adapter is using the things service name as url)
• Note that the majority of transactions are running through a single things pod and not load balanced
• Kill the things pod that's receiving the majority of transations
• Note the error rate spikes and connections from adapter to things remain in error and are not rebalanced.

5. In what environment did you encounter the issue?
   kubernetes

6. Additional information you deem important:
   This might be slightly out of scope, as it's possible to solve this with Istio/Linkerd or some k8s ingress that is aware of gRPC. But want to bring it up as it looks like you're exploring similar replacements for nginx.

[drasko]

@nwest1 thanks for filing this one.

[drasko]

@nwest1 we've analyzed this one, and surely there is a missing LB part for gRPC. Problem is that although k8 adds LB in front of the services, gRPC pass via HTTP/2 and is more like a stream and not balanced via L7 LB. For gRPC we need L4 LB, and most often used is Istio approach with Envoy

This is explained well in this video: https://www.youtube.com/watch?v=F2znfxn_5Hg

Additional info:

• https://grpc.io/blog/loadbalancing
• https://blog.bugsnag.com/envoy/

We already wanted to go this route, please consult issue https://github.com/mainflux/mainflux/issues/352.

Although there is a way to avoid using Envoy and LB gRPC on client side, we feel that Envoy approach will be better.

@janko-isidorovic already started integration of Istio/Envoy in our k8 scripts and we should have something working before end of week.

One more approach that can be taken is using NATS and it's Request-Replay mode with internal NATS LB in Queue mode + new feature of Drain for downscaling: https://medium.com/@derekcollison/nats-resilient-systems-and-drain-mode-a764d4968711. This could eventually make system simpler, but we are not sure would it make easier to debug (i.e. o figure out where the messages come from in the case of error), and we are not sure about perfs of NATS with all these modes enabled. Let's try first Isto/Envoy solution.

[nmarcetic]

#378 Closes this issue.

[drasko]

@nmarcetic it is Envoy/Istio that will close the issue (https://github.com/mainflux/mainflux/issues/352).

[nmarcetic]

@drasko Yes and its related to #378

[janko-isidorovic]

@nwest1 we have been able to reproduce the issue in Mainflux Lab. Adding Istio to Kubernets and Istio sidecar to the http-adapter pod resolves the issue and enables load balancing of the gRPC connections.
We need to decide if the solution is in the scope of Mainflux as it is more related to deployment strategy.

[drasko]

I agree that this depends only on deployment strategy (k8 in this case) and is not at all Mainflux issue. Let's see what is the best way to approach this, we should stay generic and not impose a specific deployment strategy and we need to understand how does this fall in Mainflux project scope.

[anovakovic01]

Resolved with #378.