Should we convert kube-plex into a controller

[ressu]

Lately I've been thinking a lot about reducing transcoding startup latency. I have a few ideas on how to to make that happen, but it comes at the cost of increasing complexity.

The main problem is that since the transcoder is running within a container, the startup has quite a bit of overhead like pulling images, container setup and even controller delays within Kubernetes itself. So the main idea I'm playing around with is preloading the transcoder. This would mean that there would always be a transcoder process running. When Plex needs a transcoder, it simply connects to the pod to start the transcoding. This leads into issues with scaling etc.. I'll try and break this down to pros and cons

Pros:

• Near instant transcode start, just like with Plex itself
• Better configuration error handling, kube-plex could fall back to local transcoding if all transcoders fail
• More native solution
• No need to parse annotations for random pieces of transcoding info, which could be in the controller instead

Cons:

• Complexity increases significantly
• Preloading transcoders will reserve resources. Biggest issues with GPUs
• More dependent on Kubernetes versions than today

All in all this would move kube-plex closer to the way some other transcoding multiplexers work today, but at the cost of simplicity. Even today it's difficult for a newcomer to understand how kube-plex works and after these changes it would be even more complex to troubleshoot their deployments. I also worry that this would be a bit of an overkill solution.

[brandon099]

I really like the idea of instant transcoding, something more native, and all of the other pros you have listed. What would the complexity end up looking like?

As for the requests for GPU resources, there are options to allow sharing those devices on a cluster now, instead of reserving the entire one. But that comes with yet more complexity.

[ressu]

I see two ways to solve the problem, while both follow the same pattern. There is a quick and dirty solution, which I don't really like.

Dirty solution

The transcoder launcher is modified to start, but wait for a RPC from the main process. We would have N transcoders ready at any given time. This means that there is no delay between the transcode creation and the moment when the pod is created.
We also need a control server to act as a mediator between the Plex deployment and the actual transcoders. The transcoders would register themselves to the control server so that the requests just work. This is pretty much how UnicornTranscoder works.

What makes this dirty, is that we would need a separate deployment for the control server and the connections would need to be persistent for the RPC to work nicely. There would be quite a bit of hackery around how to configure the different parts, but this would still be less effort than the "do it right" solution

Do it right / Overkill solution

Since we are using Kubernetes, the right thing to do would be to make use of the API server. We would define a CRD in the cluster, so instead of deploying a Deployment resource with a bunch of annotations, we would have a well defined Plex resource directly. The controller would then create the main Pod for Plex (most likely as a StatefulSet instead of deployment, but that's irrelevant). The controller would also set up a number of transcoder Pods and ensure that there are always enough idle Pods running. We could either use the Services object to provide transcoders to the main Plex or have a separate relay (I think a Service should be enough unless we want to build advanced features to transcoder selection)

This solution is definitely overkill, but it would also make use of the best parts of Kubernetes and would in theory be able to scale to significant deployment sizes (Considering that my deployment has a single person watching at a time, this is definitely overkill for me alone 😆 )