Can the MLServer runtime be replaced by Triton?

[OvervCW]

ModelMesh currently uses the MLServer runtime to serve sklearn, xgboost, and lightgbm models. However, it seems like recent versions of Triton now support all of those model types as well.

Does that mean that ModelMesh could support all types of models using a single runtime? And if so, would it be easy to change the adapter to handle this?

[njhill]

@OvervCW there shouldn't be anything stopping you using these other kinds of models with Triton, would be great if you could try it out!

Hopefully all that should be needed is to:

• Update the triton-2.x ServingRuntime spec to include additional supportedModelFormats as needed
• Reference the appropriate type in your Predictor and either specify the runtime name explicitly or delete/remove other runtimes like MLServer which also support the same type (or set autoSelect to false in the right places in those other runtimes)
• Make sure to include a full Triton model configuration pbtxt file in the storage location that the Predictor's path points to (like this)

[ckadner]

@rafvasq -- since you updated the Triton serving runtime last year, it might be worth trying this out and documenting it. WDYT?

[rafvasq]

@ckadner I confirmed that with a few changes, Triton's able to deploy those models too. Think it'd be worth updating the docs to reflect that which includes a couple of small updates to the example models via kserve/modelmesh-minio-examples#7.