When it comes to deploying ML models, data scientists have to make a choice based on their use case. If they need a high volume of predictions and latency is not an issue, they typically perform inference in batch, feeding the model with large amounts of data and writing the predictions into a table. If they need predictions at low latency, e.g. in response to a user action in an app, the best practice is to deploy ML models as REST endpoints. These apps allows to send requests to an endpoint that’s always up and receive the prediction immediately.
| Library Name | Description |
|---|---|
| Tensorflow Serving | High-performant framework to serve Tensorflow models via grpc protocol able to handle 100k requests per second per core |
| TorchServe | TorchServe is a flexible and easy to use tool for serving PyTorch models. |
| BentoML | BentoML is an open source framework for high performance ML model serving |
| Clipper | Model server project from Berkeley's Rise Rise Lab which includes a standard RESTful API and supports TensorFlow, Scikit-learn and Caffe models |
| Cortex | Cortex is an open source platform for deploying machine learning models—trained with nearly any framework—as production web services. |
| Multi-Model-server | Multi Model Server (MMS) is a flexible and easy to use tool for serving deep learning models trained using any ML/DL framework. |
| DeepDetect | Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain |
| Aml-Deploy | GitHub Action for deploying Machine Learning Models to Azure |
| MLOps | MLOps empowers data scientists and app developers to help bring ML models to production. |
| ForestFlow | Cloud-native machine learning model server. |
| Jina | Cloud native search framework that supports to use deep learning/state of the art AI models for search. |
| KFServing | Serverless framework to deploy and monitor machine learning models in Kubernetes - (Video) |
| NVIDIA TensorRT Inference Server | TensorRT Inference Server is an inference microservice that lets you serve deep learning models in production while maximizing GPU utilization. |
| OpenScoring | REST web service for scoring PMML models built and maintained by OpenScoring.io |
| Redis-AI | A Redis module for serving tensors and executing deep learning models. Expect changes in the API and internals. |
| Seldon Core | Open source platform for deploying and monitoring machine learning models in kubernetes - (Video) |
| model_server | OpenVINO™ Model Server is a scalable, high-performance solution for serving machine learning models optimized for Intel® architectures. The server provides an inference service via gRPC enpoint or REST API |