Follow-up work:

• https://github.com/Curt-Park/mnist-fastapi-aio-triton

You can also see the previous work from:

• https://github.com/Curt-Park/producer-consumer-fastapi-celery
• https://github.com/Curt-Park/triton-inference-server-practice (00-quick-start)

Benchmark FastAPI + Celery with / without Triton

with Triton Server

without Triton Server

Benchmark Results

See Benchmark Results

Preparation

1. Setup packages

Install Anaconda and execute the following commands:

$ make env        # create a conda environment (need only once)
$ source init.sh  # activate the env
$ make setup      # setup packages (need only once)

2. Train a CNN model (Recommended on GPU)

$ source create_model.sh

3. Check the model repository created

$ tree model_repository

model_repository
└── mnist_cnn
    ├── 1
    │   └── model.pt
    └── config.pbtxt

2 directories, 2 files

How to play

Server (Option 1 - On your Local)

Install Redis & Docker, and run the following commands:

$ make triton     # run triton server
$ make broker     # run redis broker
$ make worker     # run celery worker
$ make api        # run fastapi server
$ make dashboard  # run dashboard that monitors celery

Server (Option 2 - Docker Compose available on GPU devices)

Install Docker & Docker Compose, and run the following command:

$ docker-compose up

[Optional] Additional Triton Servers

You can start up additional Triton servers on other devices.

$ make triton

[Optional] Additional Workers

You can start up additional workers on other devices.

$ export BROKER_URL=redis://redis-broker-ip:6379    # default is localhost
$ export BACKEND_URL=redis://redis-backend-ip:6379  # default is localhost
$ export TRITON_SERVER_URL=triton-server-ip:9000   # default is localhost
$ make worker

• NOTE: Worker needs to run on the machine which Triton runs on due to shared memory settings.

Dashboard for Celery (Flower)

http://0.0.0.0:5555/

Load Test (w/ Locust)

Execute Locust

$ make load  # for load test without Triton
or
$ make load-triton  # for  load test with Triton

Open http://0.0.0.0:8089

Type url for the API server.

Issue Handling

Redis Error 8 connecting localhost:6379. nodename nor servname provided, or not known.

$ ulimit -n 1024

Docker's network_mode=bridgedegrades the network performance.

We recommend to use Linux server if you would like to run docker-compose up.

For Developers

$ make setup-dev      # setup for developers
$ make format         # format scripts
$ make lint           # lints scripts
$ make utest          # runs unit tests
$ make cov            # opens unit test coverage information