TENSORRT CPP FOR ONNX

Tensorrt codebase in c++ to inference for all major neural arch using onnx and dynamic batching

NVIDIA Driver

wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_550.54.14_linux.run

sudo sh cuda_12.4.0_550.54.14_linux.run

Docker

sudo docker build -t trt_24.02_opencv  .

sudo docker run --rm --network="host" -v $(pwd):/app -it --runtime nvidia trt_24.02_opencv bash

Models

YOLO-World

Model Conversion

url = https://github.com/AILab-CVC/YOLO-World

Clone the YOLO-World

git clone https://github.com/AILab-CVC/YOLO-World

Follow steps for installation
https://github.com/AILab-CVC/YOLO-World?tab=readme-ov-file#1-installation

Define custom classes(Any name can be defined)
echo '[["helmet"], ["head"],["sunglasses"]]' > custom_class.json

PYTHONPATH=./ python3 deploy/export_onnx.py configs/pretrain/yolo_world_v2_s_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_lvis_minival.py yolo_world_v2_s_obj365v1_goldg_pretrain-55b943ea.pth --custom-text custom_class.json --opset 12 --without-nms

After the fix https://github.com/AILab-CVC/YOLO-World/pull/416
python3 deploy/onnx_demo.py ./work_dirs/yolo_world_v2_s_obj365v1_goldg_pretrain-55b943ea.onnx ~/disk1/uncanny/projects/tensorrt-cpp-for-onnx/data/ custom_class.json  --onnx-nms

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

Adjust settings in the ./examples/yolo-world/main.cpp 
0.019/*score_threshold*/, 0.7/*iou_threshold*/, 300/*max_detections*/

// Move <model_version>.onnx file to 'examples/yolo-world'
cp cp ./work_dirs/<model_version>.onnx /app/examples/yolo-world

mkdir build
cd build
cmake ..
make -j4

./yolo-world /app/examples/yolo-world/<model_version>.onnx /app/data/

// Check the results folder

Results

Results [yolo_world_v2_s_obj365v1_goldg_pretrain, Batchsize = 1, Model size = 640x640]

RT-DETR

Model Conversion

url = https://github.com/lyuwenyu/RT-DETR.git

Clone the RT-DETR

git clone https://github.com/lyuwenyu/RT-DETR.git

Follow steps from here 

Version1
https://github.com/lyuwenyu/RT-DETR/blob/main/rtdetr_pytorch/README.md#todo

Version2
https://github.com/lyuwenyu/RT-DETR/tree/main/rtdetrv2_pytorch#quick-start

Any of above can run.

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/rt-detr'
cp <model_version>.onnx /app/examples/rt-detr

mkdir build
cd build
cmake ..
make -j4

./rt-detr /app/examples/rt-detr/<model_version>.onnx /app/data/

// Check the results folder

Results

Results [RT-DETRv2-S, Batchsize = 2, Model size = 640x640]

YOLOV10

Model Conversion

url = https://github.com/THU-MIG/yolov10

Clone the yolov10

git clone https://github.com/THU-MIG/yolov10

yolo export model=yolov10n/s/m/b/l/x.pt format=onnx opset=13 simplify dynamic

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/yolov10'
cp <model_version>.onnx /app/examples/yolov10

mkdir build
cd build
cmake ..
make -j4

./yolov10 /app/examples/yolov10/<model_version>.onnx /app/data/

// Check the results folder

Results

Results [YOLOv10m, Batchsize = 2, Model size = 640x640]

YOLOV9

Model Conversion

url = https://github.com/WongKinYiu/yolov9.git

commit 380284cb66817e9ffa30a80cad4c1b110897b2fb

Clone the yolov9

git clone https://github.com/WongKinYiu/yolov9

python3 export.py --weights <model_version>.pt --include onnx_end2end

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>-end2end.onnx file to 'examples/yolov9'
cp <model_version>-end2end.onnx /app/examples/yolov9

mkdir build
cd build
cmake ..
make -j4

./yolov9 /app/examples/yolov9/<model_version>-end2end.onnx /app/data/

// Check the results folder

Results

Results [YOLOv9-C, Batchsize = 2, Model size = 640x640]

YOLOV8-Detect

Model Conversion

url = https://github.com/ultralytics/ultralytics

ultralytics==8.1.24

Install ultralytics package in python

from ultralytics import YOLO

model = YOLO('yolov8s.pt')
model.export(format='onnx', dynamic=True)

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/yolov8'
cp <model_version>.onnx /app/examples/yolov8

mkdir build
cd build
cmake ..
make -j4

./yolov8-detect /app/examples/yolov8/<model_version>.onnx /app/data/

// Check the results folder

Results

Results [YOLOv8s, Batchsize = 2, Model size = 640x640]

YOLOV8-Segment

Model Conversion

url = https://github.com/ultralytics/ultralytics

ultralytics==8.1.24

Install ultralytics package in python

from ultralytics import YOLO

# Load a model
model = YOLO('yolov8n-seg.pt')

# Export the model
model.export(format='onnx', dynamic=True)

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/yolov8'
cp <model_version>.onnx /app/examples/yolov8

mkdir build
cd build
cmake ..
make -j4

./yolov8-segment /app/examples/yolov8/<model_version>.onnx /app/data/

// Check the results folder

Results

Results [YOLOv8n, Batchsize = 2, Model size = 640x640]

YOLOV8-Pose

Model Conversion

url = https://github.com/ultralytics/ultralytics

ultralytics==8.1.24

Install ultralytics package in python

from ultralytics import YOLO

# Load a model
model = YOLO('yolov8n-pose.pt')

# Export the model
model.export(format='onnx', dynamic=True)

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/yolov8'
cp <model_version>.onnx /app/examples/yolov8

mkdir build
cd build
cmake ..
make -j4

./yolov8-pose /app/examples/yolov8/<model_version>.onnx /app/data/

// Check the results folder

Results

Results [YOLOv8n, Batchsize = 2, Model size = 640x640]

YOLOV8-OBB

Model Conversion

url = https://github.com/ultralytics/ultralytics

ultralytics==8.1.24

Install ultralytics package in python

from ultralytics import YOLO

# Load a model
model = YOLO('yolov8n-obb.pt')

# Export the model
model.export(format='onnx', dynamic=True)

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/yolov8'
cp <model_version>.onnx /app/examples/yolov8

mkdir build
cd build
cmake ..
make -j4

./yolov8-obb /app/examples/yolov8/<model_version>.onnx /app/data/obb/

// Check the results folder

Results

Results [YOLOv8n, Batchsize = 2, Model size = 640x640]

YOLOV8-Classify

Model Conversion

url = https://github.com/ultralytics/ultralytics

ultralytics==8.1.24

Install ultralytics package in python

from ultralytics import YOLO

# Load a model
model = YOLO('yolov8n-cls.pt')

# Export the model
model.export(format='onnx', dynamic=True)

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/yolov8'
cp <model_version>.onnx /app/examples/yolov8

mkdir build
cd build
cmake ..
make -j4

./yolov8-classify /app/examples/yolov8/<model_version>.onnx /app/data/classify/

// Check the results folder

Results

Results [YOLOv8n, Batchsize = 2, Model size = 224x224]

YOLOV5-Face

Model Conversion

url = https://github.com/deepcam-cn/yolov5-face

Install onnx==1.16.2, tqdm, thop, seaborn, torch==1.9, torchvision==0.10
Changes for dynamic shape

diff --git a/export.py b/export.py
index 1aa7dae..2502e23 100644
--- a/export.py
+++ b/export.py
@@ -75,14 +75,22 @@ if __name__ == '__main__':
     print('\nStarting ONNX export with onnx %s...' % onnx.__version__)
     f = opt.weights.replace('.pt', '.onnx')  # filename
     model.fuse()  # only for ONNX
-    input_names=['input']
-    output_names=['output']
-    torch.onnx.export(model, img, f, verbose=False, opset_version=12, 
+    
+    # Dynamic batching support
+    input_names = [ "input" ]
+    output_names = [ "output" ]
+    dynamic_axes={'input' : {0 : 'batch_size'}, 'output' : {0 : 'batch_size'}}
+    dynamic_axes['input'][2] = 'height'
+    dynamic_axes['input'][3] = 'width'
+    
+    torch.onnx.export(model, img, f, verbose=True, opset_version=12, 
         input_names=input_names,
         output_names=output_names,
+        # dynamic_axes = dynamic_axes
         dynamic_axes = {'input': {0: 'batch'},
                         'output': {0: 'batch'}
                         } if opt.dynamic else None)
+    # )
 
     # Checks
     onnx_model = onnx.load(f)  # load onnx model

Convert pytorch to onnx

python3 export.py --weights ./yolov5s-face.pt --dynamic

git clone https://github.com/PrinceP/tensorrt-cpp-for-onnx

// Move <model_version>.onnx file to 'examples/yolov5-face'
cp <model_version>.onnx /app/examples/yolov5-face

mkdir build
cd build
cmake ..
make -j4

./yolov5-face /app/examples/yolov5-face/<model_version>.onnx /app/data/yolov5-face/

// Check the results folder

Results

Results [YOLOv5s-face, Batchsize = 2, Model size = 640x640]

NOTES

Issues

Dynamic batching is supported. The batchsize and image sizes can be updated in the codebase.
Dynamic batch issue resolved for yolov10: THU-MIG/yolov10#27
Dynamic batch not present for Yolo-World
If size issue happens while building. Increase the workspaceSize

    Internal error: plugin node /end2end/EfficientNMS_TRT requires XXX bytes of scratch space, but only XXX is available. Try increasing the workspace size with IBuilderConfig::setMemoryPoolLimit().

    config->setMaxWorkspaceSize(1U << 26) 
    //The current memory is 2^26 bytes

Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
data		data
examples		examples
include		include
kernels		kernels
results		results
src		src
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

TENSORRT CPP FOR ONNX

NVIDIA Driver

Docker

Models

YOLO-World

RT-DETR

YOLOV10

YOLOV9

YOLOV8-Detect

YOLOV8-Segment

YOLOV8-Pose

YOLOV8-OBB

YOLOV8-Classify

YOLOV5-Face

NOTES

About

Languages

License

PrinceP/tensorrt-cpp-for-onnx

Folders and files

Latest commit

History

Repository files navigation

TENSORRT CPP FOR ONNX

NVIDIA Driver

Docker

Models

YOLO-World

RT-DETR

YOLOV10

YOLOV9

YOLOV8-Detect

YOLOV8-Segment

YOLOV8-Pose

YOLOV8-OBB

YOLOV8-Classify

YOLOV5-Face

NOTES

About

Topics

Resources

License

Stars

Watchers

Forks

Languages