The ZED SDK can be interfaced with Tensorflow for adding 3D localization of custom objects detected with Tensorflow Object Detection API. In this Python 3 sample, we will show you how to detect, classify and locate objects in 3D space using the ZED stereo camera and Tensorflow SSD MobileNet inference model.
The 3D Object Detection project depends on the following libraries:
- Python 3
- CUDA
- ZED SDK 3
- ZED Python API
- cuDNN
- Tensorflow 1
- Tensorflow Object Detection API
- OpenCV
Install the ZED SDK and the ZED Python API.
Install cuDNN. Read the support matrix for the corresponding CUDA and driver version.
Install Tensorflow 1 with GPU support by reading the following instructions for your target platform.
# GPU package for CUDA-enabled GPU cards
python -m pip install tensorflow==1.15
Install Tensorflow Object Detection API by following these instructions and download the model repository.
git clone https://github.com/tensorflow/models
Test that you have correctly installed the Tensorflow Object Detection API by running the following command:
python object_detection/builders/model_builder_test.py
Note: If you get an import error, make sure that tensorflow/models/research/slim directories have been added to PYTHONPATH. This can be done by running the following command:
# From tensorflow/models/
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
Make sure the virtualenv is active.
source ~/tensorflow/bin/activate
Run the code with python3.
python3 object_detection_zed.py
In this example, we're using the computationally efficient MobileNet model for detecting objects. You can change this by updating the MODEL_NAME variable and selecting another one from Tensorflow model zoo. These models will be downloaded and extracted automatically. For example, a ResNet model can used by changing MODEL_NAME
to :
# Full model name required
MODEL_NAME = ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03
Other custom object detection models can be loaded by modifying the PATH_TO_FROZEN_GRAPH, variable typically called frozen_inference_graph.pb
.
A DockerFile is provided in the docker folder.
Please refer to the Tensorflow compatibility table to know the appropriate cuDNN and CUDA versions for a given Tensorflow version. At the moment of writing, Tensorflow requires CUDA 9.0 and cuDNN 7.
This sample uses 2 threads, one for the ZED images capture and one for the Tensorflow detection. While it may seem complex at first, it actually solves 2 issues:
-
Performance is increased, as depth computation is done in parallel to inference.
-
Tensorflow and the ZED SDK uses CUDA GPU computation and therefore requires the use of CUDA contexts. Since we currently can't share the CUDA Context between the ZED and TF, we have to separate the GPU computation. Each CUDA context must therefore have its own thread.
If you need assistance go to our Community site at https://community.stereolabs.com/