Optimize computer vision performance on single core boards

[cedricve]

Kerberos.io works "fine" on a Raspberry Pi 2 and 3. When recording video we can write at a framerate of 7-10FPS at a resolution of 640x480. On a Raspberry Pi Zero (W) or Raspberry Pi 1, the framerate is at average 3FPS. The CPU is "acceptable", but we believe it can be reduced a lot. A possible solution would be to move some stuff to the GPU, and use hardware acceleration for video recording.

Functions to optimize

With this issue we want to tackle different parts of the machinery, that may be improved. Most of them are related to standard OpenCV functions.

1. (TODO) Differential images, calculating the changed pixels between frames.
   https://github.com/kerberos-io/machinery/blob/master/src/kerberos/machinery/algorithm/DifferentialCollins.cpp#L28-L40

   • Possible solution is to resize the images to a lower res, before applying the different math operations.

2. (TODO) Expositor, iterating over pixels to find the changed pixels within a ROI.
   https://github.com/kerberos-io/machinery/blob/master/src/kerberos/machinery/expositor/RectangleExpositor.cpp#L31-L44

3. (TODO) Encoding MJPEG streaming, for viewing a live stream from the web interface.
   https://github.com/kerberos-io/machinery/blob/master/src/kerberos/capture/Stream.cpp#L167~~

   • Possible solution is to use libturbo-jpeg instead of traditional jpeg. However this is powered by NEON, which isn't available on Raspberry Pi 1 or Zero (W).
   • Edit: We started focusing on the Raspberry Pi Camera, and integrated the OpenMax CPP library which comes with native hardware acceleration to encode MJPEG. A todo is to integrate this hw encoding also for IP cameras and USB cameras.
   • Achievement: When using a Raspberry Pi and Raspberry Pi camera: you can stream MJPEG at 10-20FPS on a Raspberry Pi 1.
   • Investigate how to do it for other cameras as well.

4. (Done) Video writing with FFMPEG (x264)
   https://github.com/kerberos-io/machinery/blob/master/src/kerberos/machinery/io/IoVideo.cpp#L300-L323

   • Recording with the h264_omx encoder on a Raspberry Pi 1 at 30FPS (640x480), with only consuming 30% of CPU power. Need to find a way to include this in OpenCV or if not possible directly in the machinery. Not an easy task atm.
   • For the Raspberry Pi Camera, we also use OpenMax to encode h264 videos at 30FPS. Similar to the MJPEG streaming, we should also replace (or create a new) IoVideo device (e.g. IoVideoOMX) which will call the ffmpeg (avcodec) directly and uses the h264_omx codec. At the moment of writing this is not available in the OpenCV repository by default. Support for CODEC_ID_H264_OMX opencv/opencv#8682
   • Achievement: When using a Raspberry Pi and Raspberry Pi camera: you can record video (1280x960) at 30FPS on a Raspberry Pi 1.
   • Todo: Recording at high speed on a USB camera will not be possible because we only have 1 videoport to connect to the camera (or we need to split the videoport). For IP cameras it is possible to open multiple streams (if the camera can handle it), and therefore we could open an additional record thread and encode it using OMX h264. This would improve the framerate enormously.
   • Started experimenting with modifying the OpenCV video writer class. Hardcoded the h264_omx codec. https://github.com/cedricve/opencv/blob/h264-preset/modules/videoio/src/cap_ffmpeg_impl.hpp#L1458-L1462. Encoding is successful on 800 x 600 and 1920x1080, however doesn't work on other resolutions (600x480 and 1280x720). Need to investigate why. @jasaw reported a similar issue in his PR for the motioneyeos project. ffmpeg 3.1 supports h264 hardware assist encoding motioneye-project/motioneyeos#365
   • Achievement: Successfully encoding with h264_omx codec, for USB and IP cameras, after upgrading to FFMPEG 3.4.3 and removing zero_copy with patch (Enable hwaccel h264 encoding on raspberry pi motioneye-project/motioneyeos#998). Thanks to @jasaw.

[muten84]

Hi any chance to get it work with IPCamera too? I'm interested in video recording optmize from IPCamera.

[cedricve]

it's possible for IP camera's also. The idea is instead of using OpenCV to write video files (which is slow and heavy operation), we'll fire up an FFMPEG command with omx hardware encoder. This will give same performance as for the Raspberry Pi camera (30FPS).

Problem that needs to be solved is compiling the omx libraries properly on Raspberry Pi (KiOS).
motioneye-project/motioneyeos#998

[muten84]

I'm going to check this reddit. However do you think Raspbian users of Kerberos.io can just use hardware accelerated video encoding??

[cedricve]

Yes, but you'll need to adjust the IoVideo class to use FFMPEG instead of OpenCV.

[muten84]

i will give it a try using a command line pipe from live stream and ffmpeg encoding .... if it works the next step could be to use the pipe for writing images and then start ffmpeg encoding that reads from the just created pipe. I didn't remember how to use pipes or shared memory in c++ but i think this is possible

[douggb]

If you need people to test, I have a Pi2, Pi3 and an OrangePi+2e. IP camera Hikvision 4mp. Trying to get 1080p at 15fps. Still learning Linux, so would need some guidance.

[cedricve]

Update on hardware encoding

Started experimenting with modifying the OpenCV video writer class. Hardcoded the h264_omx codec. https://github.com/cedricve/opencv/blob/h264-preset/modules/videoio/src/cap_ffmpeg_impl.hpp#L1458-L1462. Encoding is successful on 800 x 600 and 1920x1080, however doesn't work on other resolutions (600x480 and 1280x720). Need to investigate why. @jasaw reported a similar issue in his PR for the motioneyeos project. motioneye-project/motioneyeos#365

[cedricve]

Successfully encoding with h264_omx codec, for USB and IP cameras, after upgrading to FFMPEG 3.4.3 and removing zero_copy with patch (motioneye-project/motioneyeos#998). Thanks to @jasaw.