Skip to content
This repository has been archived by the owner on Jan 3, 2023. It is now read-only.

06 Cameras and Video

Paul Guermonprez edited this page Nov 9, 2017 · 43 revisions

Camera Streaming Daemon will stream the video feeds over the network with RTSP, a standard protocol. RTSP video can be received by QGroundControl, VLC and typical video players on mot platforms. It cover the RGB sensor of Intel RealSense, but also the bland and white downward facing camera, the depth and Infrared sensors.

  • If you're using the Yocto build provided by Intel (v1.6 or newer), the Camera Streaming Daemon is already installed and configured.
  • If you're using Ubuntu, you'll need to install it manually.

On Yocto (v1.6 minimum), Intel Aero is proposing 5 RTSP video feeds:

  • RealSense R200, HD camera: rtsp://192.168.8.1:8554/video13
  • RealSense R200, depth sensor: rtsp://192.168.8.1:8554/rsdepth
  • RealSense R200, infrared first camera: rtsp://192.168.8.1:8554/rsir
  • RealSense R200, infrared second camera: rtsp://192.168.8.1:8554/rsir2
  • Bottom facing black and white global shutter: rtsp://192.168.8.1:8554/bottom

On Ubuntu, Intel Aero is proposing 3 RTSP video feeds:

  • RealSense R200, HD camera: rtsp://AERO_IP_ON_YOUR_NETWORK:8554/video2
  • RealSense R200, infrared first camera: rtsp://AERO_IP_ON_YOUR_NETWORK:8554/video1
  • RealSense R200, infrared second camera: rtsp://AERO_IP_ON_YOUR_NETWORK:8554/video0

Procedure:

  • Following the calibration procedure, you should have QGroundControl already installed and connected to Intel Aero.

  • Make sure you have disabled any sort of firewall/VPN on your Host PC.

  • Launch QGroundControl on Host PC. It will open into the flight path mode. You need to click on the QGC Icon to see the configuration and scroll down:

  • Video settings: RTSP

  • URL on Yocto: rtsp://192.168.8.1:8554/video13

  • URL on Ubuntu: rtsp://AERO_IP_ON_YOUR_NETWORK:8554/video2

  • Go back to the flight view. The bottom left view should show the video.

Note: It may be possible that video stream is blocked by firewall/VPN. In that case, you may want to disable the firewall/VPN on the host PC

You can access the RTSP feeds from QGroundControl or any compatible video player. If you have a linux PC, try gstreamer with the latency=0 setting to receive video from CSD running on Intel Aero:

gst-launch-1.0 rtspsrc location=rtsp://192.168.8.1:8554/bottom latency=0 ! decodebin ! autovideosink

Note: It may be possible that video stream is blocked by firewall/VPN. In that case, you may want to disable the firewall/VPN on the host PC

Using Yocto, open the network video rtsp://192.168.8.1:8554/video13.

If you're using Ubuntu, the IP will depend on your network settings and the video device will be video2 rtsp://AERO_IP_ON_YOUR_NETWORK:8554/video2 VLC RTSP

Note: It may be possible that video stream is blocked by firewall/VPN. In that case, you may want to disable the firewall/VPN on the host PC

We covered the Camera Streaming Daemon to stream RTSP feeds. But you can also use gstreamer directly to encode (using the GPU) and send video over the network. gstreamer is not as easy to use as CSD but very flexible. Here an example using Ubuntu, launching a gstreamer command to get video from the RGB sensor of the R200 camera (/dev/video2), select a VGA resolution at 15fps, use the hardware accelerated h264 encoder and send it to my laptop's IP (my laptop is 192.168.1.147 and I have a RTSP software ready to receive the video on this machine).

sudo gst-launch-1.0 v4l2src  device=/dev/video2 do-timestamp=true ! video/x-raw, format=YUY2, width=640, height=480, framerate=15/1 ! autovideoconvert ! vaapih264enc ! rtph264pay !  udpsink host=192.168.1.147 port=5600

It is an advanced topic posted for reference, don't worry if it looks too complex you won't need it.

Let's focus on the Intel RealSense R200 camera included in the Intel Aero Ready-To-Fly Drone. A single R200 camera has 3 feeds. The first two are for depth, the third for RGB. But other than that, it's a standard video device on linux.

  • If you're working with Ubuntu 16.04.3, as installed on this page. You should see the camera as a video device when you run ls /dev/video*:
/dev/video0  /dev/video1  /dev/video2
  • If you using Yocto, the R200 feeds are /dev/video[11-13].

gstreamer is a great tool to work with video devices:

sudo gst-launch-1.0 v4l2src device=/dev/video2 num-buffers=1 ! jpegenc ! filesink location="rs.jpg"
identify rs.jpg 

should see a photo file created with the following FullHD specs:

rs.jpg JPEG 1920x1080 1920x1080+0+0 8-bit sRGB 50.4KB 0.000u 0:00.000

This example is for Ubuntu, as we used /dev/video2 for the device.

let's use vaapih264enc to take advantage of the hardware accelerated video encoding (Ctrl-C to stop):

sudo gst-launch-1.0 -e v4l2src device=/dev/video2 ! autovideoconvert format=i420 width=1920 height=1080 framerate=30/1 ! vaapih264enc rate-control=cbr tune=high-compression ! qtmux ! filesink location=encoded_video.mp4
ffmpeg -i encoded_video.mp4

you should see as output the specs of the FullHD video encoded:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'encoded_video.mp4':
  Metadata:
    major_brand     : qt  
    minor_version   : 537199360
    compatible_brands: qt  
    creation_time   : 2017-11-09 02:01:08
  Duration: 00:00:05.70, start: 0.000000, bitrate: 11712 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 11705 kb/s, 30 fps, 30 tbr, 3k tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2017-11-09 02:01:08
      handler_name    : VideoHandler

This example is for Ubuntu, as we used /dev/video2 for the device.

and limit the recording to 2000 frames:

sudo gst-launch-1.0 v4l2src device=/dev/video2 num-buffers=2000 ! autovideoconvert format=i420 width=1920 height=1080 framerate=30/1 ! vaapih264enc rate-control=cbr tune=high-compression ! qtmux ! filesink location=encoded_video.mp4

This example is for Ubuntu, as we used /dev/video2 for the device.

On Ubuntu, we'll use the standard V4L2 tools: sudo v4l2-ctl --list-devices :

Intel RealSense 3D Camera R200 (usb-0000:00:14.0-4):
	/dev/video0
	/dev/video1
	/dev/video2

List possible controls with sudo v4l2-ctl --list-ctrls -d /dev/video2:

                     brightness (int)    : min=0 max=255 step=1 default=56 value=56
                       contrast (int)    : min=16 max=64 step=1 default=32 value=32
                     saturation (int)    : min=0 max=255 step=1 default=128 value=128
                            hue (int)    : min=-2200 max=2200 step=1 default=0 value=0
 white_balance_temperature_auto (bool)   : default=1 value=1
                          gamma (int)    : min=100 max=280 step=1 default=220 value=220
                           gain (int)    : min=0 max=256 step=1 default=32 value=32
           power_line_frequency (menu)   : min=0 max=2 default=0 value=0
      white_balance_temperature (int)    : min=2000 max=8000 step=1 default=6500 value=6500 flags=inactive
                      sharpness (int)    : min=0 max=7 step=1 default=0 value=0
         backlight_compensation (int)    : min=0 max=4 step=1 default=1 value=1
                  exposure_auto (menu)   : min=0 max=3 default=3 value=3
              exposure_absolute (int)    : min=1 max=666 step=1 default=1 value=1 flags=inactive

And change a setting, as a test:

sudo v4l2-ctl -d /dev/video2 -c brightness=56

If you are connected to Intel Aero graphically, with a HDMI cable, you can run a graphical tool. The simplest tool is gstreamer:

sudo gst-launch-1.0 v4l2src device=/dev/video2 ! xvimagesink

You can play around with the settings, like changing to VGA, 30fps:

sudo gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=640,height=480,framerate=30/1  ! xvimagesink

For a more user friendly tool, install guvcview with sudo apt install guvcview on Ubuntu. Remember, there's 3 feeds for the R200 Camera. Select the third one for the color output.

To plug another camera, you have to consider the type of IO and the drivers.

  • We have lots of possible ports, including. SPI, CAN, USB2 or USB3.
  • Intel Aero is linux based (yocto as factory install or Ubuntu 16.04.3 as manual installation), you should target cameras with drivers for this selection of linux distributions. Typically it means UVC, USB Video Class. Note: most action cameras do NOT have linux drivers publicly available.

Here's an example with e-con systems See3Cam USB3 camera. It is not an Intel recommended/supported hardware. We just provide the documentation as a typical example of linux camera integration.

Plug the camera on the USB3 with the USB3-OTG cable (sold by Intel as an accessory).

Note: we're working with Ubuntu 16.04.3 in this example.

Run lsusb:

Bus 002 Device 002: ID 8086:0a80 Intel Corp. 
Bus 002 Device 003: ID 2560:c1d0  
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

There's a new line (2nd line) corresponding to the camera, showing it's correctly connected as a USB device.

Run dmesg:

...
[ 2580.288919] usb 2-1: new SuperSpeed USB device number 3 using xhci_hcd
[ 2580.302449] usb 2-1: LPM exit latency is zeroed, disabling LPM.
[ 2580.305441] uvcvideo: Found UVC 1.00 device See3CAM_CU130 (2560:c1d0)
[ 2580.307009] uvcvideo 2-1:1.0: Entity type for entity Extension 3 was not initialized!
[ 2580.307023] uvcvideo 2-1:1.0: Entity type for entity Processing 2 was not initialized!
[ 2580.307032] uvcvideo 2-1:1.0: Entity type for entity Camera 1 was not initialized!
[ 2580.307580] input: See3CAM_CU130 as /devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0/input/input5
[ 2580.310773] hid-generic 0003:2560:C1D0.0001: hiddev0: USB HID v1.11 Device [e-con systems See3CAM_CU130] on usb-0000:00:14.0-1/input2

The camera is recognized as a UVC device, as expected (and as a HID device, too).

Run ls /dev/video* to list the video devices:

/dev/video0  /dev/video1  /dev/video2  /dev/video3

Before connecting the camera, we could see video0,1,2 corresponding to the Intel RealSense R200 camera (depth and RGB sensors). We now have a new video device. Depending on how the device is found by linux, it may be device0 or device2. Use sudo v4l2-ctl --list-devices to get the details.

It means this camera can be used as a video device, not only to take pictures.

To capture a frame from the /dev/video3 device, you could use a variety of methods, but gstreamer is incredibly versatile:

gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=1 ! jpegenc ! filesink location="test.jpg"

A 13Mpixels, 2.4Mo jpg file is created, here are the identify test.jpg results:

test.jpg JPEG 4224x3156 4224x3156+0+0 8-bit sRGB 2.398MB 0.000u 0:00.000

The quality of the photo may not be optimal, as we have not played with the camera settings yet.

As with Intel RealSense R200, you can use guvcview and v4l2-ctl. v4l2-ctl --list-devices:

See3CAM_CU130 (usb-0000:00:14.0-1):
	/dev/video3
Intel RealSense 3D Camera R200 (usb-0000:00:14.0-4):
	/dev/video0
	/dev/video1
	/dev/video2

List possible controls with v4l2-ctl --list-ctrls -d /dev/video3:

                     brightness (int)    : min=-15 max=15 step=1 default=0 value=0
                       contrast (int)    : min=0 max=30 step=1 default=10 value=10
                     saturation (int)    : min=0 max=60 step=1 default=16 value=16
 white_balance_temperature_auto (bool)   : default=1 value=1
                          gamma (int)    : min=40 max=500 step=1 default=220 value=220
                           gain (int)    : min=0 max=100 step=1 default=0 value=0
      white_balance_temperature (int)    : min=1000 max=10000 step=50 default=5000 value=5000 flags=inactive
                      sharpness (int)    : min=0 max=127 step=1 default=16 value=16
                  exposure_auto (menu)   : min=0 max=3 default=1 value=0
              exposure_absolute (int)    : min=1 max=10000 step=1 default=156 value=312 flags=inactive
                   pan_absolute (int)    : min=-648000 max=648000 step=3600 default=0 value=0
                  tilt_absolute (int)    : min=-648000 max=648000 step=3600 default=0 value=0
                  zoom_absolute (int)    : min=100 max=800 step=1 default=100 value=100

And change one, as a test:

v4l2-ctl -d /dev/video2 -c brightness=56