RGB Alignment with Depth

[Luxonis-Brandon]

Start with the why:

In cases where both color is required for neural inference to work properly -and- the neural data needs to be perfectly aligned with the depth data, alignment between the RGB and depth information needs to exist.

A canonical example is doing semantic segmentation of color defects and needing to know their physical location. In this case, both color-based neural inference is needed (and it is per-pixel, since the network is a semantic segmentor) and depth information is needed to be aligned per-pixel.

Move to the how:

The Myriad X already has the capability to perform the transform to provide alignment between cameras. What is needed is a system for doing the calibration to determine this transform matrix, including at the differing resolutions of the color camera and the grayscale cameras (which are the source of the depth map).

Move to the what:

Provide a system for DepthAI users to calibrate the RGB camera against the right grayscale camera and the mechanism to do this alignment.

And for units with onboard cameras, improve our calibration system to perform this RGB-right calibration in the factory for all future production.

[Luxonis-Brandon]

As an update, we have the calibration stage working now. And future DepthAI builds (after this writing) are actually having RGB-right calibration performed. An example with semantic segmentation is shown below:

The right grayscale camera is shown on the right and the RGB is shown on the left. You can see the cameras are slightly different aspect ratios and fields of view, but the semantic segmentation is still properly applied.

[Luxonis-Brandon]

We now have an initial prototype of this running on-device. See below for the remapping of depth between the view of the right, RGB, and left cameras:

View from the right:

View from the RGB (center):

View from the left:

So in each of these cases, the depth is mapped (or re-mapped) to be centered to the respective camera. So then the remaining step for RGB is the WARP of the depth to fit the intrinsics/etc. of the RGB camera module.

This is working, but we have some crash that is preventing this from being mainlined. We've been debugging the crash.

[Luxonis-Brandon]

First version is here: https://github.com/luxonis/depthai-shared/pull/23/files/a6eac9d87eda4e1f7b08e7dc9c815039c93abc2e..c2d7a82ad91647ff0a8b0c0530b6395c54e5ae79

[Luxonis-Brandon]

This was just initially released in https://github.com/luxonis/depthai-python and https://github.com/luxonis/depthai-core (2.4+)

[Luxonis-Brandon]

Code sample here: https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/#rgb-depth

[TheBricktop]

Sorry to inform You but the link is dead :(

[Luxonis-Brandon]

Thanks @TheBricktop the link above is now fixed.

[TheBricktop]

Thank You ☺️