Pre/Post Processing Steps for Further Disparity improvements

[mohanen]

The Disparity from libSGM, when projected in a point cloud the output contains noise like a pyramid in front of the camera covering the entire scene or the edges of the of the object seem to replicate multiple times in various disparity levels.

I have some knowledge on how sgm works and for post-processing I have been looking into Nerian's post-processing Steps and Hirshmullers Disparity Refinement Steps to improve the disparity map.

I was able to remove the noise (mentioned above) using a speckle filter which retained the correct data but at the cost of making it sparse almost losing more than 50% of the data in some scenes.

And looking at the other post-processing steps like Uniqueness check, Consistency check, Filtering of untextured image areas, Noise reduction seems to make the disparity even sparser and applying an optimal hole/gap filling technique wouldn't be enough I think.

I know SGM can do it better it even won 11th place in the ROB 2018.

So do you guys have any idea on

1. How can I improve the disparity accuracy without making it sparse, what are the Pre/Post processing steps can be implemented to further improve it?
2. How to include sub-pixel accuracy, any plans on including it?

Thanks and Regards,
Mohanen B

[atakagi-fixstars]

Hi, Mohanen

Thank you for your contribution.
I answer your questions.

1. Pre/Post processing

We have heard about Disparity map post-filtering which we can use from OpenCV.
https://docs.opencv.org/3.3.0/d3/d14/tutorial_ximgproc_disparity_filtering.html

We are going to apply this method to libSGM and evaluate it's ability.

2. Including sub-pixel accuracy

We are willing to implement sub-pixel interpolation, maybe it will lead to some performance loss however.

Regards,

[mohanen]

Hi, @atakagi-fixstars,

In your latest code, the pyramid-like noise mentioned above is almost gone but still some noise exist, I was using an older commit before.

I already tried the WLS filter.

• doesn't do well on the edges,
• doesn't fill the occlusion hole and
• again produces the pyramid-like noise.

I was using WLS with Confidence off and lambda and sigma values as preferred by OpenCV documentation.

Note: This same issue occurred for OpenCV SGBM (confidence on) too.

libSGM output

WLS applied

[atakagi-fixstars]

Hi, @mohanen

Thank you for your work!
The WLS doesn't perform well... so we must find another way.

[0Jiahao]

Hi @mohanen, I am having the same pyramid result as the one in your first comment. Could you please share the method for removing it?

Kind regards,
Jiahao

[mohanen]

If you are upscaling the disparity map means use nearest neighbor interpolation.

[0Jiahao]

@mohanen Thank you for your reply! Could you please give me an example or a link of which code I could refer to?

[lingjiankong]

For those who are wondering how to use OpenCV DisparityWLSFilter:

The trick is that you would need the negate the right disparity when passing it to DisparityWLSFilter::filter.

Below is an example:

// To calculate right disparaity, you need to flip the images and then flip the disparity back.
cv::Mat rightDisparity(right.size(), CV_16S);
cv::Mat flippedRightDisparity(right.size(), CV_16S);

cv::Mat flippedLeft(left.size(), CV_8U);
cv::Mat flippedRight(right.size(), CV_8U);

cv::flip(left, flippedLeft, 1);
cv::flip(right, flippedRight, 1);

sgm::StereoSGM ssgm_right(left.cols, left.rows, disp_size, input_depth, output_depth, sgm::EXECUTE_INOUT_HOST2HOST, param);
ssgm_right.execute(flippedRight.data, flippedLeft.data, flippedRightDisparity.data);

flip(flippedRightDisparity, rightDisparity, 1);

// OpenCV's DisparityWLSFilter implicitly assumes that the disparity is 16SC1 format
// and has been scaled up by 16 (yeah this part is very confusing).
// Therefore, for right disparity, you need to multiply by 16, and negate it (hence -16).

rightDisparity *= -16;

...

// Now you can call OpenCV's DisparityWLSFilter to give you a decent filtered disparity:
Ptr<DisparityWLSFilter> wlfFilter = createDisparityWLSFilterGeneric(true);
cv::Mat filteredDisp;

...

wlsFilter ->filter(disparity, left, filteredDisp, rightDisparity);

[mohanen]

// OpenCV's DisparityWLSFilter implicitly assumes that the disparity is 16SC1 format
// and has been scaled up by 16 (yeah this part is very confusing).
// Therefore, for right disparity, you need to multiply by 16, and negate it (hence -16).

@lingjiankong The 16 is due to the fact that its a fixed point representation fixed<16,4> and requires conversion to float to be used.

fixed<16,4> denotes a 16-bit fixed-point number, of which 4 rightmost bits are fractional.

To convert that, perceive the bit string as an integer and divide it with 2 power number of bits used for the fraction part. 2^4 = 16.

[sotsuka-fixstars]

@lingjiankong Thank you for your work!

If we shared param with left matcher (or we shared sgm::StereoSGM instance as left and right matcher),
it is not needed to multiply 16 as sub pixel scale.

- rightDisparity *= -16;
+ rightDisparity *= -1;

disc	image
input
libSGM
libSGM(4PATH, w/ subpixel)+WLS
libSGM(4PATH, w/o subpixel)+WLS

[funmonty]

@sotsuka-fixstars Great results. The WLS filter looks surprisingly promising. Can you share any pointclouds of your version of WLS filter? Does it have the same pyramid effect that @mohanen has described in this thread?

[sotsuka-fixstars]

@funmonty
I thought so at a first glance.
However, WLS Filter does not work fine in KITTI benchmark at our experience.
@8BitCatJQW made another point cloud application using libSGM. It may help you.

https://github.com/8BitCatJQW/libsgmIssueDetails

I modified it, earned following outputs by using different filter.

filter	KITTI stereo 2015 000006_10	Middlebury 2006 Baby
None
WLS
Bilateral

[mohanen]

@sotsuka-fixstars, Can you share the filter code snippets.

[sotsuka-fixstars]

@mohanen Is a following snippet an answer to the request?

// WLS
cv::Ptr<cv::ximgproc::DisparityWLSFilter> wls_f = cv::ximgproc::createDisparityWLSFilterGeneric(true);
cv::Mat wls_filtered_disp;
wls_f->filter(disparity, I1, wls_filtered_disp, disp_right);

// Bilateral
cv::cuda::GpuMat g_disp, g_filtered_disp;
g_disp.upload(disparity);
cv::cuda::bilateralFilter(g_disp, g_filtered_disp, kernel_size, sigma_color, sigma_spatial);
cv::Mat bilateral_filtered_disp;
g_filtered_disp.download(bilateral_filtered_disp);

kernel_size, sigma_color, sigma_spatial come from command line argument.
I decided these params slackly because I didn't know proper values of these...

[ynma-hanvo]

is this implementation the cuda version of opencv StereoSGBM?
i found the result of the two is different. there is a blocksize parameter in StereoSGBM, which i cann't found here.
thanks for the explanation!

[atakagi-fixstars]

Hi, @ynma-hanvo

is this implementation the cuda version of opencv StereoSGBM?

No.
In libSGM, 9x7 center-symmetric census transform is used for matching cost.

[mcelhennyi]

@mohanen Is a following snippet an answer to the request?

// WLS
cv::Ptr<cv::ximgproc::DisparityWLSFilter> wls_f = cv::ximgproc::createDisparityWLSFilterGeneric(true);
cv::Mat wls_filtered_disp;
wls_f->filter(disparity, I1, wls_filtered_disp, disp_right);

// Bilateral
cv::cuda::GpuMat g_disp, g_filtered_disp;
g_disp.upload(disparity);
cv::cuda::bilateralFilter(g_disp, g_filtered_disp, kernel_size, sigma_color, sigma_spatial);
cv::Mat bilateral_filtered_disp;
g_filtered_disp.download(bilateral_filtered_disp);

kernel_size, sigma_color, sigma_spatial come from command line argument.
I decided these params slackly because I didn't know proper values of these...

Does the bilateral filter work in subpixel mode? Do we need to filter off the invalid pixels first for better or accurate filtering?