Charuco board detection is not suitable for calibration?

[victor1234]

System Information

OpenCV python version: 4.7.0
Operating System / Platform: Debian GNU/Linux 11 (bullseye)
Python version: 3.9.2

Detailed description

It's my first time using charuco and I did not expect to encounter so many problems with this method.

According to the OpenCV docs, the idea of introducing Charuco was to make the classic chessboard pattern robust to partial view.

But when I ran the charuco corners detection, I realized that the detected chessboard corners did not match the real ones

According to the documentation and source code of cv::CharucoDetector::detectBoard(), this function doesn't detect chessboard corners. Instead, it detects inner ArUco markers and uses them to estimate the camera pose and project back the chessboard corners. It has two options: when you have camera intrinsic parameters and when you don't (it's our option). For the second option, it uses a homography matrix to estimate the chessboard corners, which means that this method does not account for distortion. Therefore, the reprojected positions of corners may be incorrect. After that, the method calls the cv::cornerSubPix() method to refine the corners' positions, and if the initial reprojection position is not far away from the real one, this call could produce accurate detection. That's why not all corners are found incorrectly.

Although this method is mentioned in many tutorials, it is not suitable for calibration.

Steps to reproduce

I got this problem running my own code, but for baseline I ran the official example

./example_aruco_calibrate_camera_charuco -v=<...>/frame-%03d.png -d 0 -h=6 -w=4 --sl=0.3175 --ml=0.2540 --sc --rs

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[qq978358810]

66666

[AleksandrPanov]

@viktor1234 could you attach the original sample image?

[victor1234]

[asmorkalov]

@victor1234 Thanks a lot for the issue report! It's definitely critical issue for ChAruco detector and calibration. Will try to solve it ASAP. Also big thanks for the original image! Could you provide ChaRuco board parameters: dictionary, geometry parameters, etc?

[victor1234]

"markerLength": 0.2540,
"squareLength": 0.3175,
"numberOfMarkersX": 4,
"numberOfMarkersY": 6,
"dictionary": "DICT_4X4_50",

You are welcome.
But Charuco calibration has been in many tutorials and examples for years. I could not understand how it was possible to miss this error. What was the original idea of using reprojected(!) corner positions for calibration?
Low unit test coverage makes OpenCV too risky to use even for prototyping

[asmorkalov]

The code was migrated from opencv_contrib and we are actively working on its stabilization.

[AleksandrPanov]

@victor1234, I found the problem. You are using the deprecated ChArUco template. Just add this code to fix your problem:

board.setLegacyPattern(True)

Charuco templates was updated in opencv/opencv_contrib#3174. The behavior has changed only for templates with an even number of rows. This was done to make the ChArUco patterns compatible with the chessboard.
Use gen_pattern.py to create new pattern or use setLegacyPattern.

Also OpenCV team tested CharucoDetector with another camera:

The results were also quite good. Nevertheless, we need to make a big comparison of findChessboardCorners vs CharucoDetector.
If the CharucoDetector does not work well enough, the OpenCV team can try to improve the algorithm using cornerHarris or other methods. But we need a large dataset and a community request.

[AleksandrPanov]

The script to generate the ChArUco board:

cols = 4
rows = 6
square_size = 100
marker_size = 0.8*square_size
aruco_dict = cv.aruco.getPredefinedDictionary(0) # cv.aruco.DICT_4X4_250
board = cv.aruco.CharucoBoard((cols, rows), square_size, marker_size, aruco_dict)
board.setLegacyPattern(True) # comment this line to create the new template
from_cv_img = board.generateImage((cols*square_size, rows*square_size))
cv.imshow("board_6_4_dict_0", from_cv_img)
cv.imwrite("board_6_4_dict_0.png", from_cv_img)

The deprecated ChArUco template:

The new ChArUco template (this template is compatible with the chessboard):

[victor1234]

Thanks. Will check in days.

[AleksandrPanov]

@victor1234, any updates?

[victor1234]

Legacy parameter helped.
But in v4.7 there are lot outliers markers detected on landscape, outside of the charuco board. And if I set the refine parameter, opencv produces segfault.
In v 4.8 small boards detected fine, without outliers, but bigger ones (12x8 for example) detected a few times worse than in v4.7. only about 20% boards detected

[asmorkalov]

@victor1234 Thanks for the feedback. We are looking on options how to rework Charuco pipeline and data with corner cases will be useful. Could you share some images with us?

[AlonSpinner]

Hi,

I am also facing some problems with ChAruCo detection.
This is the board used (opencv-python, 4.8)

marker_length: 7.35294117647 #mm
square_length: 12.5 #mm
pattern_size: [20,10] 
aruco_dict: 1 #aruco.DICT_4X4_100

code used to detect corners:

charuco_detector_left = cv2.aruco.CharucoDetector(board, charucoParams_left, detectorParams, refineParams)
...
charucoCorners_left_i, charucoIds_left_i, _ ,_  =  charuco_detector_left.detectBoard(image_left)

Images:

encv/opencv/assets/46029153/8635fa6e-6832-40ce-b947-2ec8ef951498)

If you would like, we can supply a lot more images

Thanks,
Alon

[victor1234]

@victor1234 Thanks for the feedback. We are looking on options how to rework Charuco pipeline and data with corner cases will be useful. Could you share some images with us?

#24117

[victor1234]

I want to finalize this issue

1. The tutorial text has not been updated.
2. The charuco template was changed recently, and users still have a large number of datasets with the old template. If the detector does not notify that an outdated template has been detected, the documentation must specify that the algorithm has changed right in the description of the module where it is visible, not in the description of a function that no one knows.
3. Using the values K, D in the calibration algorithm, which need to be found, I consider to be unstable. Here's an example.
   
   Check point with id=1
   if I change the refinement method from subpix to contour, the error will disappear, but this is not a solution.

[damaha]

@victor1234, I found the problem. You are using the deprecated ChArUco template. Just add this code to fix your problem:

board.setLegacyPattern(True)

Charuco templates was updated in opencv/opencv_contrib#3174. The behavior has changed only for templates with an even number of rows. This was done to make the ChArUco patterns compatible with the chessboard. Use gen_pattern.py to create new pattern or use setLegacyPattern.

Also OpenCV team tested CharucoDetector with another camera: The results were also quite good. Nevertheless, we need to make a big comparison of findChessboardCorners vs CharucoDetector. If the CharucoDetector does not work well enough, the OpenCV team can try to improve the algorithm using cornerHarris or other methods. But we need a large dataset and a community request.

I would say that the cornerHarris should be an integrated part of any charuco detector pipeline when using detections for camera calibration. It is fundemental to place detected corners as precise as possible, based on the information in the pixel values, not by reprojections as we otherwise might introduce a bias in estimation of camera parameters.

Here's a picture of an 'old' charuco pattern taken with my iPhone:

Which I can detect with:

aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_250)
board_shape = (12,8)
checker_size = 20
marker_size = 15

img = cv2.imread("IMG_1024.png")
img_b = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
board = cv2.aruco.CharucoBoard(board_shape, checker_size, marker_size, aruco_dict)
board.setLegacyPattern(True)
c_detector = cv2.aruco.CharucoDetector(board)
c_crs, c_ids, _, _ = c_detector.detectBoard(img_b)
img_markd = cv2.aruco.drawDetectedCornersCharuco(img.copy(), c_crs, c_ids)
plt.imshow(img_markd)
plt.show()

but I am not so happy with the precision of corners, e.g. id=38

I have tried adding:
c_detector.setRefineParameters(cv2.aruco.RefineParameters(cv2.aruco.CORNER_REFINE_[SUBPIX/CONTOUR]))
before running detectBoard(), but see no difference. Perhaps I am doing something wrong?! I find it difficult to find documentation on how I call the refinement procedure in newer OpenCV-python versions.

Heres a charuco pattern generated in OpenCV-4.8 with above parameters.

but due to printer scaling

checker_size = 21.5 #mm
marker_size = 16.4 #mm

Not sure how I initiate a community request. Is it a pull request on the repo you need?

[damaha]

For now my problems were solved by calling the cv2.cornerSubPix(...) on the corners from the CharucoDetector. That functions seems to do a really good job when initial corners estimates are not too far from their true position.

[asmorkalov]

Related PR: #24479

[AleksandrPanov]

I want to finalize this issue

1. The tutorial text has not been updated.
2. The charuco template was changed recently, and users still have a large number of datasets with the old template. If the detector does not notify that an outdated template has been detected, the documentation must specify that the algorithm has changed right in the description of the module where it is visible, not in the description of a function that no one knows.
3. Using the values K, D in the calibration algorithm, which need to be found, I consider to be unstable. Here's an example.
   
   Check point with id=1
   if I change the refinement method from subpix to contour, the error will disappear, but this is not a solution.

@victor1234, thx, the module description and tutorial will be corrected. Can you attach the original image of the example?

[asmorkalov]

Partially fixed in 4.9.0. Moved to 4.10.0 milestone to finalize remaining issues.