Advanced camera calibration

[NiklasNeugebauer]

This adds functionality to work with undistorted images and convert points to and from the distorted image.

It also adds support for fisheye and omnidirectional camera models.
This adds an extra model: CameraModel parameter to Intrinsics, which is CameraModel.PINHOLE per default,
where CameraModel is a simple enum.

• add Tests
• add and adapt docstrings
• final cleanup
• fix AttributeError in pytests with omnidirectional cameras (OpenCV version?)
• fix test_omnidirectional_calibration_from_points
• documentation/overview about new methods (what's the difference, when to use which)
• review

[NiklasNeugebauer]

test_fisheye_calibration_from_points still fails.
An open question is which parameters to use for the demo camera.
Right now, based on camera parameters, the calibration seems to either diverge or converge to a focal length of 255 instead of the expected 570.

[NiklasNeugebauer]

Solved the failing calibration by fixing skew and using an intrinsic guess

[NiklasNeugebauer]

I have started adding OpenCVs omnidirectional model as well.
I will also add this to the pull request

[falkoschindler]

@NiklasNeugebauer I'm still a bit confused by the many new methods:

• What is the difference between undistort_points and points_to_rays?
• Who uses undistort_points, distort_points, undistorted_size, undistort_image?

Maybe we can improve naming and/or documentation?
Maybe a module docstring could give an overview over the different transformations the Calibration class provides?

[falkoschindler]

@NiklasNeugebauer I managed to fix the AttributeError on GitHub by changing the requirement from "opencv-contrib-python-headless" to "opencv-contrib-python". Now the omnidirectional cameras pass most tests and only one conflict remains:

assert 290.9771728515625 == 800 ± 8.0e-04

I guess there's a problem with the layout of the test setup. All world points are located underneath the camera which is, however, omnidirectional. Even though I'd expect it to converge anyway because of the absence of noise, maybe it works better by placing points all around the camera? Then we shouldn't assume all image points to be non-zero, but filter them to get a subset of successful projections.

Can you take it from here?

[falkoschindler]

I just experimented with

def demo_omnidirectional_data() -> tuple[CalibratableCamera, list[Point3d]]:
    cam = CalibratableCamera(id='1')
    cam.set_perfect_calibration(width=800, height=600, focal_length=800,
                                x=0.1, y=0.2, z=0.3,
                                roll=np.deg2rad(180+10), pitch=np.deg2rad(20), yaw=np.deg2rad(30))
    assert cam.calibration and cam.calibration.intrinsics
    cam.calibration.intrinsics.distortion = [-0.3, 0.06, -0.001, 0.0002]
    cam.calibration.intrinsics.xi = 0.8
    cam.calibration.intrinsics.model = CameraModel.OMNIDIRECTIONAL

    world_points = [
        Point3d(x=x, y=y, z=z)
        for x in np.linspace(-5.0, 5.0, 5)
        for y in np.linspace(-5.0, 5.0, 5)
        for z in np.linspace(-5.0, 5.0, 5)
    ]
    return cam, world_points

and filtering the points with

image_points, world_points = zip(*[(i, w) for i, w in zip(image_points, world_points) if i is not None])

And I think we need to pass the point arrays to cv2.omnidir.calibrate like this

                objectPoints=world_point_array[0],
                imagePoints=image_point_array[0],

But somehow OpenCV keeps complaining about size and type of these arrays.