sofima.warp.render.tiles raise cv2.error: OpenCV(4.7.0) (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'

[youssefmys]

Following the stitching example for Sofima, the stitching is fine for small number of tiles, however I have 1631 EM tiles for stitching using the following tile_id_map

id_map = np.arange(1632).reshape(34, -1)
tile_id_map = id_map.copy() 
for i in range(id_map.shape[0]):
    if i % 2 == 1:
        tile_id_map[i, :] = tile_id_map[i, :][::-1]

Each tile is down sampled to about 5 MB

def read_image(image_path, x, y):
    f = open(image_path, 'rb')
    image_data = f.read()
    image = PIL.Image.open(io.BytesIO(image_data)).convert("L")
    image_np = np.array(image) 
    image.close()
    print(f"Read image {os.path.basename(image_path)}   \r", end=" ")
    return x, y, image_np 

with futures.ThreadPoolExecutor(max_workers=20) as executor:

    for y in range(tile_id_map.shape[0]):
        for x in range(tile_id_map.shape[1]):
            tile_id = tile_id_map[y, x]
                      
            image_path =  os.path.join(args.folder, f"down_sampled_Stitched_{tile_id:0>4}.tiff")
            f = executor.submit(read_image, image_path, x, y)
            future_to_tile_id.update({f: tile_id})


for f in futures.as_completed(future_to_tile_id):
    try:
        x, y, image_np = f.result()
        tile_map[(x, y)] = image_np
    except Exception as exc:
        print(f"Generated Execption: {exc}")

The rest of the stitching code is the same as Sofima stitching example using tile_space = (tile_id_map.shape[0],tile_id_map.shape[1]) and cx, cy = stitch_rigid.compute_coarse_offsets(tile_space, tile_map, ((1500, 500),(1500, 500)))

However, warp_subvolume raise error when trying to warp the tile using sofima.warp.render.tiles .
opencv version is 4.7

File "/home/youssef/workspace/sofima_trials/./stitchsupertiles.py", line 289, in <module>
    # Warp the tiles into a single image.
  File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 517, in render_tiles
    _render_tile(tile_x=x, tile_y=y, coord_map=coord_map)
  File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 465, in _render_tile
    warped_img, warped_mask = warp_subvolume(
  File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 182, in warp_subvolume
    f.result()
  File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/youssef/anaconda3/envs/sofima/lib/python3.10/site-packages/sofima/warp.py", line 169, in _warp_section
    warped[c, z, ...] = cvx2.remap(
cv2.error: OpenCV(4.7.0) /home/conda/feedstock_root/build_artifacts/libopencv_1678824110780/work/modules/imgproc/src/imgwarp.cpp:1724: error: (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'

There are several issues concerning opencv remap() dealing with large images. I am not sure if further down sampling of tiles would solve the issue and whether it is possible to pickle the result tile meshes for down sampled tiles and re-apply for large images.
Thanks in advance.

[arentkievits]

OpenCV version 4.9.0.80

I'm also experiencing this issue, with 3D alignment of a single large volume of 54 sections, i.e.

data_2_warp.shape
(34039, 28719, 54, 1)

I'm warping section for section:

warped = [np.transpose(data_2_warp[:, :, 0:1, 0], [2, 1, 0])]

for z in tqdm(range(1, data_2_warp.shape[2])):
  data_box = bounding_box.BoundingBox(start=(0, 0, 0), size=(data_2_warp.shape[0], data_2_warp.shape[1], 1))
  out_box = bounding_box.BoundingBox(start=(0, 0, 0), size=(data_2_warp.shape[0], data_2_warp.shape[1], 1))

  data = np.transpose(data_2_warp[data_box.start[0]:data_box.end[0],
                                  data_box.start[1]:data_box.end[1],
                                  z:z+1, 0:1], [3, 2, 1, 0])
  warped.append(
      warp.warp_subvolume(data, data_box, inv_map_full[:, z:z+1, ...], box_high, stride, out_box, 'lanczos', parallelism=1)[0, ...])

Warping a subvolume (e.g. 2000x2000x54) of this array works fine and the result is as expected. But applying the warping to the full data_2_warp array gives the exact same error after about 5 minutes, for the first iteration of the loop:

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
Cell In[155], line 11
      5 out_box = bounding_box.BoundingBox(start=(0, 0, 0), size=(data_2_warp.shape[0], data_2_warp.shape[1], 1))
      7 data = np.transpose(data_2_warp[data_box.start[0]:data_box.end[0],
      8                                 data_box.start[1]:data_box.end[1],
      9                                 z:z+1, 0:1], [3, 2, 1, 0])
     10 warped.append(
---> 11     warp.warp_subvolume(data, data_box, inv_map_full[:, z:z+1, ...], box_high, stride, out_box, 'lanczos', parallelism=1)[0, ...])

File ~/miniconda3/envs/sofima/lib/python3.9/site-packages/sofima/warp.py:176, in warp_subvolume(image, image_box, coord_map, map_box, stride, out_box, interpolation, offset, parallelism)
    173     fs.add(exc.submit(_warp_section, z=z))
    175   for f in futures.as_completed(fs):
--> 176     f.result()
    178 # Map IDs back to the original space, which might be beyond the range of
    179 # int32.
    180 if orig_to_low is not None:

File ~/miniconda3/envs/sofima/lib/python3.9/concurrent/futures/_base.py:439, in Future.result(self, timeout)
    437     raise CancelledError()
    438 elif self._state == FINISHED:
--> 439     return self.__get_result()
    441 self._condition.wait(timeout)
    443 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File ~/miniconda3/envs/sofima/lib/python3.9/concurrent/futures/_base.py:391, in Future.__get_result(self)
    389 if self._exception:
    390     try:
--> 391         raise self._exception
    392     finally:
    393         # Break a reference cycle with the exception in self._exception
    394         self = None

File ~/miniconda3/envs/sofima/lib/python3.9/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     55     return
     57 try:
---> 58     result = self.fn(*self.args, **self.kwargs)
     59 except BaseException as exc:
     60     self.future.set_exception(exc)

File ~/miniconda3/envs/sofima/lib/python3.9/site-packages/sofima/warp.py:163, in warp_subvolume.<locals>._warp_section(z)
    155 dx, dy = cv.convertMaps(
    156     dx,
    157     dy,
    158     dstmap1type=maptype,
    159     nninterpolation=(interpolation == cv.INTER_NEAREST),
    160 )
    162 for c in range(0, image.shape[0]):
--> 163   warped[c, z, ...] = cv.remap(
    164       image[c, z, ...], dx, dy, interpolation=interpolation
    165   )

error: OpenCV(4.9.0) /io/opencv/modules/imgproc/src/imgwarp.cpp:1744: error: (-215:Assertion failed) dst.cols < SHRT_MAX && dst.rows < SHRT_MAX && src.cols < SHRT_MAX && src.rows < SHRT_MAX in function 'remap'

Would a workaround be to use the distributed mesh optimization? Unfortunately I am running into some other error there.

[arentkievits]

After doing some Googling, I found the following on the openCV forum: https://forum.opencv.org/t/about-shrt-max-in-function-cv-remap/2884

It appears that imwarp.cpp uses (short) integer arithmetics for performance reasons, so the image cannot be larger than 0x7fff (32767 pixels). It seems that this is an unchangeable parameter.

I tried cropping the x dimension of my data array to 32767-1 before warping (setting it to 32767 still made it fail), since all other dimensions are within the limit.

When doing this, it doesn't throw the error on my side. So it seems that there is a hard limit of 32766 pixels on each dimension of your image array.

[mjanusz]

The best workaround here is likely to simply work with smaller image fragments, even if the sections themselves are very large. Mesh and flow processing can still be done as usual, but the rendering of the warped data should be done tile-wise (with user-defined tiles -- something like 1k^2 or 2k^2 pixels would be typical). In the em_alignment notebook, rendering is done section-wise, but warp.warp_subvolume can process arbitrary bounding boxes.