Corners bounding box format

[leondgarse]

I got an issue when trying to port two of my currently using bounding box augment related functions resize_and_crop_bboxes and bboxes_apply_affine. I think we better set a default expecting bbox input format. For my usage, the corner bounding box is in same format with tfds COCO, that I think is [top, left, bottom, right] with value in range [0, 1].

import tensorflow_datasets as tfds
ds, info = tfds.load('coco/2017', with_info=True)
aa = ds['train'].as_numpy_iterator().next()
print(aa['image'].shape)
# (462, 640, 3)
print(aa['objects'])
# {'area': array([17821, 16942,  4344]),
#  'bbox': array([[0.54380953, 0.13464062, 0.98651516, 0.33742186],
#         [0.50707793, 0.517875  , 0.8044805 , 0.891125  ],
#         [0.3264935 , 0.36971876, 0.65203464, 0.4431875 ]], dtype=float32),
#  'id': array([152282, 155195, 185150]),
#  'is_crowd': array([False, False, False]),
#  'label': array([3, 3, 0])}

imm = aa['image']
plt.imshow(imm)

for bb in aa["objects"]["bbox"]:
    bb = np.array([bb[0] * imm.shape[0], bb[1] * imm.shape[1], bb[2] * imm.shape[0], bb[3] * imm.shape[1]])
    plt.plot(bb[[1, 1, 3, 3, 1]], bb[[0, 2, 2, 0, 0]])

It will save some effort checking format if we can set a standard. So my questions are:

• Currently in bounding_box.py, we are assuming it's LEFT, TOP, RIGHT, BOTTOM. Are we gonna set this as default?
• Is value range expecting in [0, 1] or scaled with image actual shape?

[innat]

Similar query: #21 (comment)

[leondgarse]

Right, close this then.

[innat]

@leondgarse no, you don't need to close, I didn't mean that. You should re-open it.

( I just mentioned that I've asked a similar query as a comment there but sadly still no response from the keras team. Maybe they didn't decide yet. )

[leondgarse]

Ya, personally, I cannot move on without a clarify. :)

[innat]

same here.

[LukeWood]

My instinct is that corners format makes the most sense as it is a denser format. Any thoughts?

[bhack]

Could be a bounding box a specialized type of a bounding polygon that it could be also more precise aproximation of a segmentation mask? See cocodataset/cocoapi#131

[LukeWood]

Could be a bounding box a specialized type of a bounding polygon that it could be also more precise aproximation of a segmentation mask? See cocodataset/cocoapi#131

I think we want to special case bounding boxes and explicitly support them.

[leondgarse]

My option is strict the input / output format of these bbox augment functions. Other not matching format or segmentation mask should be transformed before calling these functions. It can also benefit further anchor / anchor assign / detection lossses, that we don't need to consider other format.

[qlzh727]

1. Currently I don';t think we assume any default format for bbox, but I think we should have one default format (probably the corner bbox format).
2. I don't think we should normalized the bbox value to [0, 1], otherwise it will require the original image to decode the actual pixel.

[bhack]

@qlzh727 Check also https://discuss.tensorflow.org/t/are-bounding-box-bbox-definitions-consistent-across-datasets/8468

[qlzh727]

Thanks for the context, will take a look.

[LukeWood]

I think @qlzh727 and I have decided to move forward with corners format as the default. This is XYWH with true pixel values.

If there is a strong reason to deviate, please let us know and we can reconsider.

[leondgarse]

Ya, it's good for me. Will close this issue then.