StrongSORT + custom appearance descriptor

[tkra90]

Hello!
I built a custom tracker using your implementation of StrongSORT. I have a the detections from a yoloR detector, trained a custom appearance descriptor for my use case (traffic signs) and I am working on images (snapshots 1-2 frames /s, I don't have access to videos). So I have images taken from a car, I need to track each traffic sign (camera is moving, objects are fixed).

After running the algorithm, I noticed that some fine-tuning is needed:
Because of

N_INIT: 3 # Number of frames that a track remains in initialisation phase

the first 2 frames don't have track ids assigned, after setting N_INIT: 0, ids are still missing from the first frame.

BUT the biggest problem is something else:

frame 12:
array([ 1764, 859, 1795, 933, 1, 123, 0.94644]) # output of tracker
array([[ 1765, 860, 1795, 933, 0.94644, 123]]) # det I am passing to the tracker

frame 13:
array([ 1610, 816, 1643, 895, 1, 123, 0.92171]) # output
array([[ 1610, 816, 1644, 896, 0.92171, 123]]). # det

frame 14:
array([ 1488, 760, 1523, 843, 1, 123, 0.92171]) # output
array([[ 817, 732, 849, 821, 0.73291, 123]]). # det

I noticed some of the bboxes get shifted, even if I pass the correct ones to the tracker. (when I write the jpg with the bbox / check the txt, it is not in the right place, track id is correct)
In this particular case, there is a significant change from frame 13 to frame 14.
tracker.camera_update(prev_frames[i], curr_frames[I]) does this compensation I guess, any ideea how could I fine tune ECC #Compute the warp matrix from src to dst ?

[mikel-brostrom]

The tracker returns the bbox predictions ahead:

https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/blob/92cd8447b915e5aea87b398b787d60f1f0be95e7/trackers/strong_sort/strong_sort.py#L76

Not the yolo predicted bboxes. When there is intrinsic motion from the camera, this has to be taken into consideration as well when doing the prediction ahead.

[tkra90]

Thanks for your answer!
These are predictions are computed individually? I have another example where on frame A are 2 bboxes, both of them are tracked correctly, where on frame B one of the detections is in the right place, the 2nd is shifted, due to motion of the camera?
How could I fine tune it, what's your suggestion? :)

Another issue, what I am facing is the ID assignment: on frames 1-10 there is a detection tracked with ID-1, on frame 33 a totally different class of box gets assigned an ID-1, although my understanding is that this is controlled by MAX_AGE: 5

[tkra90]

Also I found a little bug:
https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/blob/92cd8447b915e5aea87b398b787d60f1f0be95e7/trackers/strong_sort/sort/track.py#L121

When using cv2.MOTION_HOMOGRAPHY as warp_mode, the output's shape is (3,3), so here it will crash:
https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/blob/92cd8447b915e5aea87b398b787d60f1f0be95e7/trackers/strong_sort/sort/track.py#L225

if warp_matrix.shape[0] == 2:
            [a,b] = warp_matrix
            warp_matrix=np.array([a,b,[0,0,1]])

Solves the issue, for warp_matrix computed with AFFINE, EUCLIDEAN, or TRANSLATION.

[mikel-brostrom]

Thx for pointing this out. Will add this 😊