Add Tracking with OpenCV #1003

mistermult · 2019-12-24T16:42:25Z

Here is the promised tracking feature. It works, see the video (https://streamable.com/jmq72).

There are still some open TODOs for which I need some feedback:

It is currently implemented as a plugin. It needs to access the shapeCollectionModel. Moreover, ShapeModel has to be modified (as done here) or patched in the plugin. A new attribute has to be added to each position, to keep track which position has been created by the tracker and not the user. Should it be a plugin or added to the 'engine' code? Should the ShapeModel be patched in the plugin instead of modifying the file shapes.js ? Do the plugins work with the new React UI?
~~The tracking component is enabled by default.~~ Tracking is enabled by environment variable TRACKING. Ok?
Put tracking on server into a background task.
Minor code clean up.

Add enginePlugin.js with Tracker MVC. Modify ShapeModel: Each position has the property byMachine. If this is true, this means the position created by the tracker and not the user. This is used to let the tracker later change positions of shapes again until the next manually inserted key frame.

nmanovic · 2019-12-25T09:10:27Z

@mistermult , thanks for the contribution! It is a feature which our community waits.

I can see some problems. Let's fix them first and after that discuss how to improve the feature futher.

Could you please look as other components install necessary files? We use -f docker-compose.tracking.yml approach. Thus you can add necessary env or build variables. It is necessary to run "download" procedure automatically for goturn model. Look at other components. Only OpenVINO requires to download the package manually but only because it requires registeration to get a valid link on the product.
I got the following error when I built with OpenVINO component (WITH_TRACKING=1 docker-compose -f docker-compose.yml -f components/openvino/docker-compose.openvino.yml up -d --build)

[2019-12-25 04:28:20,172] ERROR django.request: Internal Server Error: /tracking/track
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py", line 466, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rest_framework/decorators.py", line 55, in handler
    return func(*args, **kwargs)
  File "/home/django/cvat/apps/authentication/decorators.py", line 19, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rules/contrib/views.py", line 138, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/django/cvat/apps/tracking/views.py", line 55, in track
    tracker = RectangleTracker()
  File "/home/django/cvat/apps/tracking/tracker.py", line 47, in __init__
    'BOOSTING': cv2.TrackerBoosting_create,
AttributeError: module 'cv2' has no attribute 'TrackerBoosting_create'

Automatically install GOTRUN models for OpenCV tracker in docker container if TRACKING is set to 'yes', which is done in the file docker-compose.tracking.yml.

mistermult · 2019-12-25T17:04:08Z

Thank you for your response. I integrated the tracking into the docker container. Please try to run it with:

# Maybe try to rebuild form ground up to install all packages (?)
docker-compose -f docker-compose.yml -f components/tracking/docker-compose.tracking.yml build --no-cache
# And run
docker-compose -f docker-compose.yml -f components/tracking/docker-compose.tracking.yml up

The above works in the container. Your error message indicates that the opencv-contrib-python package is not installed or not found. It should be installed in the container because it is listed in the base.txt requirements file.

EDIT: @nmanovic I can confirm your error. I tried to enable tracking and OpenVINO at the same time with:

docker-compose -f docker-compose.yml -f components/openvino/docker-compose.openvino.yml -f components/tracking/docker-compose.tracking.yml build --no-cache
docker-compose -f docker-compose.yml -f components/openvino/docker-compose.openvino.yml -f components/tracking/docker-compose.tracking.yml up

The reason is that installing OpenVINO seems to change the OpenCV version. Especially it disables or removes opencv-contrib-python with the trackers. This is confirmed by getting a shell into the container, running python3, and getting the version cv2.__verions__('4.1.2-openvino'). Maybe there is a OpenVINIO OpenCV contrib package? (also see https://software.intel.com/en-us/forums/computer-vision/topic/802372).

I suggest fixing this at the end. If nothing works, tracking or OpenVINO can run in another container.

This adds the instuctions to install and activate tracking in dev mode.

mistermult · 2019-12-30T11:11:05Z

@bsekachev @nmanovic Is there anything else to do for me regarding this PR?

nmanovic · 2019-12-31T08:21:48Z

@mistermult , thanks again for the contribution. Please give us some time to review the PR and give you a feedback. Happy New Year!

ghost · 2020-01-10T22:39:30Z

@mistermult @nmanovic We tested tracking on Onepanel and it seems to work fine without any error.

dospore · 2020-01-22T23:51:57Z

Im getting an issue with saving the annotations. Ive tested with brand new tasks and tasks I had before pulling this fork. I get two distinct errors from a bit of testing.
What works:
Saving 1 single interpolation key frame or annotation bounding box.
What doesnt work:

Saving 2 interpolation key frames ERROR MESSAGE
Couldn't to save the job. Errors occured: Could not make create annotations. Code: 400. Message
{"tracks":[{"shapes":[{},{"z_order":["A valid integer is required."]}]}]}. Please report the problem to support team immediately.
Saving a tracked bounding box between 2 key frames ERROR MESSAGE
Couldn't to save the job. Errors occured: Could not make create annotations. Code: 400. Message: {"tracks":[{"shapes":[{},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]},{"z_order":["A valid integer is required."]}]}]}. Please report the problem to support team immediately.
Tested this with a few of the tracking methods (boosting, csrt, and medianflow)

The errors are the same across old tasks and new tasks. Im a junior dev so I will try to debug a little but no promises on results. Let me know if you'd like any more information

mistermult · 2020-01-23T10:16:13Z

@tetther1122 I fixed it quickly. It was a copy-paste-error. But I cannot test it right now.

dospore · 2020-01-23T23:08:13Z

@mistermult tested. Result: it worked. Thanks, I'll let you know if I come across anything else.

alalek

Please take a look on comments below.

Disclaimer: I'm not CVAT developer, so I don't know all project details. It is just a view from OpenCV integration perspective.

alalek · 2020-01-24T02:17:09Z

cvat/apps/tracking/static/tracking/js/enginePlugin.js

+        MEDIANFLOW: 'MEDIANFLOW',
+        TLD: 'TLD',
+        MOSSE: 'MOSSE',
+        GOTRUN: 'GOTRUN',


alalek · 2020-01-24T02:20:50Z

cvat/requirements/base.txt

 opencv-python==4.1.0.25
+opencv-contrib-python==4.1.2.30


Only one package should be installed.
Installing both packages is leading to conflicts.

alalek · 2020-01-24T02:24:39Z

.gitignore

@@ -23,4 +23,8 @@ __pycache__
 # Ignore development npm files
 node_modules

+# Ignore goturn model
+goturn.caffemodel
+goturn.prototxt


AFAIK this model is very large. So placing it in source tree would lead to docker builds slowdown.

What is about .dockerignore? (if you want to add it where)
Or remove from both places.

alalek · 2020-01-24T02:27:57Z

CONTRIBUTING.md

+- Download the GOTURN model if the the OpenCV GOTURN tracker is to be used.
+```sh
+#This downloads it into the cvat folder
+components/tracking/install.sh


into the cvat folder

Root directory doesn't look as a right place.
I believe it make sense to download it into appropriate places, likecomponents/tracking/model.

Or take a look in a direction (investigate) of using docker volumes / mounted shares with large models.

alalek · 2020-01-24T02:31:24Z

Dockerfile

+ENV TRACKING=${TRACKING}
+ENV TRACKING_PATH=${HOME}/tracking
+RUN if [ "$TRACKING" = "yes" ]; then \
+    bash -i /tmp/components/tracking/install.sh; \


Tracking may be used without GOTURN.
Consider making using of GOTURN model optional.

alalek · 2020-02-18T19:48:41Z

BTW, There is some issue with GOTURN model inference in OpenCV DNN: opencv/opencv_contrib#2426

nmanovic

@mistermult , need to adopt the patch to our new UI. If you can help it will be awesome. @Priya4607 , are you going to help here?

09panesara · 2020-03-23T14:33:15Z

I'm having trouble getting the tracking to run.

I ran the commands suggested earlier:

docker-compose -f docker-compose.yml -f components/tracking/docker-compose.tracking.yml build --no-cache docker-compose -f docker-compose.yml -f components/tracking/docker-compose.tracking.yml up

But the second command seems to never terminate and occasionally logs this error: ImportError: No module named 'gitdb.utils.compat'. Is there something I am missing?

nmarchal · 2020-03-26T14:47:26Z

I tried to copy manually each of the new files and run the tracker. It seems to work (I have the tracker options in the settings etc) but when I right click and press track I get the following error :

Error: Could not track shape. Code: 405. Message: <title>405 Not Allowed</title>

405 Not Allowed

nginx/1.16.1 <!-- a padding to disable MSIE and Chrome friendly error pag

ghost · 2020-06-04T19:36:53Z

@nmanovic @azhavoro If I use the "Track" box then it places a bounding box in every frame. But the user typically runs tracking on some number frames. So if I run tracking on "Track" box then it has two boxes for frames that user-selected and one Track box in the remaining frames which shouldn't be there.. To solve this issue, we are using tracking on "Shape" box. I am not sure if that's the right thing to do. If you can provide some advice on how can we remove those boxes from other frames, then I can update it to use Track rather than the Shape.

here is a code from this (old ui) PR that might be relevant

# If we in a large task this creates unnessary many shapes
  # We only need them between start_frame and stop_frame
  shapes_of_track = TrackManager([tracking_job['track']]).to_shapes(
      stop_frame)
  first_frame_in_track = shapes_of_track[0]['frame']

  def shape_to_db(tracked_shape_on_wire):
      s = copy.copy(tracked_shape_on_wire)
      s.pop('group', 0)
      s.pop('attributes', 0)
      s.pop('label_id', 0)
      s.pop('byMachine', 0)
      s.pop('keyframe')
      return TrackedShape(**s)

  # This bounding box is used as a reference for tracking
  start_shape = shape_to_db(shapes_of_track[start_frame-first_frame_in_track])

  # Do the actual tracking and serializee back
  tracker = RectangleTracker()
  new_shapes = tracker.track_rectangles(task, start_shape, stop_frame)
  new_shapes = [TrackedShapeSerializer(s).data for s in new_shapes].

nmanovic · 2020-06-09T09:30:18Z

@san999p ,

When you create an initial track you can generate "outside" property for the latest shape. Thus when users ask to track a shape from [start_frame, end_frame] for the frame end_frame+1 you can generate a rectangle with outside=True. It will finish the track.
After that you can continue tracking. Let say you choose end_frame and say I want to track the object further. In this case you need to remove the intermediate "outside" flag and continue tracking. At the end of the tracking interval you need to generate a rectangle with "outside" property again.

Please let me know if the explanation isn't clear. Does it look right?

nmanovic · 2020-07-08T18:50:06Z

@mistermult , thanks again for the great PR. We are going to port it into new UI. I will close the PR because the old UI is going to be removed soon.

ghost · 2020-07-09T00:15:29Z

@san999p ,

When you create an initial track you can generate "outside" property for the latest shape. Thus when users ask to track a shape from [start_frame, end_frame] for the frame end_frame+1 you can generate a rectangle with outside=True. It will finish the track.

After that you can continue tracking. Let say you choose end_frame and say I want to track the object further. In this case you need to remove the intermediate "outside" flag and continue tracking. At the end of the tracking interval you need to generate a rectangle with "outside" property again.

Please let me know if the explanation isn't clear. Does it look right?

Thanks @nmanovic I will update the backend when I get some time. Would it be okay if we send PR for just UI?

mistermult added 4 commits December 24, 2019 17:16

[Tracking] Add serverside for OpenCV tracker.

9c16460

[Tracking] Fix requirements.

6168c7c

[tracking] Remove unnecessary file.s

e5f7f06

[Tracking] Integrate tracking into container.

4ee7d99

Automatically install GOTRUN models for OpenCV tracker in docker container if TRACKING is set to 'yes', which is done in the file docker-compose.tracking.yml.

[Tracking] Add Instructions for development.

9bc60e0

This adds the instuctions to install and activate tracking in dev mode.

nmanovic mentioned this pull request Jan 16, 2020

Tracking functionality for bounding boxes #211

Closed

[Tracking] Fix wrong value of z_order

5f71b01

alalek reviewed Jan 24, 2020

View reviewed changes

nmanovic requested changes Mar 18, 2020

View reviewed changes

nmanovic closed this Jul 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Tracking with OpenCV #1003

Add Tracking with OpenCV #1003

mistermult commented Dec 24, 2019 •

edited

Loading

nmanovic commented Dec 25, 2019

mistermult commented Dec 25, 2019 •

edited

Loading

mistermult commented Dec 30, 2019 •

edited

Loading

nmanovic commented Dec 31, 2019

ghost commented Jan 10, 2020

dospore commented Jan 22, 2020

mistermult commented Jan 23, 2020

dospore commented Jan 23, 2020

alalek left a comment

alalek Jan 24, 2020

alalek Jan 24, 2020

alalek Jan 24, 2020

alalek Jan 24, 2020

alalek Jan 24, 2020

alalek commented Feb 18, 2020

nmanovic left a comment

09panesara commented Mar 23, 2020

nmarchal commented Mar 26, 2020

ghost commented Jun 4, 2020 •

edited by ghost

Loading

nmanovic commented Jun 9, 2020

nmanovic commented Jul 8, 2020

ghost commented Jul 9, 2020

		opencv-python==4.1.0.25
		opencv-contrib-python==4.1.2.30

Add Tracking with OpenCV #1003

Add Tracking with OpenCV #1003

Conversation

mistermult commented Dec 24, 2019 • edited Loading

nmanovic commented Dec 25, 2019

mistermult commented Dec 25, 2019 • edited Loading

mistermult commented Dec 30, 2019 • edited Loading

nmanovic commented Dec 31, 2019

ghost commented Jan 10, 2020

dospore commented Jan 22, 2020

mistermult commented Jan 23, 2020

dospore commented Jan 23, 2020

alalek left a comment

Choose a reason for hiding this comment

alalek Jan 24, 2020

Choose a reason for hiding this comment

alalek Jan 24, 2020

Choose a reason for hiding this comment

alalek Jan 24, 2020

Choose a reason for hiding this comment

alalek Jan 24, 2020

Choose a reason for hiding this comment

alalek Jan 24, 2020

Choose a reason for hiding this comment

alalek commented Feb 18, 2020

nmanovic left a comment

Choose a reason for hiding this comment

09panesara commented Mar 23, 2020

nmarchal commented Mar 26, 2020

405 Not Allowed

ghost commented Jun 4, 2020 • edited by ghost Loading

nmanovic commented Jun 9, 2020

nmanovic commented Jul 8, 2020

ghost commented Jul 9, 2020

mistermult commented Dec 24, 2019 •

edited

Loading

mistermult commented Dec 25, 2019 •

edited

Loading

mistermult commented Dec 30, 2019 •

edited

Loading

ghost commented Jun 4, 2020 •

edited by ghost

Loading