Validation of work of annotators

[mistermult]

Usecase: Many annotators annotate images according to the instructions Bob has created. Now Bob wants to make sure all annotations are correct before they get used e.g. for training NN.
Only certain people, e.g. Bob, have the rights to 'validate' annotations.

Possible Solutions:

• A annotation (possible over many frames) can either be 'not-validated' (default), 'accepted', or 'rejected'. It is only possible to accept/reject a whole annotation and not single frames. When a annotation is rejected, a message can be attached with the reason (or other hints). The message is fixed to a certain frame. There might be multiple messages for multiple frames.
• When Bob opens the task in the Annotation Editor, he can reject/accept the current selected annotation with a shortcut or the context menu. When he rejects a annotation, a popup pops up and asks for a reason. The message is attached to the shape and current frame. There is a shortcut to accept all annotations that have not been rejected yet.
• Accepted/Rejected annotations are marked with a red/green dot in the top-left corner of the shape. When clicking on the red dot, the message pops up with all the reasons (This might be improved in later versions).
• In the main menu, for each task the number of rejected/accepeted/non-validated annotations are displayed. So annotators can see if they have to correct their work.
• If a annotation is modified (e.g. by the annotator), the state of the annotation is changed to non-validated again.
• So after the annotators have corrected their work, there are non-validated annotations again. This can be seen in the main menu and Bob only has to recheck the corrected annotations.

I'm not sure if @nmanovic or @bsekachev have mentioned that they have something like that internally. What are your ideas on this proposal? Any improvements or completely different ideas?

[mistermult]

Also see #851, where this was discussed in a wider setting.

[nmanovic]

@mistermult , great use case! In general I like the whole description. Some comments:

• Need to generate a validation task. In many cases you don't want to validate all annotations. Only some subset. Thus in validation mode you should be able to see only subset of annotations and frames (let's say 1%-10% of data). The part is critical. Also annotations always will have errors. Do we want to fix them all (probably impossible)?
• I like the suggested hierarchical approach. You can accept all annotations in the validation task, by frames, or individually review each object.
• Definitely need to have a conversation thread. In Google Docs you can select a required word or sentence and start writing comments. I think the same approach will work here. Thus it will be possible to select the whole frame, choose a shape/set of shapes, or select a region on an image and write comment. Comments need only to support and explain non-obvious decisions (e.g. why the image/object was rejected)
• I agree that it is necessary to have statistics with unknown/rejected/accepted objects and frames. It will be great to calculate annotation quality using the statistics.

Note: The primary effect of validation is not to improve annotations but calculate its quality (% of errors). If annotation quality is acceptable we should not try to update it (useless to fix specific problems in a subset of a big dataset).

Note: It is necessary to reuse CVAT annotation functionality for validation. It looks like a regular annotation task with some specific visual effects. Need to think if it is possible but it will be in accordance with "eating your own dog food" approach. Probably UI will be slightly different (extra visual elements) but the server should reuse core features.

[mistermult]

@nmanovic What do you think about the following proposal?

• Each annotation has the states unkown/to-validate/accepted/rejected.
• There can be messages for each annotation. Messages for frames and regions of images are implemented with dummy annotations.
• For an existing annotation task a validation task can be created. When creating the validation task, Bob can select what percentage (0%-100%) he wants to validate (For later: Create a filter to validate 5% of 'car'-annotations etc.). Then a subset of annotations is marked with 'to-validate'.
• Bob opens the validation task with the usual CVAT functionality. He only sees the annotation that with state to-validate/accepted/rejected and can reject/accept them (as described above).
• The annotations are displayed in the right menu (as usual). The messages of a annotation are displayed below each annotation. Long messages are shortened and can get expanded. Each message also has a frame number. Clicking on it navigates to the respective frame. There is a button to add additional messages.
• Alternative: Display the messages in an extra region (right or below).

I suggest first implementing it with reduced message support and adding all the message features later.

[nmanovic]

@mistermult ,

I can say the that approach looks reasonable. You can add messages as an attribute "comment" with text type.

[VL3DGA]

@mistermult @nmanovic It would be really great to have such a mode.

How about a Validation Mode similar to the Attribute Annotation Mode?

• Can also be accessed in an annotation task
• Has a Shortcut to toggle the "Validation Mode" (similar to AAM)
• Leads to a close-up of the to-be-validated annotation
• View is locked on annotation and can only be zoomed-in and zoomed-out (Maybe checkbox to allow other movement)
• Buttons and Shortcuts for "Accept", "Reject", "Next" and "Previous"
• Maybe Button and Shortcut for "Revome Validation"? Which resets the "accepted/rejected" property
• On "Reject": Window-Popup to insert comment -> Just Enter to not leave a comment
• On "Reject"/"Accept": Move to next annotation? Or should it be manual via "Next"? (Maybe a checkbox " Next on Accept/Reject"; default: checked?)
• Options/Flags to skip "Accepted" and/or "Rejected" annotations (probably via checkboxes; default: checked?)
• This "Validation Mode" could also be the default for a validation task and could be opted out of.
• In "Annotation Mode" use shortcuts for "Accept" and "Reject". This way it is possible to quickly validate a few annotations without having to go into "Validation Mode". Also add option to right-click context menu for annotations. Maybe add Buttons if there is enough space for it.
• On the option to accept all non-rejected annotations: How about an additional "Accept all previous frames"? Sometimes tasks are too big to validate all in one go and it would be nice to just accept all previous frames (probably including the current frame).

[Rub21]

The validation of the right attributes for objects and the right box draws for the objects are important to generate good quality data, Leaving a message or give options to flag certain images on the app for the different annotator will be helpful to be more communicative across annotators.

On my team, we use CVAT for labeling thousands of images with different attributes for each object. We've done our own validator, something simple which read the XML dump files and display many images at once in a grid, it helps visually go faster through many images and if the image is wrong tagged we click in the image and it goes to the CVAT validator to fix them. #793