Distance based AP metric

[swsuggs]

For #1945.

I decided to take the simple approach of measuring distance in pixels from the center of the patch to the center of the objects.
This means it does not regard patch or object size. E.g. an object 200 pixels away from the patch center may or may not be overlapping with the patch. So there is no distinction between objects overlapping the patch and objects not overlapping the patch. It would be easy enough to normalize the distance by the size of the patch, but this would still be a little ambiguous if, say, the patch was a rectangle. I'm open to thoughts on this.

Output is of the following format, where the dictionary key is the distance in pixels:

benign_AP_per_class_by_distance_from_patch on benign examples w.r.t. ground truth labels: {
50: {'mean': 1.0, 'class': {2: 1.0}}, 
100: {'mean': 0.5900000000000001, 'class': {1: 0.27, 2: 0.91}}, 
150: {'mean': 0.755, 'class': {1: 0.6, 2: 0.91}}, 
200: {'mean': 0.7250000000000001, 'class': {1: 0.54, 2: 0.91}}, 
250: {'mean': 0.745, 'class': {1: 0.58, 2: 0.91}}, 
300: {'mean': 0.79, 'class': {1: 0.67, 2: 0.91}}, 
350: {'mean': 0.775, 'class': {1: 0.65, 2: 0.9}}, 
400: {'mean': 0.7, 'class': {1: 0.59, 2: 0.81}}, 
450: {'mean': 0.73, 'class': {1: 0.65, 2: 0.81}}, 
500: {'mean': 0.7, 'class': {1: 0.59, 2: 0.81}}, 
550: {'mean': 0.7, 'class': {1: 0.6, 2: 0.8}}, 
600: {'mean': 0.7250000000000001, 'class': {1: 0.65, 2: 0.8}}, 
650: {'mean': 0.6950000000000001, 'class': {1: 0.59, 2: 0.8}}, 
700: {'mean': 0.69, 'class': {1: 0.58, 2: 0.8}}, 
750: {'mean': 0.685, 'class': {1: 0.58, 2: 0.79}}, 
800: {'mean': 0.685, 'class': {1: 0.58, 2: 0.79}}, 
850: {'mean': 0.685, 'class': {1: 0.58, 2: 0.79}}, 
900: {'mean': 0.685, 'class': {1: 0.58, 2: 0.79}}, 
950: {'mean': 0.6799999999999999, 'class': {1: 0.58, 2: 0.78}}, 
1000: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}, 
1050: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}, 
1100: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}, 
1150: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}
}

[dxoigmn]

One idea is to use GIoU as a proxy for whether any object overlaps the patch: https://giou.stanford.edu. It extends IoU to take into account the case when boxes do not overlap.

[swsuggs]

Trying it out with GIoU.

GIoU ranges from -1 to 1 where anything greater than 0 indicates some overlap with the patch. However, it is also possible for overlapping boxes to have negative GIoU, especially if the box and patch have very different sizes.

Consider an example of a pedestrian box of area 10 totally inside a patch of area 100.
GIoU = IoU - (C - U) / C
where U is the union and C is the smallest box enclosing both boxes under consideration.
In this case, IoU = 10/100, C = 100, and U = 10. So GIoU = 0.1 - 0.9 = -0.8

This makes it hard to interpret exactly which boxes are included if I report an mAP value for boxes at a given GIoU.

To further complicate things, the patch is not expected to be square due to foreshortening, so I have used the smallest box enclosing the patch instead. Thus, a positive GIoU does not necessarily imply patch overlap but it will be close.

These are potential downsides of this approach. Any thoughts @deprit @dxoigmn @jprokos26 ?

The output is now as follows:

benign_AP_per_class_by_min_giou_from_patch on benign examples w.r.t. ground truth labels: {
0.0: {'mean': 0.735, 'class': {1: 0.57, 2: 0.9}}, 
-0.1: {'mean': 0.7, 'class': {1: 0.58, 2: 0.82}}, 
-0.2: {'mean': 0.695, 'class': {1: 0.57, 2: 0.82}}, 
-0.30000000000000004: {'mean': 0.69, 'class': {1: 0.57, 2: 0.81}}, 
-0.4: {'mean': 0.68, 'class': {1: 0.56, 2: 0.8}}, 
-0.5: {'mean': 0.69, 'class': {1: 0.58, 2: 0.8}}, 
-0.6000000000000001: {'mean': 0.685, 'class': {1: 0.58, 2: 0.79}}, 
-0.7000000000000001: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}, 
-0.8: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}, 
-0.9: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}, 
-1.0: {'mean': 0.6499999999999999, 'class': {1: 0.58, 2: 0.72}}}

It would of course be possible to threshold above 0 as well, but my impression was that we wanted to consider all "overlapping" objects together. Thresholding above 0 might tend to sort by box size -- larger overlapping boxes would better match the patch and so have higher GIoU.

[dxoigmn]

Consider an example of a pedestrian box of area 10 totally inside a patch of area 100. GIoU = IoU - (C - U) / C where U is the union and C is the smallest box enclosing both boxes under consideration. In this case, IoU = 10/100, C = 100, and U = 10. So GIoU = 0.1 - 0.9 = -0.8

U should be 100 since the size of union of these boxes is just the larger, encompassing box. Thus GIoU = 10/100 - (100 - 100)/(100) = 0.1.

I tried constructing this scenario using torchvision's generalized_box_iou_loss:

>>> import torch
>>> from torchvision.ops import generalized_box_iou_loss
>>> 1 - generalized_box_iou_loss(torch.tensor([0, 0, 5, 2]), torch.tensor([0, 0, 10, 10]))
tensor(0.1000)

Note that GIoU = 1 - L_GIoU

[swsuggs]

Good catch, thank you. In fact, if the box is completely within the patch, then U = C and GIoU = IoU.

[deprit]

To further complicate things, the patch is not expected to be square due to foreshortening, so I have used the smallest box enclosing the patch instead. Thus, a positive GIoU does not necessarily imply patch overlap but it will be close.

@swsuggs Would it be too cumbersome to handle non-square patches? We'd need to compute the following:

• Convex hull of points (this is in scipy)
• Intersection of 2 polygons
• Area of a polygon

[dxoigmn]

It would also be nice to produce metrics for y_d < threshold.

[swsuggs]

Do you think output like this is sufficiently clear and useful? Note the latter subdict does not include -0.9 because there were no boxes that distant.

adversarial_AP_per_class_by_giou_from_patch on benign examples w.r.t. ground truth labels: {
'cumulative_by_min_giou': {
        0.0: {'mean': 0.005, 'class': {1: 0.0, 2: 0.01}}, 
        -0.1: {'mean': 0.065, 'class': {1: 0.05, 2: 0.08}},
        -0.2: {'mean': 0.15000000000000002, 'class': {1: 0.07, 2: 0.23}}, 
        -0.30000000000000004: {'mean': 0.185, 'class': {1: 0.13, 2: 0.24}}, 
        -0.4: {'mean': 0.2, 'class': {1: 0.15, 2: 0.25}}, 
        -0.5: {'mean': 0.265, 'class': {1: 0.2, 2: 0.33}}, 
        -0.6000000000000001: {'mean': 0.285, 'class': {1: 0.22, 2: 0.35}}, 
        -0.7000000000000001: {'mean': 0.305, 'class': {1: 0.25, 2: 0.36}}, 
        -0.8: {'mean': 0.315, 'class': {1: 0.27, 2: 0.36}}, 
        -0.9: {'mean': 0.355, 'class': {1: 0.29, 2: 0.42}}, 
        -1.0: {'mean': 0.355, 'class': {1: 0.29, 2: 0.42}}
        }, 
'cumulative_by_max_giou': {
        0.0: {'mean': 0.505, 'class': {1: 0.4, 2: 0.61}}, 
        -0.1: {'mean': 0.63, 'class': {1: 0.58, 2: 0.68}}, 
        -0.2: {'mean': 0.645, 'class': {1: 0.6, 2: 0.69}}, 
        -0.30000000000000004: {'mean': 0.6799999999999999, 'class': {1: 0.65, 2: 0.71}}, 
        -0.4: {'mean': 0.655, 'class': {1: 0.6, 2: 0.71}}, 
        -0.5: {'mean': 0.6399999999999999, 'class': {1: 0.59, 2: 0.69}}, 
        -0.6000000000000001: {'mean': 0.675, 'class': {1: 0.67, 2: 0.68}}, 
        -0.7000000000000001: {'mean': 0.675, 'class': {1: 0.65, 2: 0.7}}, 
        -0.8: {'mean': 0.755, 'class': {1: 0.79, 2: 0.72}}
        }
}

[dxoigmn]

Yeah, I think this makes sense. I do wonder, however, if a histogram of GIoUs might be more intuitive to interpret. The cocoapi does this, for example, with APs computed at different groundtruth object mask areas (see "AP Across Scales" https://cocodataset.org/#detection-eval):

In COCO, there are more small objects than large objects. Specifically: approximately 41% of objects are small (area < 322), 34% are medium (322 < area < 962), and 24% are large (area > 962). Area is measured as the number of pixels in the segmentation mask.

[jprokos26]

I'm getting an error when running the carla_obj_det_adversarialpatch_undefended.json config with the object_detection_AP_per_class_by_giou_from_patch task added:

final=lambda x: final_metric(*identity_zip(x), **final_kwargs),                                                                                                                                
             │  │             │            │     └ {}                                                                             
             │  │             │            └ [({'area': array([2603,  874, 2128, 1807, 1724, 8154, 1619, 3621, 2952,  247,  114,  
             │  │             │                      168,  103,  100,  103,  106,  100...                                         
             │  │             └ <function identity_zip at 0x7fbd3ab7db80>                                                         
             │  └ <function object_detection_AP_per_class_by_giou_from_patch at 0x7fbd2cc56dc0>                                   
             └ [({'area': array([2603,  874, 2128, 1807, 1724, 8154, 1619, 3621, 2952,  247,  114,                                
                       168,  103,  100,  103,  106,  100...                                                                       
TypeError: object_detection_AP_per_class_by_giou_from_patch() missing 1 required positional argument: 'y_patch_metadata_list'

Config used

{ "_description": "CARLA single modality object detection, contributed by MITRE Corporation", "adhoc": null, "attack": { "knowledge": "white", "kwargs": { "batch_size": 1, "learning_rate": 0.003, "max_iter": 1000, "optimizer": "pgd", "targeted": false, "verbose": true }, "module": "armory.art_experimental.attacks.carla_obj_det_adversarial_patch", "name": "CARLAAdversarialPatchPyTorch", "use_label": true }, "dataset": { "batch_size": 1, "eval_split": "test", "framework": "numpy", "modality": "rgb", "module": "armory.data.adversarial_datasets", "name": "carla_over_obj_det_test" }, "defense": null, "metric": { "means": true, "perturbation": "l0", "record_metric_per_sample": false, "task": [ "carla_od_AP_per_class", "carla_od_disappearance_rate", "carla_od_hallucinations_per_image", "carla_od_misclassification_rate", "carla_od_true_positive_rate", "object_detection_AP_per_class_by_giou_from_patch", "object_detection_mAP_tide" ] }, "model": { "fit": false, "fit_kwargs": {}, "model_kwargs": { "max_size": 1280, "min_size": 960, "num_classes": 3 }, "module": "armory.baseline_models.pytorch.carla_single_modality_object_detection_frcnn", "name": "get_art_model", "weights_file": "carla_rgb_weights_eval7and8.pt", "wrapper_kwargs": {} }, "scenario": { "export_batches": true, "kwargs": {}, "module": "armory.scenarios.carla_object_detection", "name": "CarlaObjectDetectionTask" }, "sysconfig": { "docker_image": "twosixarmory/armory", "external_github_repo": null, "gpus": "all", "output_dir": null, "output_filename": null, "use_gpu": false } }

[swsuggs]

@jprokos26 The automatic loading of metrics from the config does assume a certain argument set of y, y_pred, y_pred_adv. There's really no good way that I could see to list arbitrary metrics in the config and have the MetricsLogger pass the correct arguments. Since this metric also requires y_patch_metadata, I manually add the metric in the carla_object_detection scenario as you can see in the diff associated with this PR. Hence, there is no need to modify the config; it will use the metric automatically.

[jprokos26]

@swsuggs Understood, yes I see that behavior on my end; thank you for the clarification

[jprokos26]

After suggested changes getting the following using python3 -m armory utils plot-mAP-by-giou /home/jonathan.prokos/.armory/outputs/2023-08-07T182041.696236/CarlaObjectDetectionTask_1691432448.json --output ~/tmp.png. Note x11 forwarding works as well.

Should default size be changed to prevent overlapping titles?

[swsuggs]

@jprokos26

Should default size be changed to prevent overlapping titles?

I didn't set a default size since I envisioned mainly using show=True, i.e. not saving plots sight unseen. I suppose it would still be nice though. Would you care to add some default sizes based on the number of rows and columns?

[swsuggs]

@jprokos26 Ultimately the fig size isn't the root of the display issues. I took a whack at automating it based on number of columns and rows, but there continue to be elements that are too close together or too far apart. These could certainly be fixed as well, but I'd rather not hold up evals for it. We can address it later if needed.