Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OD poison update #1990

Merged
merged 1 commit into from
Sep 22, 2023
Merged

OD poison update #1990

merged 1 commit into from
Sep 22, 2023

Conversation

swsuggs
Copy link
Contributor

@swsuggs swsuggs commented Aug 31, 2023

ART has corrected an off-by-1 situation so it no longer needs to be accounted for here.

@swsuggs swsuggs requested a review from jprokos26 September 21, 2023 15:30
Copy link
Contributor

@jprokos26 jprokos26 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with scenario_configs/eval7/poisoning/obj_det_dlbd_OGA.json

Output

2023-09-21 15:51:20 8m35s INFO     armory.instrument.instrument:_write:709 Meter Record: name=AP_per_class_on_benign_test_data, batch=None, result={'mean': 0.08666666666666667, 'class': {0: 0.01, 1: 0.11, 2: 0.14}}
2023-09-21 15:51:20 8m35s INFO     armory.instrument.instrument:_write:709 Meter Record: name=AP_per_class_on_adv_test_data_with_poison_labels, batch=None, result={'mean': 0.12666666666666668, 'class': {0: 0.01, 1: 0.26, 2: 0.11}}
2023-09-21 15:51:21 8m35s INFO     armory.instrument.instrument:_write:709 Meter Record: name=mAP_tide_on_benign_test_data, batch=None, result={'mAP': {0.5: 0.06, 0.55: 0.05, 0.6: 0.03, 0.65: 0.01, 0.7: 0.01, 0.75: 0.0, 0.8: 0.0, 0.85: 0.0, 0.9: 0.0, 0.95: 0.0}, 'errors': {'main': {'dAP': {'Cls': 0.0029, 'Loc': 0.1038, 'Both': 0.0033, 'Dupe': 0.0035, 'Bkg': 0.0046, 'Miss': 0.1206}, 'count': {'Cls': 5, 'Loc': 191, 'Both': 34, 'Dupe': 38, 'Bkg': 40, 'Miss': 456}}, 'special': {'dAP': {'FalsePos': 0.0802, 'FalseNeg': 0.2641}, 'count': {'FalseNeg': 524}}}}
2023-09-21 15:51:22 8m36s INFO     armory.instrument.instrument:_write:709 Meter Record: name=mAP_tide_on_adv_test_data_with_poison_labels, batch=None, result={'mAP': {0.5: 0.1, 0.55: 0.06, 0.6: 0.03, 0.65: 0.01, 0.7: 0.0, 0.75: 0.0, 0.8: 0.0, 0.85: 0.0, 0.9: 0.0, 0.95: 0.0}, 'errors': {'main': {'dAP': {'Cls': 0.0015, 'Loc': 0.2054, 'Both': 0.0013, 'Dupe': 0.0043, 'Bkg': 0.0011, 'Miss': 0.1011}, 'count': {'Cls': 10, 'Loc': 916, 'Both': 39, 'Dupe': 72, 'Bkg': 51, 'Miss': 485}}, 'special': {'dAP': {'FalsePos': 0.1336, 'FalseNeg': 0.2166}, 'count': {'FalseNeg': 670}}}}
2023-09-21 15:51:22 8m36s INFO     armory.instrument.instrument:_write:709 Meter Record: name=attack_success_rate_generation, batch=None, result=0.6774193548387096
2023-09-21 15:51:22 8m36s INFO     armory.instrument.instrument:_write:709 Meter Record: name=attack_success_rate_generation, batch=None, result=0.6774193548387096
2023-09-21 15:51:22 8m36s INFO     armory.scenarios.scenario:save:489 Saving evaluation results to path /armory/outputs/2023-09-21T154244.442657/ObjectDetectionPoisoningScenario_1695310975.json inside container.
Scenario has finished running cleanly
2023-09-21 15:51:26 8m42s SUCCESS  armory.docker.management:exec_cmd:102 command exited cleanly

@swsuggs swsuggs merged commit c3353e5 into twosixlabs:develop Sep 22, 2023
10 checks passed
@swsuggs swsuggs deleted the od-poison-update branch September 22, 2023 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants