carla OD dev set + attack integration #1182

lcadalzo · 2021-10-19T14:28:12Z

No description provided.

This reverts commit f566e0e.

…-od-dev-integration

lcadalzo · 2021-10-21T14:33:28Z

re-marking as WIP until we determine how to ensure non-CARLA classes are ignored by metric functions

…e considered

lcadalzo · 2021-10-21T21:36:42Z

@swsuggs I'm removing the WIP as this is now ready for review. My previous comment has been addressed

swsuggs

There's a lot here but as far as I can tell it looks good.

* update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b20. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: matt wartell <matt.wartell@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com>

* update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b20. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing * fix carla video attack (#1213) * update carla object detection patch attack to increase its efficacy (#1212) * adding databuilder for the test data for carla single and multi-modal… (#1211) * adding databuilder for the test data for carla single and multi-modality object detection scenarios * ran black * adding cached checksum file * update url * update url in code as well * add dset function * adding tests * updating docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update ci python (#1222) * update python 3.6 to 3.8 for ci * 3.7, not 3.8 * add CARLA multimodality object detection robust fusion model as the b… (#1217) * add CARLA multimodality object detection robust fusion model as the baseline defended model * adding test for new model Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * refactor video tracking attack: newly instantiate attack each generate() (#1223) * Dev carla video tracking test (#1219) * add CARLA video tracking test databuilder * ran black and flake8 * update urls * adding cached checksum file * adding test * add dataset fn * update docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * also measure performance using benign predictions as labels (#1225) * moving import inside method (#1228) * bump versions to 0.14.1 * python version 3.6 obsoleted by github * updating max_iter and LR (#1230) moved default parameters to something more meaningful as requested by performers * json formatting Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com>

* update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> * 0.13.3 (#1111) * bump art to 1.6.2 * tweak OD label format to comply with ART 1.6.2 * bump armory version to 0.13.3 * bump armory version to 0.13.3 (#1105) * bump armory version to 0.13.3 * bump version * prior to ART 1.6.2, installing ART also installed pandas; this appears to no longer be the case with 1.6.2 * adding ffmpeg-python package, since it's no longer installed with ART > 1.6.1 * use ground-truth labels for xview attack, now that this is supported in ART 1.6.2 * add ffmpeg-python to host-requirements too * bump art version to 1.6.2 (#1106) * bump art to 1.6.2 * tweak OD label format to comply with ART 1.6.2 * bump armory version to 0.13.3 * prior to ART 1.6.2, installing ART also installed pandas; this appears to no longer be the case with 1.6.2 * adding ffmpeg-python package, since it's no longer installed with ART > 1.6.1 * use ground-truth labels for xview attack, now that this is supported in ART 1.6.2 * add ffmpeg-python to host-requirements too * better describe no-docker install (#1112) Co-authored-by: matt wartell <matt.wartell@gmail.com> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * 0.13.4 (#1124) * updating scenario configs for eval 3 (#1123) * handling merge conflict * bump version 0.13.3 --> 0.13.4 * log that random patch location is being used Co-authored-by: Yusong Tan <ytan@mitre.org> * small updates for Eval 3 (#1126) * small updates for Eval 3 * updated kwargs and moved assert -> valueerror * lint Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> * Dev 0.13.5 (#1137) * abstain metric (#1132) * abstain metric * update doc and add test * Update scenarios md (#1136) * updating scenarios.md with Eval 3 results * formatting * formatting * formatting * added a note indicating DAPRICOT results are from the dev test data Co-authored-by: Yusong Tan <ytan@mitre.org> * Dev 0.13.5 poison (#1133) * nan instead of valueerror * fully instrumented gtsrb * fully instrumented resisc10 * minor update * clean label * Bug fixes suggested by Mike Tan and Kevin Eykholt. Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> * bug fix and updates (#1143) Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> * Version bump to 13.5 (#1146) * abstain metric (#1132) * abstain metric * update doc and add test * Update scenarios md (#1136) * updating scenarios.md with Eval 3 results * formatting * formatting * formatting * added a note indicating DAPRICOT results are from the dev test data Co-authored-by: Yusong Tan <ytan@mitre.org> * Dev 0.13.5 poison (#1133) * nan instead of valueerror * fully instrumented gtsrb * fully instrumented resisc10 * minor update * clean label * Bug fixes suggested by Mike Tan and Kevin Eykholt. Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> * bug fix and updates (#1143) * version bump * version bump in config files Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> * 0.14.0 release (#1204) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e3bb6844f8d7e650049a0869d2d578895b. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b208605715d1493f42241cfc540a2cbf108e. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: matt wartell <matt.wartell@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com> * Dev 0.14.1 (#1229) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e3bb6844f8d7e650049a0869d2d578895b. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b208605715d1493f42241cfc540a2cbf108e. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing * fix carla video attack (#1213) * update carla object detection patch attack to increase its efficacy (#1212) * adding databuilder for the test data for carla single and multi-modal… (#1211) * adding databuilder for the test data for carla single and multi-modality object detection scenarios * ran black * adding cached checksum file * update url * update url in code as well * add dset function * adding tests * updating docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update ci python (#1222) * update python 3.6 to 3.8 for ci * 3.7, not 3.8 * add CARLA multimodality object detection robust fusion model as the b… (#1217) * add CARLA multimodality object detection robust fusion model as the baseline defended model * adding test for new model Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * refactor video tracking attack: newly instantiate attack each generate() (#1223) * Dev carla video tracking test (#1219) * add CARLA video tracking test databuilder * ran black and flake8 * update urls * adding cached checksum file * adding test * add dataset fn * update docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * also measure performance using benign predictions as labels (#1225) * moving import inside method (#1228) * bump versions to 0.14.1 * python version 3.6 obsoleted by github * updating max_iter and LR (#1230) moved default parameters to something more meaningful as requested by performers * json formatting Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com> * R0.14.2 (#1231) * bump release version to 0.14.2 * update release version * obsolete dev branch by removing critical files * repoint CI build to develop branch * nuke 'em from orbit. it's the only way to be sure Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com>

* update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b20. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing * Initial commit to poisoning metrics update. * Fixed bugs, consolidated filter perplexity code. * Fixed attribute bug. * draft new poison metrics and associated interpretive model (#1226) * Sridevi's file: second metric using K-means on BEAN regularization models. * measure perplexity between benign class distribution and false positives distribution * First implementation of Statistical Parity Difference (SPD). * moving some perplexity code from poison.py to metrics.py * revise 'make_contingency_tables' to be more general * add function to convert subclass info to binary arrays * Fixed compute_spds signature. * Contingency table metric integration. * Fixed bug in a corner case. Deleted unused functions. * Cleaned up poisoning metric code. * Removed a line of testing code. * Fixed metrics 2.1/2.2 to use clean data. * Updated metrics with GTSRB integration. * add function to export arbitrary per-sample data * load explanatory model weights on appropriate device * update get_majority_mask functions to take/return majority_ceilings * sets up sample exporting, and computes metric 2.1 on the test set * fix potential divide by zero in filter perplexity computation * makes sure the whole test set is used for metric 2.1 computation * compute filter_perplexity in finalize_results() instead of in filter() * fix filter_perplexity so it doesn't crash with 0% poison * refactor lots of poison metric computation out of scenario code and into separate class. Also simplifies config usage * update explanatory model weight filenames * removing new_poisoning_metrics files, since the parts we needed are copied into utils/poisoning.py * update baseline poisoning scenario_configs to compute new metrics * de-obfuscate names of poisoning metrics (formerly Metric 2.1 and Metric 2.2, now Model Subclass Bias and Filter Subclass Bias) * update scenario docs with information about new poisoning metrics * minor update to comments * fix minor textual merge errors * formatting * check if filtering defense before applying filter metric * remove lines duplicated by merge * move global definition to top * align host-requirements with develop branch * remove lines duplicated by merge * remove more lines duplicated by merge * remove unused/outdated logging import * update logging * removing preloaded attack config since poison.py (L164) doesn't support that * force code-formatting test to use python 3.7 * pin click to fix black issue Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: matt wartell <matt.wartell@twosixtech.com> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixlabs.com>

copying in the attack mike sent

24ce750

lcadalzo mentioned this pull request Oct 19, 2021

Dev carla object detection attack #1159

Closed

lucas.cadalzo added 17 commits October 19, 2021 10:30

formatting

234b1a2

incorporating changes from pr 1173

f566e0e

Revert "incorporating changes from pr 1173"

2a38dae

This reverts commit f566e0e.

update new url

f1ce5c9

update checksum

95e759c

ignore black for this line

a5cc033

update url checksum

919692d

update url

99c2f0b

formatting

2d9b33b

tweaking attack to suit armory data format

94dbd18

adding preprocessing modality logic

51266a4

adding test for carla_obj_det_dev set

d3643b3

updating preprocessing for dev set

330a76a

update get_art_model assertion messages

5b56b3d

adding configs

fe1135c

add scenario

2d1ace4

formatting

3e464ae

lcadalzo changed the title ~~WIP: carla OD dev set + attack integration~~ carla OD dev set + attack integration Oct 20, 2021

lcadalzo requested review from swsuggs and removed request for swsuggs October 20, 2021 21:15

lcadalzo changed the title ~~carla OD dev set + attack integration~~ WIP: carla OD dev set + attack integration Oct 20, 2021

lucas.cadalzo added 4 commits October 20, 2021 17:24

upgrading ART since it's needed for OD attack; this will break CI

72ab15f

adding 4 new metrics for object detection

f2ac768

add test for new metric functions

bcdfc64

Merge branch 'dev' of https://github.com/twosixlabs/armory into carla…

bc6a773

…-od-dev-integration

lcadalzo requested a review from swsuggs October 21, 2021 14:02

lcadalzo changed the title ~~WIP: carla OD dev set + attack integration~~ carla OD dev set + attack integration Oct 21, 2021

lcadalzo changed the title ~~carla OD dev set + attack integration~~ WIP: carla OD dev set + attack integration Oct 21, 2021

lucas.cadalzo added 5 commits October 21, 2021 16:48

adding carla-specific metrics which ensure that only carla classes ar…

3e25a6b

…e considered

resolve merge conflict

722b56a

adding back what got accidentally deleted in last commit

8318ddc

formatting

1c4f2cf

formatting

7c14ff8

lcadalzo changed the title ~~WIP: carla OD dev set + attack integration~~ carla OD dev set + attack integration Oct 21, 2021

swsuggs and others added 7 commits October 22, 2021 10:03

refactor dataset kwarg loading

86458c7

updated dataset modality kwarg in configs

d3f8270

black

9921065

don't assume 'eval_split' exists in dataset_config

da094cc

reverting things to 7c14ff8

88cc450

rename metric and don't log % symbol

ee4dba9

enable export_sample for carla multimodal

891b0da

swsuggs approved these changes Oct 22, 2021

View reviewed changes

lcadalzo merged commit 6b6abc6 into twosixlabs:dev Oct 22, 2021

lcadalzo deleted the carla-od-dev-integration branch October 22, 2021 20:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

carla OD dev set + attack integration #1182

carla OD dev set + attack integration #1182

lcadalzo commented Oct 19, 2021

lcadalzo commented Oct 21, 2021

lcadalzo commented Oct 21, 2021

swsuggs left a comment

carla OD dev set + attack integration #1182

carla OD dev set + attack integration #1182

Conversation

lcadalzo commented Oct 19, 2021

lcadalzo commented Oct 21, 2021

lcadalzo commented Oct 21, 2021

swsuggs left a comment

Choose a reason for hiding this comment