bump art version #1062

lcadalzo · 2021-05-11T14:34:11Z

ng390

LGTM

* update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org>

* update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org>

* update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b20. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: matt wartell <matt.wartell@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com>

* update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b20. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing * fix carla video attack (#1213) * update carla object detection patch attack to increase its efficacy (#1212) * adding databuilder for the test data for carla single and multi-modal… (#1211) * adding databuilder for the test data for carla single and multi-modality object detection scenarios * ran black * adding cached checksum file * update url * update url in code as well * add dset function * adding tests * updating docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update ci python (#1222) * update python 3.6 to 3.8 for ci * 3.7, not 3.8 * add CARLA multimodality object detection robust fusion model as the b… (#1217) * add CARLA multimodality object detection robust fusion model as the baseline defended model * adding test for new model Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * refactor video tracking attack: newly instantiate attack each generate() (#1223) * Dev carla video tracking test (#1219) * add CARLA video tracking test databuilder * ran black and flake8 * update urls * adding cached checksum file * adding test * add dataset fn * update docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * also measure performance using benign predictions as labels (#1225) * moving import inside method (#1228) * bump versions to 0.14.1 * python version 3.6 obsoleted by github * updating max_iter and LR (#1230) moved default parameters to something more meaningful as requested by performers * json formatting Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com>

* update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> * 0.13.3 (#1111) * bump art to 1.6.2 * tweak OD label format to comply with ART 1.6.2 * bump armory version to 0.13.3 * bump armory version to 0.13.3 (#1105) * bump armory version to 0.13.3 * bump version * prior to ART 1.6.2, installing ART also installed pandas; this appears to no longer be the case with 1.6.2 * adding ffmpeg-python package, since it's no longer installed with ART > 1.6.1 * use ground-truth labels for xview attack, now that this is supported in ART 1.6.2 * add ffmpeg-python to host-requirements too * bump art version to 1.6.2 (#1106) * bump art to 1.6.2 * tweak OD label format to comply with ART 1.6.2 * bump armory version to 0.13.3 * prior to ART 1.6.2, installing ART also installed pandas; this appears to no longer be the case with 1.6.2 * adding ffmpeg-python package, since it's no longer installed with ART > 1.6.1 * use ground-truth labels for xview attack, now that this is supported in ART 1.6.2 * add ffmpeg-python to host-requirements too * better describe no-docker install (#1112) Co-authored-by: matt wartell <matt.wartell@gmail.com> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * 0.13.4 (#1124) * updating scenario configs for eval 3 (#1123) * handling merge conflict * bump version 0.13.3 --> 0.13.4 * log that random patch location is being used Co-authored-by: Yusong Tan <ytan@mitre.org> * small updates for Eval 3 (#1126) * small updates for Eval 3 * updated kwargs and moved assert -> valueerror * lint Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> * Dev 0.13.5 (#1137) * abstain metric (#1132) * abstain metric * update doc and add test * Update scenarios md (#1136) * updating scenarios.md with Eval 3 results * formatting * formatting * formatting * added a note indicating DAPRICOT results are from the dev test data Co-authored-by: Yusong Tan <ytan@mitre.org> * Dev 0.13.5 poison (#1133) * nan instead of valueerror * fully instrumented gtsrb * fully instrumented resisc10 * minor update * clean label * Bug fixes suggested by Mike Tan and Kevin Eykholt. Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> * bug fix and updates (#1143) Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> * Version bump to 13.5 (#1146) * abstain metric (#1132) * abstain metric * update doc and add test * Update scenarios md (#1136) * updating scenarios.md with Eval 3 results * formatting * formatting * formatting * added a note indicating DAPRICOT results are from the dev test data Co-authored-by: Yusong Tan <ytan@mitre.org> * Dev 0.13.5 poison (#1133) * nan instead of valueerror * fully instrumented gtsrb * fully instrumented resisc10 * minor update * clean label * Bug fixes suggested by Mike Tan and Kevin Eykholt. Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> * bug fix and updates (#1143) * version bump * version bump in config files Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> * 0.14.0 release (#1204) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e3bb6844f8d7e650049a0869d2d578895b. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b208605715d1493f42241cfc540a2cbf108e. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: matt wartell <matt.wartell@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com> * Dev 0.14.1 (#1229) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e3bb6844f8d7e650049a0869d2d578895b. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b208605715d1493f42241cfc540a2cbf108e. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing * fix carla video attack (#1213) * update carla object detection patch attack to increase its efficacy (#1212) * adding databuilder for the test data for carla single and multi-modal… (#1211) * adding databuilder for the test data for carla single and multi-modality object detection scenarios * ran black * adding cached checksum file * update url * update url in code as well * add dset function * adding tests * updating docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update ci python (#1222) * update python 3.6 to 3.8 for ci * 3.7, not 3.8 * add CARLA multimodality object detection robust fusion model as the b… (#1217) * add CARLA multimodality object detection robust fusion model as the baseline defended model * adding test for new model Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * refactor video tracking attack: newly instantiate attack each generate() (#1223) * Dev carla video tracking test (#1219) * add CARLA video tracking test databuilder * ran black and flake8 * update urls * adding cached checksum file * adding test * add dataset fn * update docs Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * also measure performance using benign predictions as labels (#1225) * moving import inside method (#1228) * bump versions to 0.14.1 * python version 3.6 obsoleted by github * updating max_iter and LR (#1230) moved default parameters to something more meaningful as requested by performers * json formatting Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com> * R0.14.2 (#1231) * bump release version to 0.14.2 * update release version * obsolete dev branch by removing critical files * repoint CI build to develop branch * nuke 'em from orbit. it's the only way to be sure Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: swsuggs <jsmitherson2@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com>

* update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach * Update dev version to 0.14.0 (#1084) * Update version * Update jsons * Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs * sweep attacks (#1071) * added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0 * Integrate tfds (#1061) * * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script * UCF config bug (#1092) * remove extra kwarg * formatting * Create tarfile with directory structure expected by armory (#1101) * Merging 13.2 to dev (#1109) * update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * eval-update smoke test (#1114) * existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> * pillow version bump (#1115) * Optimize Kenansville attack and fixes bug (#1113) * Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com> * Poison reimagined (#1117) * poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs * Add non-preloaded dirty-label backdoor attack with bullethole trigger (#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Dataset split tools for bullseye polytope attack (#1121) Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * fix object array issue (#1131) * merge r0.13.4 into dev (#1139) * merge r0.13.4 into dev A rather complex manual merge. There may well be extra, or unmodified scenario_configs * copied r0.13.4 configs and bumped container versions to 0.14.0 this was done to ensure congruence between the dev branch and the 6e90b37 merge this yielded 4 extra files which I'll remove in the next commit * removed extra scenario_configs from the r0.13.4 merge it should be pretty clear that these have been supplanted * adding back configs which use new dev feature Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Update README on ART (#1153) Signed-off-by: Beat Buesser <beat.buesser@ie.ibm.com> * updating RESISC-10 from 64x64 images to 256x256 images (#1155) * updating RESISC-10 from 64x64 images to 256x256 images * formatting * updated cached checksum file; modified datasets.py * update expected dataset shape in CI tests * updating docstring Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Train dataset builder for CARLA object detection scenarios (#1157) * Train dataset builder for CARLA object detection scenarios * update checksum file for train dataset * integrates carla train dataset. Note: throws error * integrates carla train dataset. * update to tfds 4.4.0 and modify affected python code accordingly * update host-requirements * renaming some functions to be more specific * going back to tfds 3.2 (undoing bb90ed2) * adding incomplete test for carla train set * slight modification to align with tfds 3.2; formatting * formatting, had to change my black version to that used by CI * update checksum again * yet another cached_checksum update * modifying host-requirements.txt Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * Dev dataset builder for CARLA object detection scenarios (#1156) * Dev dataset builder for CARLA object detection scenarios * changed split from 'train' to 'dev' * checksum file for dev dataset * updates to checksum * update URLS and added fix to be compatible with tfds 3.2 * adding dataset function for carla_obj_det_dev * adding cached checksum * to avoid flake8 error * enforce batch size of 1 * np squeezing label keys * minor bug fix to RGB and depth pairing * Update dataset version number Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * CARLA single modality object detection model (#1160) * rename to deconflict from carla multimodality object detection model * remove duplicate file Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * CARLA multimodality object detection model (#1161) * add carla multimodality object detection model * flake8 * update s3 object name; update version call to 1.0.1 (#1177) * minor bug fix and update checksum for final train dataset * black * update s3 object name; update version call to 1.0.1 * ignoring black since it's converting the string to a tuple? Co-authored-by: Yusong Tan <ytan@mitre.org> * update carla_obj_det_train cached checksum file (#1178) * update cached checksum file * just updated file permissions in s3, retriggering CI Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * fix dependency ordering in tf1 docker creation (#1179) * reorder pip to after conda install * add more packages to conda purview * repin python library versions as it happens, the installed version of these by the conda satisfier is the same as those pinned in this commit * pin to what resolver chose today when unpinned * enable variable_y=True even when variable_length is False (#1169) * enable variable_y=True even when variable_length is False * edit type hint * added shell script to run scenario configs in --check mode (#1167) Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * Update and fix carla obj det train dataset (#1173) * minor bug fix and update checksum for final train dataset * black * add label preprocessing for carla_obj_det_train * fix apparent typo ('pytorch' -> 'tensorflow') * carla_train dataset uses config kwarg to determine which modality of data to serve. * black * updated URL for dataset builder * fixed multimodal option for carla preprocessing * fix dictionary key problem by adding a default value * updates checksum files after corrected data annotations * new carla data preprocessing function * dataset test function asserts correct data shape depending on modality * black * carla dataset allows more flexible use of custom preprocessing functions * fix typo Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * update ART (#1181) * updated Dockerfiles as well as ART imports for 1.8 renamed modules * update KerasClassifier import * formatting * pinning numba to 0.53.1 * Video tracking integration (#1170) * full dev dataset for CARLA video tracking scenario * ran black and flake8 * baseline GOTURN model for CARLA video tracking scenario * art_experimental adversarial texture attack for CARLA video tracking scenario * integrating carla_video_tracking_dev, pushing progress * forgot to add these files to previous commit * adding cached checksum * adding test * pushing progress on added scenario, config, metric * typos and formatting * refactoring, define pred format, point to weights file; can now run --skip-attack w/o error * to comply with ART, refactor label format to mirror pred format; got attack working * renaming config * formatting * adding updated tf1 dockerfile to fix ci tests * update tests to reflect label refactor * adding test for carla video tracking model * remove unused variables * update pytorch Dockerfile to use newer ART * download external_repo in video_tracking test Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> * moving cv2 import inside fn (#1189) * adding ci tests for baseline models (#1188) * adding ci tests for baseline models * deleting line that was accidentally pushed * carla OD dev set + attack integration (#1182) * copying in the attack mike sent * formatting * incorporating changes from pr 1173 * Revert "incorporating changes from pr 1173" This reverts commit f566e0e. * update new url * update checksum * ignore black for this line * update url checksum * update url * formatting * tweaking attack to suit armory data format * adding preprocessing modality logic * adding test for carla_obj_det_dev set * updating preprocessing for dev set * update get_art_model assertion messages * adding configs * add scenario * formatting * upgrading ART since it's needed for OD attack; this will break CI * adding 4 new metrics for object detection * add test for new metric functions * adding carla-specific metrics which ensure that only carla classes are considered * adding back what got accidentally deleted in last commit * formatting * formatting * refactor dataset kwarg loading * updated dataset modality kwarg in configs * black * don't assume 'eval_split' exists in dataset_config * reverting things to 7c14ff8 * rename metric and don't log % symbol * enable export_sample for carla multimodal Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> * Refactor dataset config loading (#1194) * refactor dataset config loading * update carla configs for new dataset config loading * refactor how check_run is passed through, so it doesnt get passed to the tfds ds function * formatting Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> * index and class filtering from command line; also doc update (#1162) * don't use y_pred as generate() y kwarg (#1190) * give generate() an optional y to comply with api every other ART attack uses * Revert "give generate() an optional y to comply with api every other ART attack uses" This reverts commit 8884b20. * give kenansville a y kwarg; dont have default scenario set y kwarg to y_pred * don't use y_pred when use_label is false * deleting comment * flake8 * train_split kwarg shouldnt be passed along to ds function (#1198) * disable filter by class for carla datasets (#1197) * adding frame rate fixes issue (#1195) * first check if y is numpy array before checking dtype (#1196) * first check if y is numpy array before checking dtype * refactor * Make metric kwargs configurable (#1187) * make metric kwargs configurable * removing new code that wasn't meant for this PR * set targeted to whatever the attack is actually using (#1201) * set targeted to whatever the attack is actually using * slight refactor * WIP: updating docs (#1199) * updating docs * copying over scenarios.md from 0.13.5 which never got merged back into dev * adding carla scenarios * adding a note on how to specify metric kwargs * addressing comments * update dataset licensing * Initial commit to poisoning metrics update. * Fixed bugs, consolidated filter perplexity code. * Fixed attribute bug. * draft new poison metrics and associated interpretive model (#1226) * Sridevi's file: second metric using K-means on BEAN regularization models. * measure perplexity between benign class distribution and false positives distribution * First implementation of Statistical Parity Difference (SPD). * moving some perplexity code from poison.py to metrics.py * revise 'make_contingency_tables' to be more general * add function to convert subclass info to binary arrays * Fixed compute_spds signature. * Contingency table metric integration. * Fixed bug in a corner case. Deleted unused functions. * Cleaned up poisoning metric code. * Removed a line of testing code. * Fixed metrics 2.1/2.2 to use clean data. * Updated metrics with GTSRB integration. * add function to export arbitrary per-sample data * load explanatory model weights on appropriate device * update get_majority_mask functions to take/return majority_ceilings * sets up sample exporting, and computes metric 2.1 on the test set * fix potential divide by zero in filter perplexity computation * makes sure the whole test set is used for metric 2.1 computation * compute filter_perplexity in finalize_results() instead of in filter() * fix filter_perplexity so it doesn't crash with 0% poison * refactor lots of poison metric computation out of scenario code and into separate class. Also simplifies config usage * update explanatory model weight filenames * removing new_poisoning_metrics files, since the parts we needed are copied into utils/poisoning.py * update baseline poisoning scenario_configs to compute new metrics * de-obfuscate names of poisoning metrics (formerly Metric 2.1 and Metric 2.2, now Model Subclass Bias and Filter Subclass Bias) * update scenario docs with information about new poisoning metrics * minor update to comments * fix minor textual merge errors * formatting * check if filtering defense before applying filter metric * remove lines duplicated by merge * move global definition to top * align host-requirements with develop branch * remove lines duplicated by merge * remove more lines duplicated by merge * remove unused/outdated logging import * update logging * removing preloaded attack config since poison.py (L164) doesn't support that * force code-formatting test to use python 3.7 * pin click to fix black issue Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> Co-authored-by: matt wartell <matt.wartell@twosixlabs.com> Co-authored-by: Guillaume Leclerc <guillaume.leclerc.work@gmail.com> Co-authored-by: ng390 <gupta.neal@gmail.com> Co-authored-by: lucas.cadalzo <lucas.cadalzo@twosixlabs.com> Co-authored-by: Beat Buesser <49047826+beat-buesser@users.noreply.github.com> Co-authored-by: Sterling Suggs <sterling.suggs@twosixtech.com> Co-authored-by: lcadalzo <lucas.cadalzo@twosixtech.com> Co-authored-by: matt wartell <matt.wartell@twosixtech.com> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixtech.com> Co-authored-by: Reed Gordon-Sarney <reed.gordon-sarney@twosixlabs.com>

bumping ART 1.6.0 --> 1.6.1

c65bd59

lcadalzo linked an issue May 11, 2021 that may be closed by this pull request

Bump ART to 1.6.1 #1054

Closed

ng390 approved these changes May 11, 2021

View reviewed changes

ng390 merged commit a647f96 into twosixlabs:dev May 11, 2021

ng390 mentioned this pull request May 11, 2021

Bump ART to 1.6.1 #1054

Closed

lcadalzo deleted the 1054-bump-art-version branch December 22, 2021 22:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bump art version #1062

bump art version #1062

lcadalzo commented May 11, 2021

ng390 left a comment

bump art version #1062

bump art version #1062

Conversation

lcadalzo commented May 11, 2021

ng390 left a comment

Choose a reason for hiding this comment