Merge poisoning branch into develop #1360

swsuggs · 2022-03-23T16:15:20Z

* update version * update json version

* update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com>

* update build script * added command echoes

* Add CIFAR100 dataset * Typo

* renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs'

* refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format

…1063)

* * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing

* call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8

* adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org>

* * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com>

* Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach

* Update version * Update jsons

…fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs

* added SweepAttack functionality * adding docs * adding docs for attack type field * adding clarification to docs * improved logging for how attack success is measured * specify possible values for attack type and throw warning if unexpected value * added mAP function which returns scalar value instead of dict returned by object_detection_AP_per_class() * update metric and max_iter of xview sweep config * refactor how metrics are computed for SweepAttack; enforce that returned value is scalar * set record_metric_per_sample true; add a note on this in docs * update mkdocs.yml * removing unused type field from poisoning configs * adding clarification about what the attack returns * consistent log prefix at end of generate() regardless of failure/success * update sweep configs to 0.14.0

* * TFDS integration script * Move S3 upload tool to main repo from armory-private * Fail fast, indentation, fix upload typo * Update dataset docs * Improved code organization * Update template to include all parameters (except indexing params) * Update docs * Remove args typically passed through **kwargs * More logical step numbering * Add ref to docs in script

* remove extra kwarg * formatting

* update version * revert version * 0.13.1 release (#1068) * update version (#1034) * update version * update json version * set channels_first False for relevant pytorch models (#1037) * Resisc10 poison dataset (#1038) * update version * revert version * added resisc10 poison dataset * Update refs to point to S3, add cached dataset * Add test for resisc10 dataset Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Build tag script (#1035) * update build script * added command echoes * pinning to numpy 1.19.2 to avoid ART error (#1056) * updating comment on relevant np issue (#1057) * CIFAR-100 dataset (#1048) * Add CIFAR100 dataset * Typo * label targeter refactor (#1052) * renamed file * fix typo while remaining backwards compatible * refactored label targeter config loading logic * updating configs accordingly * adding one more config * changing filename back to labels.py * adding warning message for deprecated 'scheme' key * removing code that shouldn't have been pushed/fixing typo * update configs for label_targeters.py --> labels.py change * removing configs i didn't meant to push * keyword-only args; change config 'args' --> 'kwargs' * refactor object detection metrics (#1046) * refactored object_detection_AP_per_class * refactor dapricot and apricot AP functions * update tests for od metrics refactor * removing od metrics that aren't useful * modify od format check function; renamed a couple variables * refactor to remove unnecessary elifs; rename append() to add_results() * formatting * renamed method * document function input format * bumping ART 1.6.0 --> 1.6.1 (#1062) * updating baseline config to be compatible with newer versions of ART (#1063) * don't assume default branch is named master (#1064) * Poisoning scenario with blended trigger (#1049) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * Use armory.__file__ to simplify relative pathing * preprocessing defense fixes (#1060) * call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8 * dapricot updates (#1040) * adjust scale for insert_patch(); make patch shape square * force dapricot attacks to be targeted * formatting * increment label index in loss_gradient for baseline 0-indexed model * need to decrement not increment * adding dapricot_patch_target_success metric * resetting this variable to empty list since dparicot has no nontargeted tasks * this workaround is no longer necessary per previous commit * deleting commented out code that was accidentally pushed * removing config since DPatch doesn't support targeted attack yet * formatting * reshape box to flat array * add docs for fn input format * formatting * updated dapricot RobustDPatch attack and associated files * ran black, flake8, and format_json * adding targeted Dpatch to file itself so we dont need to use dev version of ART * minor documentation/error msg update * removing channels_first logic since x will always be channels_last with armory * black formatting * adding clarifying comment * set num_images_per_patch in scenario code; force threat model to be specified in scenario code * minor modifications to error messages * dont overwrite model kwargs; add 'batch_size' kwarg to baseline models get_art_model() * add warning if batch_size model_kwarg isnt set; also edited comment at top of script * removing unused line of code * removing code that has no effect on attack * avoid warning message by renaming colour fn to its updated name * set check on lower bound of brightness range * fix typo * point to armory 0.13.1 in config * point to armory 0.13.1 in pgd config too * only display warning for physical attacks * flake8 * the code in this file was moved to inside the attack * removing dapricot robust dpatch attack and associated utility functions * flake8 Co-authored-by: Yusong Tan <ytan@mitre.org> * Resisc10 poison (#1065) * * Update image-based trigger to allow blending * Use blended trigger to enable bullethole clbd attack * Update docker image reference in config * Update pathing to load image path when armory is pip installed * resisc10 poison scenario related files * Updated poisoning attack call based on ART updates, fix channel ordering for image data * Update metrics method names * Update config to work with pip-installed armory Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> * Poisoning scenario Pytorch example (#1067) * Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach Co-authored-by: davidslater <david.slater@twosixlabs.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com> Co-authored-by: Yusong Tan <ytan@mitre.org> * Update dockerfile for tf1 (#1086) * 0.13.2 (#1102) * Increment version to 0.13.2 (#1095) * Bump version * Update configs * dapricot test set (#1096) * cherry-picked dapricot test commits from 1088 * correct checksum filename * Coco (#1097) * cherry-picking commits from 1085, excluding the commit merging in dev branch * adding coco tests, skipping if not available locally * adding note to docs about apricot class indexing * updated checksum after new upload to s3 Co-authored-by: ng390 <neal.gupta@twosixlabs.com> Co-authored-by: David Slater <david.slater@twosixlabs.com> Co-authored-by: lcadalzo <39925313+lcadalzo@users.noreply.github.com> Co-authored-by: yusong-tan <59029053+yusong-tan@users.noreply.github.com> Co-authored-by: Yusong Tan <ytan@mitre.org>

* existing updates * updated evasion scenarios * update * dapricot update * so2sat update * poisoning * scenario updates * remove base * typedef hint for JSON-like config dict * add jupyter text * typehints and docstrings * avoid name error if attack_type is preloaded * unbound local errors * calls via super have implied self * self reference removed * torchvision is back-versioned * typo metrics for metric * align torchvision version with pytorch version as prescribed by https://pypi.org/project/torchvision/ * black19.10b0 and flake8 compliant * update workflow * forgot to push latest commit * name changes * updated names * simplify * simplification * update ART api usage Co-authored-by: matt wartell <matt.wartell@twosixlabs.com>

* Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com>

* poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs

…#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com>

Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com>

…ic 2.2, now Model Subclass Bias and Filter Subclass Bias)

mwartell

I looked only at things that were visibly wrong in this. I will fix the things I flagged.

This PR still needs to be reviewed by our local poisoning expert.

armory/art_experimental/attacks/pgd_patch.py

armory/art_experimental/attacks/robust_dpatch.py

armory/art_experimental/attacks/sweep.py

armory/baseline_models/pytorch/micronnet_gtsrb_bean_regularization.py

davidslater

I did a light review, but I haven't run anything. A couple of merge issues, but overall looks pretty good.

armory/__init__.py

armory/baseline_models/pytorch/micronnet_gtsrb_bean_regularization.py

armory/scenarios/poison.py

host-requirements.txt

davidslater

Another small change. I will test by running the code now.

armory/__main__.py

armory/baseline_models/pytorch/micronnet_gtsrb_bean_regularization.py

armory/baseline_models/pytorch/resnet18_bean_regularization.py

davidslater · 2022-03-28T17:07:24Z

When running armory run scenario_configs/gtsrb_scenario_poison.json --gpus=6, I get this error:

2022-03-28 17:05:14 35s ERROR    armory.scenarios.scenario:evaluate:361 Encountered error during scenario evaluation.                                                         [3/182]
Traceback (most recent call last):                                                                                                                                                   
File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,                                                                                                                                           
│         │     └ {'__name__': '__main__', '__doc__': '\nMain script for running scenarios. Users will run a scenario by calling:\n    armory r...
│         └ <code object <module> at 0x7f9d92248710, file "/workspace/armory/scenarios/main.py", line 1>
└ <function _run_code at 0x7f9ed9cf2d30>
File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
│     └ {'__name__': '__main__', '__doc__': '\nMain script for running scenarios. Users will run a scenario by calling:\n    armory r...
└ <code object <module> at 0x7f9d92248710, file "/workspace/armory/scenarios/main.py", line 1>
File "/workspace/armory/scenarios/main.py", line 226, in <module>
run_config(
└ <function run_config at 0x7f9d911b6d30>
File "/workspace/armory/scenarios/main.py", line 144, in run_config
scenario.evaluate()
│        └ <function Scenario.evaluate at 0x7f9d90dd4310>
└ <armory.scenarios.poisoning_gtsrb_scenario.GTSRB object at 0x7f9ed9206580>
> File "/workspace/armory/scenarios/scenario.py", line 352, in evaluate
results = self._evaluate()
          │    └ <function Scenario._evaluate at 0x7f9d90dd4280>
          └ <armory.scenarios.poisoning_gtsrb_scenario.GTSRB object at 0x7f9ed9206580>
File "/workspace/armory/scenarios/scenario.py", line 342, in _evaluate
self.load()
│    └ <function Poison.load at 0x7f9d90dd4f70>
└ <armory.scenarios.poisoning_gtsrb_scenario.GTSRB object at 0x7f9ed9206580>
File "/workspace/armory/scenarios/poison.py", line 320, in load
self.load_poisoner()
│    └ <function Poison.load_poisoner at 0x7f9d90dd4b80>
└ <armory.scenarios.poisoning_gtsrb_scenario.GTSRB object at 0x7f9ed9206580>
File "/workspace/armory/scenarios/poison.py", line 164, in load_poisoner
raise ValueError("preloaded attacks not currently supported for poisoning")
ValueError: preloaded attacks not currently supported for poisoning

davidslater

A couple additional logging changes (these are due to the change from the built-in logger to loguru)

armory/utils/poisoning.py

…rt that

davidslater

Great, that worked!

Good to merge in once the remaining checks pass.

davidslater and others added 30 commits March 25, 2021 17:25

update version (#1034)

19458b3

* update version * update json version

set channels_first False for relevant pytorch models (#1037)

f1f754f

Build tag script (#1035)

903a7c9

* update build script * added command echoes

pinning to numpy 1.19.2 to avoid ART error (#1056)

9770caa

updating comment on relevant np issue (#1057)

b15941c

CIFAR-100 dataset (#1048)

37d455c

* Add CIFAR100 dataset * Typo

bumping ART 1.6.0 --> 1.6.1 (#1062)

a647f96

updating baseline config to be compatible with newer versions of ART (#…

2806e87

…1063)

don't assume default branch is named master (#1064)

7136887

preprocessing defense fixes (#1060)

f964595

* call set_params() so classifier.all_framework_preprocessing attribute is updated * no longer using kwarg which ART has removed * use get_params() to append defenses; removed if ART < 1.5 logic * flake8

Poisoning scenario Pytorch example (#1067)

fc87d21

* Pytorch compatibility for poisoning scenarios, example Pytorch config for dlbd * Configs closer to eval approach

Update dev version to 0.14.0 (#1084)

7ac7b18

* Update version * Update jsons

Hotfix: Docker tf1 fix to allow tensorflow.keras to load h5 weights (…

3bc0041

…fixes CI testing) (#1080) * Update dockerfile for tf1, temporary logging to check need for fix * Remove logging/group pip installs

UCF config bug (#1092)

15418dc

* remove extra kwarg * formatting

Create tarfile with directory structure expected by armory (#1101)

3784f60

pillow version bump (#1115)

2defb34

Optimize Kenansville attack and fixes bug (#1113)

c961bb4

* Optimize Kenansville attack and fixes bug Resolves #1103 Was tested outside of Armory * lint * update with rfft * update with rfft * length mismatch Co-authored-by: David Slater <david.slater@twosixlabs.com>

Poison reimagined (#1117)

8c774d8

* poison update * update to new names * nit * even more nit * match scenario * use * dataset kwargs

Add non-preloaded dirty-label backdoor attack with bullethole trigger (…

71f280f

…#1120) * Add non-preloaded dirty-label backdoor attack with bullethole trigger * Fix docker image version Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com>

Dataset split tools for bullseye polytope attack (#1121)

d8bae9b

Co-authored-by: Neal Gupta <neal.gupta@twosixlabs.com>

swsuggs and others added 5 commits March 23, 2022 11:43

update baseline poisoning scenario_configs to compute new metrics

07ee7ac

de-obfuscate names of poisoning metrics (formerly Metric 2.1 and Metr…

e811fac

…ic 2.2, now Model Subclass Bias and Filter Subclass Bias)

update scenario docs with information about new poisoning metrics

9f73ee6

minor update to comments

10863af

Merge branch 'develop' into dev_0.14.1_poisoning

10ed3a9

mwartell requested changes Mar 24, 2022

View reviewed changes

mwartell and others added 3 commits March 24, 2022 16:20

fix minor textual merge errors

14f622e

Merge branch 'develop' into dev_0.14.1_poisoning

a3e7c9a

formatting

febdcf4

davidslater suggested changes Mar 28, 2022

View reviewed changes

armory/__init__.py Outdated Show resolved Hide resolved

armory/baseline_models/pytorch/micronnet_gtsrb_bean_regularization.py Outdated Show resolved Hide resolved

armory/scenarios/poison.py Show resolved Hide resolved

host-requirements.txt Outdated Show resolved Hide resolved

swsuggs added 4 commits March 28, 2022 09:07

check if filtering defense before applying filter metric

45f633e

remove lines duplicated by merge

e243610

move global definition to top

ec6e379

align host-requirements with develop branch

ba5c548

davidslater suggested changes Mar 28, 2022

View reviewed changes

armory/__main__.py Outdated Show resolved Hide resolved

remove lines duplicated by merge

976c8ef

swsuggs changed the title ~~WIP: Merge poisoning branch into develop~~ Merge poisoning branch into develop Mar 28, 2022

davidslater suggested changes Mar 28, 2022

View reviewed changes

armory/baseline_models/pytorch/micronnet_gtsrb_bean_regularization.py Outdated Show resolved Hide resolved

armory/baseline_models/pytorch/resnet18_bean_regularization.py Outdated Show resolved Hide resolved

remove more lines duplicated by merge

1565f88

davidslater suggested changes Mar 28, 2022

View reviewed changes

armory/utils/poisoning.py Outdated Show resolved Hide resolved

armory/utils/poisoning.py Outdated Show resolved Hide resolved

swsuggs added 5 commits March 28, 2022 10:19

remove unused/outdated logging import

833935d

update logging

19946af

removing preloaded attack config since poison.py (L164) doesn't suppo…

126bf94

…rt that

force code-formatting test to use python 3.7

bc2c0b3

pin click to fix black issue

f95ea54

davidslater approved these changes Mar 28, 2022

View reviewed changes

davidslater merged commit 4da43ca into develop Mar 28, 2022

swsuggs deleted the dev_0.14.1_poisoning branch March 28, 2022 20:04

swsuggs mentioned this pull request Mar 30, 2022

Add poisoning scenario metrics #1154

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge poisoning branch into develop #1360

Merge poisoning branch into develop #1360

swsuggs commented Mar 23, 2022 •

edited by davidslater

Loading

mwartell left a comment

davidslater left a comment

davidslater left a comment

davidslater commented Mar 28, 2022

davidslater left a comment

davidslater left a comment

Merge poisoning branch into develop #1360

Merge poisoning branch into develop #1360

Conversation

swsuggs commented Mar 23, 2022 • edited by davidslater Loading

mwartell left a comment

Choose a reason for hiding this comment

davidslater left a comment

Choose a reason for hiding this comment

davidslater left a comment

Choose a reason for hiding this comment

davidslater commented Mar 28, 2022

davidslater left a comment

Choose a reason for hiding this comment

davidslater left a comment

Choose a reason for hiding this comment

swsuggs commented Mar 23, 2022 •

edited by davidslater

Loading