-
Notifications
You must be signed in to change notification settings - Fork 129
Basic Use
Gary edited this page Nov 14, 2022
·
7 revisions
After starting Counterfit you will be greeted with a simple interface,
[sarah@contoso.com] -> counterfit 1 ↵ gary@red-team-shared-vm
__ _____ __
_________ __ ______ / /____ _____/ __(_) /_
/ ___/ __ \/ / / / __ \/ __/ _ \/ ___/ /_/ / __/
/ /__/ /_/ / /_/ / / / / /_/ __/ / / __/ / /
\___/\____/\__,_/_/ /_/\__/\___/_/ /_/ /_/\__/
Version: 1.1.0
counterfit>
To view the available targets execute the list targets command. Targets are user created classes that represent a prediction endpoint. Learn more about Targets.
counterfit> list targets
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ Model Type ┃ Data Type ┃ Input Shape ┃ # Samples ┃ Endpoint ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ cart_pole │ closed-box │ tabular │ (1080000,) │ 0 │ cartpole_dqn_10000.pt.gz │
│ cart_pole_initstate │ closed-box │ tabular │ (4,) │ 0 │ cartpole_dqn_10000.pt.gz │
│ creditfraud │ closed-box │ tabular │ (30,) │ 0 │ creditfraud/creditfraud_sklearn_pipeline.pkl │
│ digits_keras │ closed-box │ image │ (28, 28, 1) │ 0 │ digits_keras/mnist_model.h5 │
│ digits_mlp │ closed-box │ image │ (1, 28, 28) │ 0 │ digits_mlp/mnist_sklearn_pipeline.pkl │
│ movie_reviews │ closed-box │ text │ (1,) │ 0 │ movie_reviews/movie_reviews_sentiment_analysis.pt │
│ satellite │ closed-box │ image │ (3, 256, 256) │ 0 │ satellite/satellite-image-params-airplane-stadium.h5 │
└─────────────────────┴────────────┴───────────┴───────────────┴───────────┴──────────────────────────────────────────────────────┘
counterfit>
To view the available attacks execute the list attacks command. Internally, Counterfit will load the framework automatically. Learn more about Frameworks.
counterfit> list attacks
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Name ┃ Category ┃ Type ┃ Tags ┃ Framework ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ black_box_rule_based │ inference │ closed-box │ │ art │
│ boundary │ evasion │ closed-box │ image, tabular │ art │
│ carlini │ evasion │ open-box │ image, tabular │ art │
│ copycat_cnn │ inversion │ closed-box │ image │ art │
│ deepfool │ evasion │ open-box │ image, tabular │ art │
│ elastic_net │ evasion │ open-box │ image, tabular │ art │
│ functionally_equivalent_extraction │ inversion │ closed-box │ image, tabular │ art │
│ hop_skip_jump │ evasion │ closed-box │ image, tabular │ art │
│ knockoff_nets │ inversion │ closed-box │ image, tabular │ art │
│ label_only_boundary_distance │ inference │ open-box │ image, tabular │ art │
│ mi_face │ inference │ open-box │ image, tabular │ art │
│ newtonfool │ evasion │ open-box │ image, tabular │ art │
│ pixel_threshold │ evasion │ unknown │ image │ art │
│ projected_gradient_descent_numpy │ evasion │ open-box │ image, tabular │ art │
│ saliency_map │ evasion │ open-box │ image, tabular │ art │
│ simba │ evasion │ open-box │ image │ art │
│ spatial_transformation │ evasion │ open-box │ image, tabular │ art │
│ universal_perturbation │ evasion │ open-box │ image │ art │
│ virtual_adversarial │ evasion │ open-box │ image │ art │
│ wasserstein │ evasion │ open-box │ image │ art │
│ white_box_decision_tree │ inference │ unknown │ │ art │
│ ApplyLambda │ common-corruption │ closed-box │ image │ augly │
│ Blur │ common-corruption │ closed-box │ image │ augly │
│ Brightness │ common-corruption │ closed-box │ image │ augly │
│ ChangeAspectRatio │ common-corruption │ closed-box │ image │ augly │
│ ClipImageSize │ common-corruption │ closed-box │ image │ augly │
│ ColorJitter │ common-corruption │ closed-box │ image │ augly │
│ Contrast │ common-corruption │ closed-box │ image │ augly │
│ ConvertColor │ common-corruption │ closed-box │ image │ augly │
│ Crop │ common-corruption │ closed-box │ image │ augly │
│ EncodingQuality │ common-corruption │ closed-box │ image │ augly │
│ Grayscale │ common-corruption │ closed-box │ image │ augly │
│ HFlip │ common-corruption │ closed-box │ image │ augly │
│ MemeFormat │ common-corruption │ closed-box │ image │ augly │
│ Opacity │ common-corruption │ closed-box │ image │ augly │
│ OverlayEmoji │ common-corruption │ closed-box │ image │ augly │
│ OverlayOntoScreenshot │ common-corruption │ closed-box │ image │ augly │
│ OverlayStripes │ common-corruption │ closed-box │ image │ augly │
│ OverlayText │ common-corruption │ closed-box │ image │ augly │
│ Pad │ common-corruption │ closed-box │ image │ augly │
│ PadSquare │ common-corruption │ closed-box │ image │ augly │
│ PerspectiveTransform │ common-corruption │ closed-box │ image │ augly │
│ Pixelization │ common-corruption │ closed-box │ image │ augly │
│ RandomEmojiOverlay │ common-corruption │ closed-box │ image │ augly │
│ RandomNoise │ common-corruption │ closed-box │ image │ augly │
│ Resize │ common-corruption │ closed-box │ image │ augly │
│ Rotate │ common-corruption │ closed-box │ image │ augly │
│ Saturation │ common-corruption │ closed-box │ image │ augly │
│ Scale │ common-corruption │ closed-box │ image │ augly │
│ Sharpen │ common-corruption │ closed-box │ image │ augly │
│ ShufflePixels │ common-corruption │ closed-box │ image │ augly │
│ VFlip │ common-corruption │ closed-box │ image │ augly │
│ a2t_yoo_2021 │ evasion │ closed-box │ text │ textattack │
│ bae_garg_2019 │ evasion │ closed-box │ text │ textattack │
│ bert_attack_li_2020 │ evasion │ closed-box │ text │ textattack │
│ checklist_ribeiro_2020 │ evasion │ closed-box │ text │ textattack │
│ clare_li_2020 │ evasion │ closed-box │ text │ textattack │
│ deepwordbug_gao_2018 │ evasion │ closed-box │ text │ textattack │
│ faster_genetic_algorithm_jia_2019 │ evasion │ closed-box │ text │ textattack │
│ genetic_algorithm_alzantot_2018 │ evasion │ closed-box │ text │ textattack │
│ hotflip_ebrahimi_2017 │ evasion │ closed-box │ text │ textattack │
│ iga_wang_2019 │ evasion │ closed-box │ text │ textattack │
│ input_reduction_feng_2018 │ evasion │ closed-box │ text │ textattack │
│ kuleshov_2017 │ evasion │ closed-box │ text │ textattack │
│ morpheus_tan_2020 │ evasion │ closed-box │ text │ textattack │
│ pruthi_2019 │ evasion │ closed-box │ text │ textattack │
│ pso_zang_2020 │ evasion │ closed-box │ text │ textattack │
│ pwws_ren_2019 │ evasion │ closed-box │ text │ textattack │
│ seq2sick_cheng_2018_blackbox │ evasion │ closed-box │ text │ textattack │
│ textbugger_li_2018 │ evasion │ closed-box │ text │ textattack │
│ textfooler_jin_2019 │ evasion │ closed-box │ text │ textattack │
└────────────────────────────────────┴───────────────────┴────────────┴────────────────┴────────────┘
Note: Counterfit is designed for blackbox testing. Most frameworks have whitebox attacks available, but they are not included. The Counterfit scenario for including whitebox attacks would be using them on a stolen model from a model extraction attack.
To interact with a target, execute set_target with the target name as an argument. The terminal prompt will change to reflect the active target. Get information about the active target or attack by executing the show info command.
counterfit> set_target creditfraud
creditfraud>
From an active target there are two ways to run an attack. Either by executing the scan command, or by using the run command. There are some key differences to be aware of, scan is for automation; it has various arguments that control how many iterations of which attacks to execute against a target. scan by default uses random samples and random parameters. The scan function is useful for baselining and testing a target model. After completing all attacks, a scan summary will be printed. For example, when interacting with a target execute the following command scan --iterations 10 --attack hop_skip_jump
creditfraud> creditfraud> scan --num_iters 10 --attack hop_skip_jump
[+] success: Scanning Target: creditfraud (76d996d1)
HopSkipJump: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1.73it/s]
[+] success: Attack completed a54d393a
===============
<SCAN SUMMARY>
===============
┏━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ ┃ ┃ Best ┃ ┃
┃ Attack ┃ Total ┃ Success… ┃ Score ┃ ┃
┃ Name ┃ Runs ┃ (%) ┃ (attack… ┃ Best Parameters ┃
┡━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ art.att… │ 1 │ 1 │ N/A │ null │
│ │ │ (100.0%) │ │ │
└──────────┴──────────┴──────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[+] Time (min/avg/max) 0.6/ 0.6/ 0.6
[+] Queries (min/avg/max) 24552/24552/24552
creditfraud>
run on the other hand requires that you add an attack via use. After successfully executing the use command, the terminal prompt will change to reflect the active attack. With run you get control over each parameter and can set them individually.
To view the possible parameters to change, execute the show options. To change one or more parameters, execute the set_params command with the parameter you want to change as the argument followed by the value. For example, set max_eval=200. The terminal will print the updated parameters, including the default parameters. During manual testing it can be helpful to know where a particular value started.
creditfraud> set_attack hop_skip_jump
[+] success: Using 11d5bc52
creditfraud>HopSkipJump:11d5bc52> show options
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attack Options (type) ┃ Default ┃ Current ┃ Docs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Algo Parameters │ │ │ │
│ -------------------- │ -- │ -- │ -- │
│ batch_size (int) │ 64 │ 64 │ The size of the batch used by the estimator during inference. │
│ clip_values (list) │ [0.0, 1.0] │ [0.0, 1.0] │ Refer to attack file. │
│ curr_iter (int) │ 0 │ 0 │ Refer to attack file. │
│ init_eval (int) │ 100 │ 100 │ Initial number of evaluations for estimating gradient. │
│ init_size (int) │ 100 │ 100 │ Maximum number of trials for initial generation of adversarial examples. │
│ max_eval (int) │ 1000 │ 1000 │ Maximum number of evaluations for estimating gradient. │
│ max_iter (int) │ 50 │ 50 │ Maximum number of iterations. │
│ norm (int) │ 2 │ 2 │ Order of the norm. Possible values: "inf", np.inf or 2. │
│ targeted (bool) │ False │ False │ Should the attack target one specific class. │
│ verbose (bool) │ True │ True │ Show progress bars. │
│ target_labels (int) │ 0 │ 0 │ target labels for a targeted attack │
│ │ │ │ │
│ CFAttack Options │ │ │ │
│ -------------------- │ -- │ -- │ -- │
│ sample_index (int) │ 0 │ 0 │ Sample index to attack │
│ optimize (bool) │ False │ False │ Use Optuna to optimize attack parameters │
│ logger (str) │ basic │ basic │ Logger to log queries with │
└───────────────────────┴────────────┴────────────┴──────────────────────────────────────────────────────────────────────────┘
creditfraud>HopSkipJump:11d5bc52> set_params --max_eval=200
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Parameter (type) ┃ Default ┃ Current ┃ New ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Algo Parameters │ │ │ │
│ -------------------- │ -- │ -- │ -- │
│ batch_size (int) │ 64 │ 64 │ The size of the batch used by the estimator during inference. │
│ clip_values (list) │ [0.0, 1.0] │ (0.0, 1.0) │ Refer to attack file. │
│ curr_iter (int) │ 0 │ 0 │ Refer to attack file. │
│ init_eval (int) │ 100 │ 100 │ Initial number of evaluations for estimating gradient. │
│ init_size (int) │ 100 │ 100 │ Maximum number of trials for initial generation of adversarial examples. │
│ max_eval (int) │ 1000 │ 200 │ Maximum number of evaluations for estimating gradient. │
│ max_iter (int) │ 50 │ 50 │ Maximum number of iterations. │
│ norm (int) │ 2 │ 2 │ Order of the norm. Possible values: "inf", np.inf or 2. │
│ targeted (bool) │ False │ False │ Should the attack target one specific class. │
│ verbose (bool) │ True │ True │ Show progress bars. │
│ target_labels (int) │ 0 │ 0 │ target labels for a targeted attack │
│ │ │ │ │
│ CFAttack Options │ │ │ │
│ -------------------- │ -- │ -- │ -- │
│ sample_index (int) │ 0 │ 0 │ Sample index to attack │
│ optimize (bool) │ False │ False │ Use Optuna to optimize attack parameters │
│ logger (str) │ basic │ basic │ Logger to log queries with │
└──────────────────────┴────────────┴────────────┴──────────────────────────────────────────────────────────────────────────┘
creditfraud>HopSkipJump:11d5bc52>
After setting parameters, execute the run command to start the attack. An attack summary is printed on completion.
creditfraud>HopSkipJump:11d5bc52> run
HopSkipJump: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.47it/s]
[+] success: Attack completed 11d5bc52
For both scan and run, query logging is turned off by default. To enable logging execute run --log. Every query sent to the target will be logged and can be saved to disk. Logs are stored with the target in the attacks property, and they are saved to disk with the save command.
creditfraud>HopSkipJump:11d5bc52> save --results
[+] success: Successfully wrote counterfit/targets/results/11d5bc52/run_summary.json
There are number of reasons logging every query is useful, a functional extraction attack for example, but logging increases file size considerably. Learn more about the available commands.