TensorFlow Code for 'Towards Privacy-Preserving Visual Recognition via Adversarial Training: A Pilot Study'
TensorFlow Implementation of our ECCV 2018 paper "Towards Privacy-Preserving Visual Recognition via Adversarial Training: A Pilot Study".
This paper aims to improve privacy-preserving visual recognition, an increasingly demanded feature in smart camera applications, by formulating a unique adversarial training framework.
The proposed framework explicitly learns a degradation transform for the original video inputs, in order to optimize the trade-off between target task performance and the associated privacy budgets on the degraded video. A notable challenge is that the privacy budget, often defined and measured in task-driven contexts, cannot be reliably indicated using any single model performance, because a strong protection of privacy has to sustain against any possible model that tries to hack privacy information.
Such an uncommon situation has motivated us to propose two strategies to enhance the generalization of the learned degradation on protecting privacy against unseen hacker models. Novel training strategies, evaluation protocols, and result visualization methods have been designed accordingly.
Two experiments on privacy-preserving action recognition, with privacy budgets defined in various ways, manifest the compelling effectiveness of the proposed framework in simultaneously maintaining high target task (action recognition) performance while suppressing the privacy breach risk.
Google Drive: https://drive.google.com/file/d/1dIwr7JrFVkuo9X0WXpSqkzQNr_jhRzv7/view?usp=sharing
sudo apt install gcc g++ git libjpeg-dev libpng-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev pkg-config cmake libgtk2.0-dev libeigen3-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev sphinx-common libtbb-dev yasm libfaac-dev libopencore-amrnb-dev libopencore-amrwb-dev libopenexr-dev libgstreamer-plugins-base1.0-dev libavcodec-dev libavutil-dev libavfilter-dev libavformat-dev libavresample-dev
wget https://github.com/Itseez/opencv/archive/3.4.0.zip
unzip 3.4.0.zip
cd opencv-3.4.0
mkdir build
cd build
cmake -D WITH_FFMPEG=ON -D WITH_CUDA=OFF -D BUILD_TIFF=ON -D BUILD_opencv_java=OFF -D ENABLE_AVX=ON -D WITH_OPENGL=ON -D WITH_OPENCL=ON -D WITH_IPP=ON -D WITH_TBB=ON -D WITH_EIGEN=ON -D WITH_V4L=ON -D WITH_VTK=OFF -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_opencv_python2=OFF -D CMAKE_INSTALL_PREFIX=$(python3 -c "import sys; print(sys.prefix)") -D PYTHON3_EXECUTABLE=$(which python3) -D PYTHON3_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") -D PYTHON3_PACKAGES_PATH=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_PYTHON_EXAMPLES=ON -D INSTALL_C_EXAMPLES=OFF -D PYTHON_EXECUTABLE=/home/wuzhenyu_sjtu/anaconda2/envs/tf-1.8-py36/bin/python -D BUILD_EXAMPLES=ON ..
make -j 128
sudo make install
# If OpenCV has FFMPEG support, it will give "FFMPEG: YES"
python -c "import cv2; print(cv2.getBuildInformation())" | grep -i ffmpeg
SBU: https://www3.cs.stonybrook.edu/~kyun/research/kinect_interaction/index.html (Please download the clean version)
HMDB51: http://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/hmdb51_org.rar
UCF101: https://www.crcv.ucf.edu/data/UCF101/UCF101.rar
VISPR: https://tribhuvanesh.github.io/vpa/
PA-HMDB51: https://github.com/TAMU-VITA/PA-HMDB51
We have used Inception-v1, Inception-V2, MobileNet-V1, ResNet-V1 and ResNet-V2. They are available at https://github.com/tensorflow/models/tree/master/research/slim/nets.
- SBU
- data (please copy to this path)
- pretrain: https://drive.google.com/drive/u/0/folders/1bD-e4vtMPF04sW3vEBIWVLaPDlLaeYKl
- C3D (please copy to this path)
- target_models
- degradation_models (please copy to this path)
- budget_k%d
- adversarial_training
- two_fold_evaluation
- pretrained_budget_model (please copy to this path)
Train and test using kbeam: (run this command under adversarial_training dir)
python main_kbeam.py --GPU_id 0,1 --_K 4
Two-fold validate on kbeam model: (run this command under two_fold_evaluation dir)
python main_2fold.py --GPU_id 0,1,2,3 --adversarial_job_name kbeam-NoRest-K4
Python 3.5
If you find this code useful, please cite the following paper:
@inproceedings{wu2018towards,
title={Towards privacy-preserving visual recognition via adversarial training: A pilot study},
author={Wu, Zhenyu and Wang, Zhangyang and Wang, Zhaowen and Jin, Hailin},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
pages={606--624},
year={2018}
}Please also check our extended version highlighted by a new dataset, termed PA-HMDB51, with both target task labels (action) and selected privacy attributes (gender, age, race, nudity, and relationship) annotated on a per-frame basis:
Zhenyu Wu, Haotao Wang, Zhaowen Wang, Hailin Jin and Zhangyang Wang. "Privacy-Preserving Deep Action Recognition:An Adversarial Learning Frameworkand A New Dataset", TPAMI, 2020. [pdf][github][project page]