Dynamic Hierarchical Mimicking

Official implementation of our DHM training mechanism as described in Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives (CVPR'20) by Duo Li and Qifeng Chen on CIFAR-100 and ILSVRC2012 benchmarks with the PyTorch framework.

We dissolve the inherent defficiency inside the deeply supervised learning mechanism by introducing mimicking losses between the main branch and auxiliary branches located in different network depths, while no computatonal overheads are introduced after discarding those auxiliary branches during inference.

Requirements

Dependencies

• PyTorch 1.0+
• NVIDIA-DALI (in development, not recommended)

Dataset

Download the ImageNet dataset and move validation images to labeled subfolders. To do this, you can use the following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh

ImageNet Pretrained models

Architecture	Method	Top-1 / Top-5 Error (%)
ResNet-18	DSL	29.728 / 10.450
	DHM	28.714 / 9.940
ResNet-50	DSL	23.874 / 7.074
	DHM	23.430 / 6.764
ResNet-101	DSL	22.260 / 6.128
	DHM	21.348 / 5.684
ResNet-152	DSL	21.602 / 5.824
	DHM	20.810 / 5.396

* DSL denotes Deeply Supervised Learning

* DHM denotes Dynamic Hierarchical Mimicking (ours)

Usage

Training

Please refer to the training recipes for illustrative examples .

Inference

After downloading the pre-trained models, you could test the model on the ImageNet validation set with this command line:

python imagenet.py \
    -a <resnet-model-name> \
    -d <path-to-ILSVRC2012-data> \
    --weight <pretrained-pth-file> \
    -e

Citations

If you find our work useful in your research, please consider citing:

@InProceedings{Li_2020_CVPR,
author = {Li, Duo and Chen, Qifeng},
title = {Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}