Paleo is an analytical model to estimate the scalability and performance of deep learning systems. It can be used to
- efficiently explore the space of scalable deep learning systems,
- quickly diagnose their effectiveness for a given problem instance.
Paleo uses the following dependencies:
- numpy
- click
- six
- cuDNN (Optional. Use
--use_only_gemmto disable cuDNN heuristics)
Use pip to install the depenencies with the pinned versions:
pip install -r requirements.txt
Tested with Python 2.7, cuDNN v4 on Ubuntu 14.04.
To install Paleo, run the following command in the cloned directory:
python setup.py install
Paleo provides programmatic APIs to retrieve runtime estimations.
The following is an example of estimating SGD executions under strong scaling.
from paleo.profilers import BaseProfiler
class SGDProfiler(BaseProfiler):
def __init__(self, filename):
super(SGDProfiler, self).__init__(filename)
def simulate(self, workers, batch_size=128):
fwd_time, params_in_bytes = self.estimate_forward(batch_size //
workers)
bwd_time = self.estimate_backward(batch_size // workers)
update_time = self.estimate_update(params_in_bytes)
t_comp = fwd_time + bwd_time + update_time
t_comm = self.estimate_comm(workers, params_in_bytes)
return t_comp + t_commModel Architectures
Paleo uses a special json format to for model architecture specification. Predefined architectures can be found under the nets/ directory. Paleo also provides a convertor for Caffe prototxt format (see nets/caffe/ for details).
- AlexNet v2
- Inception v3
- NiN
- Overfeat
- VGG-16
- ResNet-50 (from Caffe spec)
- DenseNet (from Caffe spec)
Hardware
Predefined hardware specificiations are in paleo/device.py.
Hang Qi, Evan R. Sparks, and Ameet Talwalkar. Paleo: A Performance Model for Deep Neural Networks. International Conference on Learning Representations (ICLR), 2017.
@inproceedings{qi17paleo,
author={Hang Qi and Evan R. Sparks and Ameet Talwalkar},
booktitle={Proceedings of the International Conference on Learning Representations},
title={Paleo: A Performance Model for Deep Neural Networks},
year={2017}
}
Apache 2.0