Benchmark including PaddlePaddle, TensorFlow and Caffe. #219

qingqing01 · 2016-10-18T05:19:35Z

All the configs are in benchmark/ file.
use run.sh or run_multi.sh to execute.

reyoung · 2016-10-19T05:43:33Z

benchmark/tensorflow/rnn/README.md

+
+```bash
+pip install tflearn
+```


https://pip.readthedocs.io/en/1.1/requirements.html
https://github.com/BVLC/caffe/blob/master/python/requirements.txt

use requirements.txt.

luotao1

run.sh和run_multi.sh里面除了个数不一样，很多地方都一样，能不能写成一个.sh文件呢，或者共用下命令行函数。
tensorflow里面单节点的py和多节点的py，看着也非常像，是不是也就只有节点数不一样，其他都一样呢。那写成一个py就够了。

luotao1 · 2016-11-23T06:06:36Z

benchmark/README.md

+
+Platform: 
+
+- PaddlePaddle: 


是不是要加一下是Paddle的哪个版本，比如就这次release版本

Done, 增加的是这次release版本对应的Docker Image

luotao1 · 2016-11-23T06:07:14Z

benchmark/README.md

+- Tensorflow: gcr.io/tensorflow/tensorflow:0.11.0rc0-gpu 
+- Caffe: 
+
+Several convolutional neural networks and recurrent neural network are used to test.


recurrent neural network后面加s

luotao1 · 2016-11-23T06:07:44Z

benchmark/README.md

+- CPU: 12-core Intel(R) Xeon(R) CPU E5-2620 v2 @2.10GHz
+- GPU: Tesla K40m
+- cuDNN: v5.1
+- system: Docker 1.12.1, all platform are tested in docker environment.


all platforms

luotao1 · 2016-11-23T06:10:58Z

benchmark/README.md

+
+### Benchmark Model
+
+AlexNet, GooleNet and a small network which refer the config of cifar10 in Caffe are used.


a small network which refer the config of cifar10 in Caffe 不是很通。是说用了caffe中那个cifar10的网络么？a small network use cifar10 in Caffe?

luotao1 · 2016-11-23T06:11:36Z

benchmark/README.md

+- [SmallNet](https://github.com/BVLC/caffe/blob/master/examples/cifar10/cifar10\_quick\_train\_test.prototxt)
+
+
+### Singe-GPU


luotao1 · 2016-11-23T07:07:59Z

benchmark/README.md

+
+#### LSTM in Text Classification
+
+Testing network for different hidden size, batch size with `2 lstm layer + fc` network.


Testing 2 lstm layer + fc network with differnt hidden size and batch size

luotao1 · 2016-11-23T07:08:23Z

benchmark/README.md

+
+#### Seq2Seq
+
+The benchmark of sequence-to-sequence network will be add later.


will be added

luotao1 · 2016-11-23T07:08:51Z

benchmark/README.md

+
+#### Seq2Seq
+
+The benchmark of sequence-to-sequence network will be add later.


luotao1 · 2016-11-23T07:16:39Z

benchmark/paddle/image/run.sh

+    --log_period=10 \
+    --test_period=100 \
+    --config_args=$args \
+    --cudnn_dir=/home/dangqingqing/tools/cudnn-5.1/lib64 \


cudnn_dir路径写死了，下同

luotao1 · 2016-11-23T07:17:26Z

benchmark/paddle/image/run.sh

+  if [ ! -d "train.txt" ]; then
+    for ((i=1;i<=1024;i++))
+    do
+      echo "train/n09246464/n09246464_38735.jpeg 972" >> train.txt


这行是用来做什么呢，写1024个一样的jpeg？

这个是因为：

使用 paddle train命令发起程序，PaddlePaddle必须依赖DataProvider才能知道数据类型和大小(dense_vector, integer_value等)。

配置文件中define_py_data_sources2指定DataProvider的同时，必须有train_list，而且如果文件为空的话，Init parameters done之后就结束了。

所以这里生成伪数据列表，实际在DataProvider里产生的是随机数据。

修改了DataProvider，可以使用空的文件列表，已更新。

…ark_cfg_doc

qingqing01 · 2016-11-23T12:10:23Z

run.sh和run_multi.sh里面除了个数不一样，很多地方都一样，能不能写成一个.sh文件呢，或者共用下命令行函数。

这个只合并了PaddlePaddle的命令。 Caffe和Tensorflow没有合并，是因为这两个单机和多机发起程序的命令本身就不一样。

tensorflow里面单节点的py和多节点的py，看着也非常像，是不是也就只有节点数不一样，其他都一样呢。那写成一个py就够了。

TensorFlow多卡的配置本身可以跑单卡，但是单卡配置可以写的更简单、少一些逻辑。在看Tensorflow的一些教程（ https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/image/cifar10 ）、包括这里的配置 https://github.com/soumith/convnet-benchmarks/tree/master/tensorflow (当然这个没有多卡配置)时，单卡配置和多卡也是分开的，单卡并没有公用多卡配置来跑单卡。

所以TensorFlow这里单卡和多卡是分开的，如果大家觉得有必要合并的，合并一下也可以。

luotao1 · 2016-11-23T13:24:25Z

benchmark/README.md


-All the tests in caffe use `caffe time` to execute, which is not including the parameter updating process. But the time in PaddlePaddle and TensorFlow contains it.
+All the experiments in caffe use `caffe time` to execute, which does not include the time of parameter updating. The time in PaddlePaddle and TensorFlow contains it. But, compared with the total time, the time of parameter updating is relatively little.


All the single-GPU experiments in caffe use caffe time to calculate the elapsed time, which does not include the parameter updating time. However, both PaddlePaddle and TensorFlow benchmark contain this parameter updating time. As compared with the total time, this part is relatively little, we can ignore it.

luotao1 · 2016-11-23T13:26:06Z

benchmark/README.md

@@ -102,15 +102,15 @@ We use lstm network for text classfication to test benchmark.

 ### Dataset
 -  [IMDB](http://www.iro.umontreal.ca/~lisa/deep/data/imdb.pkl)
- Sequence legth=100, in fact, PaddlePaddle support training with variable-length sequence. But TensorFlow need to pad, in order to compare, we also pad sequence length to 100 in PaddlePaddle.
+- Sequence legth is 100. In fact, PaddlePaddle supports training with variable-length sequence, but TensorFlow needs to pad, we also pad sequence length to 100 in PaddlePaddle in order to compare.


Sequence length.

In fact, PaddlePaddle supports training with variable-length sequence, but TensorFlow needs to pad. Thus, we also pad sequence length to 100 in PaddlePaddle in order to compare.

luotao1 · 2016-11-23T13:30:11Z

所以TensorFlow这里单卡和多卡是分开的，如果大家觉得有必要合并的，合并一下也可以。

@wangkuiyi @reyoung @gangliao @hedaoyuan @backyes 的意见

reyoung · 2016-11-28T05:42:16Z

如果Paddle不是核心的code，那么咱也没必要非要 DIY(Don't Repeat Yourself) 吧。况且，@qingqing01 说这个tensorflow的官方也没有合并这些。

…ark_cfg_doc

add share_external_data interface demo

* Stand out toy example & fix bugs in child threads * Refine comments

Add benchmark config and document

a8342d0

qingqing01 mentioned this pull request Oct 18, 2016

Add benchmark and reduce GPU memory for cudnn_conv and speed up cudnn_conv. #217

Closed

qingqing01 assigned backyes, emailweixu, gangliao, reyoung and hedaoyuan Oct 18, 2016

qingqing01 changed the title ~~benchmark including PaddlePaddle, TensorFlow and Caffe.~~ Benchmark including PaddlePaddle, TensorFlow and Caffe. Oct 18, 2016

reyoung changed the base branch from master to develop October 26, 2016 08:13

reyoung assigned wangkuiyi and luotao1 and unassigned reyoung, backyes, gangliao, emailweixu and hedaoyuan Nov 22, 2016

reyoung reviewed Nov 22, 2016

View reviewed changes

luotao1 requested changes Nov 23, 2016

View reviewed changes

Update doc and merge run_multi.sh into run.sh for PaddlePaddle.

9d377f0

qingqing01 force-pushed the benchmark_cfg_doc branch 4 times, most recently from 0447e63 to 3046e2d Compare November 23, 2016 11:44

Merge branch 'develop' of https://github.com/baidu/Paddle into benchm…

b1cc9da

…ark_cfg_doc

qingqing01 force-pushed the benchmark_cfg_doc branch from 3046e2d to b1cc9da Compare November 23, 2016 11:51

Use empty file list for paddle/image.

88a7cbd

luotao1 reviewed Nov 23, 2016

View reviewed changes

qingqing01 added 2 commits November 30, 2016 15:53

follow LuoTao's commits

b3945f9

Merge branch 'develop' of https://github.com/baidu/Paddle into benchm…

68060aa

…ark_cfg_doc

qingqing01 force-pushed the benchmark_cfg_doc branch from bb803ac to 68060aa Compare November 30, 2016 08:04

luotao1 approved these changes Dec 5, 2016

View reviewed changes

luotao1 merged commit a0a87ac into PaddlePaddle:develop Dec 5, 2016

qingqing01 deleted the benchmark_cfg_doc branch July 7, 2017 13:35

thisjiang pushed a commit to thisjiang/Paddle that referenced this pull request Oct 28, 2021

refine Context::NewVarName (PaddlePaddle#219)

e1213f6

zhoutianzi666 pushed a commit to zhoutianzi666/Paddle that referenced this pull request May 23, 2022

Merge pull request PaddlePaddle#219 from JZZ-NOTE/share_external_data

eee5f72

add share_external_data interface demo

lizexu123 pushed a commit to lizexu123/Paddle that referenced this pull request Feb 23, 2024

Stand out toy example & fix bugs in child threads (PaddlePaddle#219)

df0dff8

* Stand out toy example & fix bugs in child threads * Refine comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark including PaddlePaddle, TensorFlow and Caffe. #219

Benchmark including PaddlePaddle, TensorFlow and Caffe. #219

qingqing01 commented Oct 18, 2016

reyoung Oct 19, 2016

luotao1 left a comment

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

luotao1 Nov 23, 2016

luotao1 Nov 23, 2016

qingqing01 Nov 23, 2016

qingqing01 Nov 23, 2016

qingqing01 commented Nov 23, 2016 •

edited

Loading

luotao1 Nov 23, 2016

luotao1 Nov 23, 2016

luotao1 commented Nov 23, 2016

reyoung commented Nov 28, 2016


		### Benchmark Model

		AlexNet, GooleNet and a small network which refer the config of cifar10 in Caffe are used.

		- [SmallNet](https://github.com/BVLC/caffe/blob/master/examples/cifar10/cifar10\_quick\_train\_test.prototxt)


		### Singe-GPU


		#### LSTM in Text Classification

		Testing network for different hidden size, batch size with `2 lstm layer + fc` network.


		#### Seq2Seq

		The benchmark of sequence-to-sequence network will be add later.


		All the tests in caffe use `caffe time` to execute, which is not including the parameter updating process. But the time in PaddlePaddle and TensorFlow contains it.
		All the experiments in caffe use `caffe time` to execute, which does not include the time of parameter updating. The time in PaddlePaddle and TensorFlow contains it. But, compared with the total time, the time of parameter updating is relatively little.


		Platform:

		- PaddlePaddle:

Benchmark including PaddlePaddle, TensorFlow and Caffe. #219

Benchmark including PaddlePaddle, TensorFlow and Caffe. #219

Conversation

qingqing01 commented Oct 18, 2016

Choose a reason for hiding this comment

luotao1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 commented Nov 23, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luotao1 commented Nov 23, 2016

reyoung commented Nov 28, 2016

qingqing01 commented Nov 23, 2016 •

edited

Loading