Fluid performance tuning plan

[reyoung]

We plan to tune fluid's performance with a loop with three steps:

1. Profile: To figure out which part of the fluid is slow.
2. Find problems & Give a fix: We will discuss and find the problems based on profile results.
3. Profile: To confirm the problems has been solved and the performance is improved.

There are several jobs for these three steps:

1. Find a machine with docker and GPU for profiling. @jacquesqiao
2. Neural network configurations for CNN, LSTM, etc. @qingqing01 @dzhwinter
   • Add benchmark configuration and scripts for convnets. #6056
3. Setup an environment for profiling. @chengduoZH
   • Use cProfile for Python, yap for Python/C++, nvprof for CUDA
4. Find problems: All members together.
5. Fix GPU problems: @jacquesqiao @qingqing01
6. Fix CPU problems: @dzhwinter
7. Fix Python problems: TODO

[dzhwinter]

1. Python code optimization
   We use CProfile profiling the python code. The tool usage can be checked out at cpu_profiling. If we need to dig into the code. You can add this snippet to the script, then you will get an accurate function call count. This is useful to measure whether there is redundant Variable, Operator is created.

import cProfile
pr = cProfile.Profile()
pr.enable()
// here the test codes. 
pr.disable()

• fine-tune the fit_a_line tests, check the components time cost. e.g.Program, Variable, Operator, Block
• fine-tune one forward pass, check total time cost in Python side.

2. C++ code optimization
   The yep + gperftools is excellent, but there is many Py_Eval, Py_Object function call disturb the focus point. Here is the method only retain necessary c++ calls.

#ifdef ENABLE_PROFILE
#include <gperftools/profiler.h>
#endif

ProfilerStart("cs.prof");
// here the test codes.
ProfilerStop()
ProfilerFlush()

• fine-tune the training forward/backward process, find the bottle-neck.

3. Cuda code
   TBD

[dzhwinter]

Also, we had created benchmark docker image and benchmark repo for every one reproduce the result.
The benchmark repo contains the tensorflow and paddle scripts.
https://github.com/dzhwinter/benchmark

The docker image contains all the profile tools.
https://hub.docker.com/r/dzhwinter/benchmark

Need to note that the docker image should be run in host mode, then you can check the profiler result in your browser. @tonyyang-svail

nvidia-docker run -it --rm --network=host -v $PWD:/paddle -v /root/.cache:/root/.cache  dzhwinter/benchmark /bin/bash  

[jacquesqiao]

profiling document

CPU

https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/optimization/cpu_profiling.md

GPU

https://github.com/PaddlePaddle/Paddle/blob/develop/doc/howto/optimization/gpu_profiling_cn.rst

[jacquesqiao]

profile steps:

1. make a work dir for your self

2. checkout paddle source code and benchmark source code
   https://github.com/dzhwinter/benchmark

3. cd to your source code

cd benchmark

4. start and enter the benchmark container

nvidia-docker run -it --rm --network=host -v $PWD:/paddle -v /root/.cache:/root/.cache  dzhwinter/benchmark /bin/bash  

5. build your source code inside the container and installed it.

# if you want to use pprof to debug, use `RelWithDebInfo`, if just want to calculate time, can use `Release`
# use RelWithDebInfo to add debug info
cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo|Release
make -j4 install

6. run benchmark code

export OMP_NUM_THREADS=1
python -m yep -v test_recognize_digits_conv.py

7. run pprof. (NOTE: please random choose a port between 8000:9000)

# stop openmp optimize
pprof -http=0.0.0.0:8123 `which python` test_recognize_digits_conv.py.prof

8. visit webpage
   open your browser and type ip:port. for example: server_ip : 8123.

[reyoung]

Basic Performance tunning has been done. Please reference Project https://github.com/PaddlePaddle/Paddle/projects/29#card-5921207 for the following works