Faster Python unittest #3210
Faster Python unittest #3210
Conversation
* Do not use `virtualenv` in Docker * Split unittest into many target, make them parallel
* because it uses `numpy.flip` in `test_image.py`
CMakeLists.txt
Outdated
| @@ -55,6 +55,7 @@ option(WITH_C_API "Compile PaddlePaddle with C-API(Prediction)" OFF) | |||
| option(WITH_GOLANG "Compile PaddlePaddle with GOLANG" OFF) | |||
| option(GLIDE_INSTALL "Download and install go dependencies " ON) | |||
| option(USE_NNPACK "Compile PaddlePaddle with NNPACK library" OFF) | |||
| option(UNITTEST_USE_VIRTUALENV "Python unittest with virtualenv" ON) | |||
There was a problem hiding this comment.
Do we need VirtualEnv when we are using Docker?
There was a problem hiding this comment.
This PR is not to use Python virtualenv in Docker. Because in docker, Python packages are managed by Docker.
Not use virtualenv in the unit tests will make unit tests faster in docker.
paddle/scripts/docker/build.sh
Outdated
| @@ -49,29 +49,27 @@ cmake .. \ | |||
| -DCUDNN_ROOT=/usr/ \ | |||
| -DWITH_STYLE_CHECK=${WITH_STYLE_CHECK:-OFF} \ | |||
| -DWITH_TESTING=${WITH_TESTING:-OFF} \ | |||
| -DCMAKE_EXPORT_COMPILE_COMMANDS=ON | |||
| -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ | |||
| -DUNITTEST_USE_VIRTUALENV=OFF | |||
There was a problem hiding this comment.
Should here refer to the flag defined in CMakeLists.txt?
paddle/setup.py.in
Outdated
| @@ -22,7 +22,7 @@ setup(name="py_paddle", | |||
| package_data={'py_paddle':['*.py','_swig_paddle.so']}, | |||
| install_requires = [ | |||
| 'nltk>=3.2.2', | |||
| 'numpy>=1.8.0', # The numpy is required. | |||
| 'numpy>=1.12.0', # The numpy is required. | |||
There was a problem hiding this comment.
Can we have a comment here explaining why 1.12 is required?
| ======================================== | ||
| Building in /paddle/build ... | ||
| ============================================ | ||
| Building and installing in /paddle/build ... |
There was a problem hiding this comment.
What is the difference between building and installing? I think we used to build only here because the installation is postponed in building the production Docker image. Do we have to run make install here so could we run Python test? Or, could we run Python test without make install?
There was a problem hiding this comment.
If we do not use virtualenv for Python unit-test in Docker. Then we must install Paddle into that container. So we cannot run Python tests without make install.
Also, the script before split make and make install in two steps, but always make install later. So just call make install once is simpler.
There was a problem hiding this comment.
What is the difference between building and installing? I think we used to build only here because the installation is postponed in building the production Docker image.
make installalways be invoked in current develop branch, https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/scripts/docker/build.sh#L72 .- I think
make installlogic is part of our unit test too. We must ensuremake installcommand can correctly installPaddlein container. And unit-tests should be based on installedPaddle.
reyoung
force-pushed
the
feature/fast_python_unittest
branch
3 times, most recently
from
83d8cbf
to
985bf51
Compare
* User must install Paddle python package before unittest. * Or use docker to build Paddle
reyoung
force-pushed
the
feature/fast_python_unittest
branch
from
985bf51
to
a720d21
Compare
virtualenvin Docker