Download, build, install External dependencies via cmake #1017

gangliao · 2016-12-27T03:26:56Z

#1001

Add dependencies
integrate into Paddle
Fix bug
clean travis ci

目前测试Mac, Centos, ubuntu已经全部通过。

文档clean工作下一个PR
docker clean工作下一个PR
third_party test code 下一个PR

由于PR比较复杂，我这里梳理一下，帮助review

已解决：

glog, gflags, protobuf, gtest, python, numpy, wheel, pip, swig, python-protobuf, zlib, setuptools, six, cython 自动下载，编译，安装。python部分最复杂。
warpctc自动下载，编译，安装。原版warpctc存在一些bug,比如没有openmp的开关，没有gpu的开关，make install写的有问题。已经给他们提交PR，并且merge了。https://github.com/baidu-research/warp-ctc/commits/master
后期会继续给他们贡献一个static库版本。

通过观察Travis CI编译和测试时间，发现第三方依赖全部下载，编译，安装所需的时间与之前没有明显区别。

之前的Travis CI: Linux 28 min 13 sec
目前的Travis CI: Linux 25 min 35 sec

未解决：

目前唯一没有解决的依赖是代数库 openblas. 原因主要是目前paddle有几个函数依赖于lapack，openblas部分代码需要默认使用gfortran编译，在linux系统上面，除了链接libopenblas.a, 最终还需要链接libgfortran.so，这部分我再考证一下。

本PR带来的好处是：

无需预装第三方依赖，一键式自动完成Paddle编译安装。cmake .. && make就行了。
所有第三方依赖全部以静态库的方式包进来。
同时解决了python解释器和python库不一致的问题

长远好处：

Paddle的第三方依赖全部来自CMake自动下载，编译，安装，与系统本身相对隔离，对今后发布二进制有好处。
CMake 第三方依赖库的自动下载，编译，安装，为不同操作系统的移植工作，提供了极大的便利。

Travis ci linux最后安装的时候有问题, 可能是pip或者python版本的问题，比较奇怪的是，我fork的版本，一样的代码，travis ci确实没有这个问题的。https://travis-ci.org/gangliao/Paddle @reyoung

First time run paddle, need to install some python dependencies.
�[33mThe directory '/home/travis/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.�[0m
�[33mThe directory '/home/travis/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.�[0m
�[31mpy_paddle-0.9.0a0-cp27-cp27m-linux_x86_64.whl is not a supported wheel on this platform.�[0m
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
pip install wheels failed. 
Please use 'sudo paddle' at the first time you use PaddlePaddle
PaddlePaddle will install some python dependencies automatically.

wangkuiyi

赞！是不是可以在这个PR里顺便创建一个 /testing 子目录，里面放上一个简单的程序 third_party_dependencies_test.cc，同时用了这几个external libraries。这个程序编译如果没有错，则说明cmake能成功使用这些external libraries。

gangliao · 2016-12-27T03:35:53Z

@wangkuiyi ok

gangliao · 2016-12-27T03:38:32Z

cmake/external/warpctc.cmake

+
+ExternalProject_Add(
+    warpctc
+    GIT_REPOSITORY  "https://github.com/gangliao/warp-ctc.git"


warp-ctc官方目前支持力度不是很大，完全融入Paddle, 其实需要做一些定制化修改，可以考虑帮助他们改进，或者自己维护一个版本。

gangliao · 2016-12-27T03:51:22Z

cmake/external/protobuf.cmake

+        "${PROTOBUF_INSTALL_DIR}/lib/libprotoc.lib" CACHE FILEPATH "protobuf libraries." FORCE)
+  SET(PROTOBUF_PROTOC_EXECUTABLE "${PROTOBUF_INSTALL_DIR}/bin/protoc.exe" CACHE FILEPATH "protobuf executable." FORCE)
+ELSE(WIN32)
+  SET(PROTOBUF_LIBRARIES


发现linux下面是lib64

… external

gangliao · 2016-12-27T15:28:49Z

CMakeLists.txt

 option(WITH_DOC "Compile PaddlePaddle with documentation" OFF)
-option(WITH_SWIG_PY "Compile PaddlePaddle with py PaddlePaddle prediction api" ${SWIG_FOUND})
+option(WITH_SWIG_PY "Compile PaddlePaddle with py PaddlePaddle prediction api" ON)
 option(ON_TRAVIS "Running test on travis-ci or not." OFF)
 option(ON_COVERALLS "Generating code coverage data on coveralls or not." OFF)
 option(COVERALLS_UPLOAD "Uploading the generated coveralls json." ON)



CMakeLists 干净很多了

gangliao · 2016-12-27T15:29:45Z

CMakeLists.txt

+find_package(Git REQUIRED)
+find_package(Threads REQUIRED)
+
+include(simd)


runtime avx搞定之后的话，simd也可以去掉了

gangliao · 2016-12-27T15:31:53Z

cmake/external/warpctc.cmake

+        "${WARPCTC_INSTALL_DIR}/lib/warpctc.dll" CACHE FILEPATH "Warp-ctc Library" FORCE)
+ELSE(WIN32)
+    SET(WARPCTC_LIBRARIES
+        "${WARPCTC_INSTALL_DIR}/lib/libwarpctc.dylib" CACHE FILEPATH "Warp-ctc Library" FORCE)


dylib 还需要添加linux so的情况

gangliao · 2016-12-27T15:33:07Z

cmake/util.cmake

@@ -81,18 +81,6 @@ function(link_paddle_exe TARGET_NAME)
        set(METRIC_LIBS "")
    endif()

-    if(PADDLE_WITH_INTERNAL)


@backyes yanfei 说可以去掉internal部分了

gangliao · 2016-12-27T15:33:56Z

paddle/cuda/CMakeLists.txt

@@ -88,6 +88,8 @@ else()
                ${CUDA_CXX_SOURCES})
 endif()

+add_dependencies(paddle_cuda ${external_project_dependencies})


添加external_project_dependencies依赖，保证外部依赖先完成编译安装

gangliao · 2016-12-27T15:34:43Z

paddle/function/cross_map_normal_op.cpp

-    CHECK_EQ(1, inputs.size());
-    CHECK_EQ(2, outputs.size());
-    CHECK_EQ(0, inouts.size());
+    CHECK_EQ(1, static_cast<int>(inputs.size()));


引入新的glog, 编译发现paddle还是存在一些类型不一致问题，顺手解决

gangliao · 2016-12-27T15:38:53Z

cmake/external/protobuf.cmake

+    PREFIX          ${PROTOBUF_SOURCES_DIR}
+    DEPENDS         zlib
+    GIT_REPOSITORY  "https://github.com/google/protobuf.git"
+#   GIT_TAG         "v3.1.0"


protobuf GIT_TAG目前没有添加，原因是支持最新Mac Serria系统的代码还没有release出来。

gangliao · 2016-12-27T16:35:33Z

cmake/external/warpctc.cmake

+    UPDATE_COMMAND ""
+)
+
+SET(WARPCTC_INCLUDE_DIR "${WARP_INSTALL_DIR}/include" CACHE PATH "Warp-ctc Directory" FORCE)


WARP_INSTALL_DIR 写错了。。WARPCTC_INSTALL_DIR

gangliao · 2017-01-04T16:35:13Z

CMakeLists.txt

+# See the License for the specific language governing permissions and
+# limitations under the License
+
+cmake_minimum_required(VERSION 3.0)


cmake最好提升到3.0, 因为外部依赖库普遍最低要求3.0

gangliao · 2017-01-05T03:27:52Z

paddle/trainer/tests/test_Trainer.cpp

@@ -96,11 +96,6 @@ TEST(checkGradient, multi) {
 TEST(checkGradient, hsigmoid) { checkGradientTest(configFile2, false, false); }

 TEST(checkGradient, chunk) {
-#if defined(__APPLE__) || defined(__OSX__)


这部分直接放在cmake里面吧

gangliao · 2017-01-05T03:28:14Z

paddle/trainer/tests/CMakeLists.txt

@@ -17,9 +17,10 @@ add_test(NAME test_Compare
 ################# test_Trainer ###########################
 add_unittest_without_exec(test_Trainer
    test_Trainer.cpp)
-set(diy_dll_dir ${CMAKE_CURRENT_BINARY_DIR}/../../gserver/tests)


没什么用，删了

gangliao · 2017-01-05T03:29:14Z

paddle/scripts/travis/before_install.osx.sh

-cd googletest-release-1.8.0/
-cmake .
-make install
+brew install openblas md5sha1sum 


openblas还在考察

gangliao · 2017-01-05T03:31:01Z

paddle/function/ContextProjectionOp.cpp

-    CHECK_EQ(3, inputs.size());
-    CHECK_EQ(1, outputs.size());
-    CHECK_EQ(0, inouts.size());
+    CHECK_EQ(3, static_cast<int>(inputs.size()));


看起来新版的glog很严格

gangliao · 2017-01-05T03:32:34Z

cmake/util.cmake

-        ${PROTOBUF_LIBRARY}
-        ${LIBGLOG_LIBRARY}
-        ${GFLAGS_LIBRARIES}
+        ${EXTERNAL_LIBS}


有了EXTERNAL_LIBS，其他都可以去掉了

gangliao · 2017-01-05T03:34:37Z

cmake/rdma.cmake

+    #if this module is not called, RDMA_INC_DIR RDMA_LIBS will be null, so top module always refer this variable
+    message(FATAL_ERROR, "RDMA libraries are not found, try to set RDMA_ROOT or check all related libraries.")
+  endif()
+else(WITH_RDMA)


如果WITH_RDMA=OFF, 就设置为空

gangliao · 2017-01-05T03:36:53Z

@wangkuiyi @reyoung 开始review吧

reyoung

请先不要merge。

我自己mac下build似乎有些问题。我先看一下。

reyoung · 2017-01-06T02:48:15Z

对于MacOS，必须安装xcode才可以编译。并且要

sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer/

并且，目前应该只能生成Linux Makefile进行编译。其他的编译方式都可能会有错误(测试了生成ninja的编译文件出错)。

参考连接 https://public.kitware.com/Bug/print_bug_page.php?bug_id=13538

需要使用automake，否则

+ test -d Tools/config
+ aclocal -I Tools/config
./autogen.sh: line 11: aclocal: command not found

不支持现在的WITH_PYTHON=OFF配置了。不过一般也不这么配置。回头fix一下。

gangliao · 2017-01-06T03:12:51Z

@reyoung 之后会增加一个PR 来find_package(SWIG)，毕竟Swig只是用来生成paddle python wrapper, 系统有swig的话，就不需要下载，编译swig了

gangliao · 2017-01-06T03:13:35Z

#1083 这个PR会帮助改进目前的paddle swig生成方式。

gangliao added 2 commits December 27, 2016 10:48

Add externel google's dependencies

8a2ceda

Add other extern dependencies

280994f

gangliao assigned wangkuiyi and reyoung Dec 27, 2016

gangliao requested a review from Xreki December 27, 2016 03:28

wangkuiyi requested changes Dec 27, 2016

View reviewed changes

wangkuiyi mentioned this pull request Dec 27, 2016

Remove the main function inside the test file #1003

Merged

gangliao commented Dec 27, 2016

View reviewed changes

gangliao added 8 commits December 27, 2016 17:11

Update external libs

6cd4b6e

Refine cmake file names

662f174

Remove paddle internals

62b55cc

Refine CMakeLists

a02ec8c

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

4b94494

… external

Update external dependencies cmake

46cadae

Fix glog check type unmatch

de3c152

Add external_project_dependencies for targets

0b95671

gangliao commented Dec 27, 2016

View reviewed changes

Fix external denpendencies build order

338dc3e

gangliao commented Dec 27, 2016

View reviewed changes

gangliao added 3 commits December 28, 2016 09:58

Fix warpctc header directory

2ffb6db

Clean Travis ci and fix bug

aee0857

Add extern python interp

8acd1ac

Fix conflict with develop

b090ce3

gangliao commented Jan 4, 2017

View reviewed changes

Fix merge conflict bug and glog

fc47492

gangliao added the enhancement label Jan 4, 2017

Add python in travis ci for paddle version command

3ecc63a

gangliao commented Jan 5, 2017

View reviewed changes

gangliao added 3 commits January 5, 2017 11:59

Add wheel for paddle version command

0407902

Update travis ci pip

19dc8df

Clean travis ci

d0a5ce2

wangkuiyi approved these changes Jan 5, 2017

View reviewed changes

reyoung requested changes Jan 5, 2017

View reviewed changes

gangliao mentioned this pull request Jan 6, 2017

Refactor the generation process of _swig_paddle.so #1083

Closed

gangliao mentioned this pull request Jan 6, 2017

make install 阶段的编译动作能否移除，否则会影响源码编译体验 #883

Closed

reyoung approved these changes Jan 6, 2017

View reviewed changes

reyoung merged commit d41ac09 into PaddlePaddle:develop Jan 6, 2017

gangliao deleted the external branch January 6, 2017 07:26

Xreki mentioned this pull request Jan 9, 2017

unti test Layer.WarpCTCLayer fails #814

Closed

This was referenced Jan 10, 2017

Update build and install docs #1106

Merged

Add external openblas #1150

Merged

zhhsplendid pushed a commit to zhhsplendid/Paddle that referenced this pull request Sep 25, 2019

Update refactorization.md (PaddlePaddle#1017)

921ca4f

wangxicoding pushed a commit to wangxicoding/Paddle that referenced this pull request Dec 9, 2021

Use double precisionto train in full procedure (PaddlePaddle#1017)

61c638f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download, build, install External dependencies via cmake #1017

Download, build, install External dependencies via cmake #1017

gangliao commented Dec 27, 2016 •

edited

Loading

wangkuiyi left a comment

gangliao commented Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016 •

edited

Loading

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Dec 27, 2016

gangliao Jan 4, 2017

gangliao Jan 5, 2017

gangliao Jan 5, 2017

gangliao Jan 5, 2017

gangliao Jan 5, 2017

gangliao Jan 5, 2017

gangliao Jan 5, 2017

gangliao commented Jan 5, 2017

reyoung left a comment

reyoung commented Jan 6, 2017 •

edited

Loading

gangliao commented Jan 6, 2017

gangliao commented Jan 6, 2017

Download, build, install External dependencies via cmake #1017

Download, build, install External dependencies via cmake #1017

Conversation

gangliao commented Dec 27, 2016 • edited Loading

已解决：

未解决：

本PR带来的好处是：

长远好处：

wangkuiyi left a comment

Choose a reason for hiding this comment

gangliao commented Dec 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gangliao Dec 27, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gangliao commented Jan 5, 2017

reyoung left a comment

Choose a reason for hiding this comment

reyoung commented Jan 6, 2017 • edited Loading

gangliao commented Jan 6, 2017

gangliao commented Jan 6, 2017

gangliao commented Dec 27, 2016 •

edited

Loading

gangliao Dec 27, 2016 •

edited

Loading

reyoung commented Jan 6, 2017 •

edited

Loading