add tensorrt #9891

Superjomn · 2018-04-13T05:11:06Z

This is a naive test for TensorRT library integration with Paddle.

…_tensorrt

dependency has been installed in docker image.

Superjomn · 2018-04-15T11:21:21Z

CMakeLists.txt

@@ -39,6 +39,7 @@ option(WITH_GPU         "Compile PaddlePaddle with NVIDIA GPU"          ${CUDA_F
 option(WITH_AMD_GPU     "Compile PaddlePaddle with AMD GPU"             OFF)
 option(WITH_AVX         "Compile PaddlePaddle with AVX intrinsics"      ${AVX_FOUND})
 option(WITH_MKL         "Compile PaddlePaddle with MKL support."        ${AVX_FOUND})
+option(WITH_TENSORRT    "Compile PaddlePaddle with TensorRT support."   ON)


Will turn off this latter. Changing TeamCity config should more cautious.

luotao1 · 2018-04-16T02:59:16Z

Dockerfile

@@ -45,6 +45,12 @@ ENV PATH=${PATH}:${GOROOT}/bin:${GOPATH}/bin
 # install glide
 RUN curl -s -q https://glide.sh/get | sh

+# Install TensorRT
+RUN wget -qO- http://paddlepaddledeps.bj.bcebos.com/TensorRT-4.0.0.3.Ubuntu-16.04.4.x86_64-gnu.cuda-8.0.cudnn7.0.tar.gz | \


这里的tar.gz和官网下载的有所不同，只包含了include和lib包，目的是为了让包减少2/3的大小，从而节省下载时间。需要加comment说明一下。

根据这个包下载的，里面还有targets目录，该目录可以在打包的时候去掉。

TensorRT ├── include ├── lib └── targets

这里使用的NvInfer.h，是对原来的版本做了一点修改的，不然会报错。可以写一个issue说明下报错情况，然后在这里加一个comment。

luotao1 · 2018-04-16T03:00:29Z

Dockerfile

@@ -57,8 +63,7 @@ RUN localedef -i en_US -f UTF-8 en_US.UTF-8
 # specify sphinx version as 1.5.6 and remove -U option for [pip install -U
 # sphinx-rtd-theme] since -U option will cause sphinx being updated to newest
 # version(1.7.1 for now), which causes building documentation failed.
-RUN pip install --upgrade pip && \
-    pip install -U wheel && \
+RUN pip install -U wheel && \


#9926 merge后，这里需要更新下。

luotao1 · 2018-04-16T03:27:56Z

paddle/fluid/platform/dynload/CMakeLists.txt

@@ -1,6 +1,6 @@
 cc_library(dynamic_loader SRCS dynamic_loader.cc DEPS glog gflags enforce)

-list(APPEND CUDA_SRCS cublas.cc cudnn.cc curand.cc nccl.cc)
+list(APPEND CUDA_SRCS cublas.cc cudnn.cc curand.cc nccl.cc tensorrt.cc)


这里需要加编译选项来选择是否添加tensorrt.cc

luotao1 · 2018-04-16T03:33:01Z

paddle/fluid/inference/tensorrt/test_tensorrt.cc

+
+// Fix the dynload issue, the following two API are implemented in TensorRT's
+// header file, cannot load from the dynamic library. So create our own
+// implementation and directly trigger the method from the dynamic library.


58-60的注释需要更新下：

fix the dynload issue: 请问issue在哪儿？

API-》APIs

but can not loaded from

luotao1 · 2018-04-16T03:41:42Z

CMakeLists.txt

@@ -179,6 +180,7 @@ set(EXTERNAL_LIBS

 if(WITH_GPU)
    include(cuda)
+    set(WITH_TENSORRT ON)


这句可以去掉。

luotao1 · 2018-04-16T03:44:57Z

Dockerfile

+RUN wget -qO- http://paddlepaddledeps.bj.bcebos.com/TensorRT-4.0.0.3.Ubuntu-16.04.4.x86_64-gnu.cuda-8.0.cudnn7.0.tar.gz | \
+    tar -xz -C /usr/local && \
+    cp -rf /usr/local/TensorRT/include /usr/local && \
+    cp -rf /usr/local/TensorRT/lib /usr/local


缺少类似cudnn.cmake这样的tensorrt.cmake，用户无法用自定义路径的安装形式，可以之后的PR补充。

目前直接安装在/usr/local/include和/usr/local/lib里，应该像cuda和go一样，有一个自己的目录，可以和1一起之后的PR改进。

现在直接复制到 /usr下了，可以后续pr改下

luotao1 · 2018-04-16T04:18:46Z

编译成功，运行单测存在：

75: unknown file: Failure
75: C++ exception with description "Failed to find dynamic library: libnvinfer.so ( libnvinfer.so: cannot open shared object file: No such file or directory ) 
75:  Please specify its path correctly using following ways: 
75:  Method. set environment variable LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on Mac OS. 
75:  For instance, issue command: export LD_LIBRARY_PATH=... 
75:  Note: After Mac OS 10.11, using the DYLD_LIBRARY_PATH is impossible unless System Integrity Protection (SIP) is disabled. at [/Paddle/paddle/fluid/platform/dynload/dynamic_loader.cc:133]
75: PaddlePaddle Call Stacks: 
75: 0             0x432bf9p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 761
75: 1             0x4b2367p
75: 2             0x4b3032p paddle::platform::dynload::GetTensorRtDsoHandle() + 98
75: 3             0x435371p void std::__once_call_impl<std::_Bind_simple<decltype (createInferBuilder_INTERNAL({parm#1}...)) paddle::platform::dynload::DynLoad__createInferBuilder_INTERNAL::operator()<nvinfer1::ILogger*, int>(nvinfer1::ILogger*, int)::{lambda()#1} ()> >() + 33
75: 4       0x7f0d873a1a99p
75: 5             0x42ce5fp createInferBuilder(nvinfer1::ILogger&) + 111
75: 6             0x42d028p CreateNetwork() + 72
75: 7             0x42f138p TensorrtTest_BasicFunction_Test::TestBody() + 40
75: 8             0x4d7da3p void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 67
75: 9             0x4cdb8ap testing::Test::Run() + 186
75: 10            0x4cdcd8p testing::TestInfo::Run() + 280
75: 11            0x4cdde5p testing::TestCase::Run() + 229
75: 12            0x4d0287p testing::internal::UnitTestImpl::RunAllTests() + 583
75: 13            0x4d05b9p testing::UnitTest::Run() + 89
75: 14            0x42c349p main + 329
75: 15      0x7f0d86645830p __libc_start_main + 240
75: 16            0x42c989p _start + 41
75: " thrown in the test body.
75: [  FAILED  ] TensorrtTest.BasicFunction (1 ms)
75: [----------] 1 test from TensorrtTest (1 ms total)
75: 
75: [----------] Global test environment tear-down
75: [==========] 1 test from 1 test case ran. (1 ms total)
75: [  PASSED  ] 0 tests.
75: [  FAILED  ] 1 test, listed below:
75: [  FAILED  ] TensorrtTest.BasicFunction
75: 
75:  1 FAILED TEST
1/1 Test #75: test_tensorrt ....................***Failed    5.52 sec

0% tests passed, 1 tests failed out of 1

将/usr/local/lib下的相关so文件拷贝到/usr/lib下即可。

luotao1 · 2018-04-16T05:54:27Z

paddle/utils/DynamicLoader.h

@@ -58,3 +58,11 @@ void GetWarpCTCDsoHandle(void** dso_handle);
 *
 */
 void GetLapackDsoHandle(void** dso_handle);


DynamicLoader.h 可以不用修改，是老paddle用的。

luotao1

lgtm

Superjomn added 10 commits April 13, 2018 13:05

add tensorrt

a3140d3

set tensorrt on as default

a60189f

add cudnn dependency

b95d819

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into fea/add…

87fc090

…_tensorrt

nvtest

8dda580

add tensorrt dynamic loader

92480b5

add tensorrt as dyload

5891896

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into fea/add…

1b475b3

…_tensorrt

finish test

9d617b8

remove tensorrt.cmake

0e8e85f

dependency has been installed in docker image.

Superjomn force-pushed the fea/add_tensorrt branch from b7cd337 to 0e8e85f Compare April 15, 2018 11:19

Superjomn commented Apr 15, 2018

View reviewed changes

Superjomn requested review from Xreki and luotao1 April 15, 2018 11:23

Superjomn added 2 commits April 15, 2018 19:24

fix pip upgrade pip error

63b6a74

add flag definition for tensorrt_dir

d492547

Xreki added the 预测原名Inference，包含Capi预测问题等 label Apr 16, 2018

luotao1 reviewed Apr 16, 2018

View reviewed changes

update

5132a2b

Superjomn force-pushed the fea/add_tensorrt branch from 0b5ce6c to 5132a2b Compare April 16, 2018 04:44

change cmake config

1fe9f63

Superjomn force-pushed the fea/add_tensorrt branch from d6d030b to 1fe9f63 Compare April 16, 2018 05:49

luotao1 reviewed Apr 16, 2018

View reviewed changes

turn WITH_TENSORRT OFF

d3a0c23

luotao1 approved these changes Apr 16, 2018

View reviewed changes

Superjomn merged commit 1866597 into PaddlePaddle:develop Apr 16, 2018

Superjomn deleted the fea/add_tensorrt branch April 16, 2018 12:33

luotao1 mentioned this pull request Apr 16, 2018

auto find tensorrt library and install in user root #9958

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add tensorrt #9891

add tensorrt #9891

Superjomn commented Apr 13, 2018 •

edited

Loading

Superjomn Apr 15, 2018

luotao1 Apr 16, 2018

luotao1 Apr 16, 2018

luotao1 Apr 16, 2018

Superjomn Apr 16, 2018

luotao1 Apr 16, 2018

Superjomn Apr 16, 2018

luotao1 Apr 16, 2018

Superjomn Apr 16, 2018

luotao1 Apr 16, 2018

Superjomn Apr 16, 2018 •

edited

Loading

luotao1 commented Apr 16, 2018

luotao1 Apr 16, 2018

luotao1 left a comment

add tensorrt #9891

add tensorrt #9891

Conversation

Superjomn commented Apr 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Superjomn Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

luotao1 commented Apr 16, 2018

Choose a reason for hiding this comment

luotao1 left a comment

Choose a reason for hiding this comment

Superjomn commented Apr 13, 2018 •

edited

Loading

Superjomn Apr 16, 2018 •

edited

Loading