nccl.h not found when compiling from source

[yu239-zz]

See the reopened #5035.

[helinwang]

CC: @tonyyang-svail @dzhwinter could you guys take a look, we probably should support compiling without docker.

[helinwang]

Hi @yu239 , we switched to nccl2, please install the dependency using sudo apt-get install libnccl2 libnccl-dev (more info is here: http://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html).

nccl2 is closed source comparing to nccl1, so we can not use cmake to download the source and compile. Maybe manually install it using apt-get is the best solution.

[typhoonzero]

@helinwang Is it necessary to add a build option to switch off nccl dependency?

[dzhwinter]

The nccl2 is not open sourced anymore, NVIDIA provide the cuda docker image with ppa(the apt source url) included inside, so we can make a apt install command. @helinwang
For build paddle out of docker, you need to install nccl2 like cudnn manually. Here is the nccl2 download link https://developer.nvidia.com/nccl

[luotao1]

Should we remain nccl1?

• If we don't remain nccl1, can we remove cmake/external/nccl.cmake, which will make users confused?
• I find that nccl2 needs cuda 8.0: http://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html#softreq
  
  If we don't remain nccl1, how about the pip install version cuda7.5_cudnn5_avx_mkl?

[wangkuiyi]

Does our codebase (develop branch) still depend on NCCL 1? If not, let us remove nccl.cmake. @luotao1

[luotao1]

Our codebase still works well with NCCL 1, and @dzhwinter will update the nccl.cmake later to get the compatibility of the NCCL 1 and 2.

[dzhwinter]

In our codebase, we provide NCCL as a DSO(Dynamic Shared Library) library. It means that we only use a nccl.h to compile, no more static library is depended.

According to the NCCL installl guide https://docs.nvidia.com/deeplearning/sdk/pdf/NCCL-Installation-Guide.pdf , we have the dependency relation below.

nccl2.1.4(latest) -> cuda9.0 or higher
nccl2.1.2 -> cuda8.0
nccl1.x -> cuda7.0 or higher

To make our Multi-GPU supported in more platform, we still need the nccl1 to compatible with older CUDA version.