port improvements from vchuravy/NCCL.jl

[vchuravy]

TODO

• figure out how to test examples
• add tests for new functionality

[vchuravy]

The "right" way of doing this is:

    lcomm = MPI.Comm_split_type(mpicomm, MPI.MPI_COMM_TYPE_SHARED,
                                MPI.Comm_rank(mpicomm))
    CUDAnative.device!(MPI.Comm_rank(lcomm))

For NCCL to work best all devices need to be visible to all MPI processes and there should only be a maxium of n local ranks per n devices

[vchuravy]

Following the MPI.jl convention to lowercase Julia specific implementations

[maleadt]

Interesting

192 Test Failed at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:9
193   Expression: size(comms[1]) == 2
194    Evaluated: 8 == 2
195 Stacktrace:
196  [1] macro expansion at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:9 [inlined]
197  [2] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
198  [3] macro expansion at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:6 [inlined]
199  [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
200  [5] top-level scope at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:5

and ensuing

213 Reduce!: Error During Test at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:66
214   Got exception outside of a @test
215   CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)
216   Stacktrace:
217    [1] macro expansion at /builds/JuliaGPU/NCCL.jl/.julia/packages/CUDAdrv/3EzC1/src/error.jl:123 [inlined]
218    [2] cuMemcpyDtoH_v2(::Ptr{Float64}, ::CuPtr{Float64}, ::Int64) at /builds/JuliaGPU/NCCL.jl/.julia/packages/CUDAdrv/3EzC1/src/libcuda.jl:444
219    [3] #unsafe_copyto!#5 at /builds/JuliaGPU/NCCL.jl/.julia/packages/CUDAdrv/3EzC1/src/memory.jl:285 [inlined]
220    [4] unsafe_copyto! at /builds/JuliaGPU/NCCL.jl/.julia/packages/CUDAdrv/3EzC1/src/memory.jl:278 [inlined]
221    [5] copyto!(::Array{Float64,1}, ::Int64, ::CuArray{Float64,1,Nothing}, ::Int64, ::Int64) at /builds/JuliaGPU/NCCL.jl/.julia/packages/CuArrays/7z7MV/src/array.jl:265
222    [6] copyto! at /builds/JuliaGPU/NCCL.jl/.julia/packages/GPUArrays/0lvhc/src/abstractarray.jl:118 [inlined]
223    [7] collect(::CuArray{Float64,1,Nothing}) at /builds/JuliaGPU/NCCL.jl/.julia/packages/CuArrays/7z7MV/src/array.jl:245
224    [8] macro expansion at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:87 [inlined]
225    [9] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
226    [10] macro expansion at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:67 [inlined]
227    [11] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
228    [12] top-level scope at /builds/JuliaGPU/NCCL.jl/test/runtests.jl:5
229    [13] include at ./boot.jl:317 [inlined]
230    [14] include_relative(::Module, ::String) at ./loading.jl:1044
231    [15] include(::Module, ::String) at ./sysimg.jl:29
232    [16] include(::String) at ./client.jl:392
233    [17] top-level scope at none:0
234    [18] eval(::Module, ::Any) at ./boot.jl:319
235    [19] exec_options(::Base.JLOptions) at ./client.jl:243
236    [20] _start() at ./client.jl:425

[maleadt]

Most of the functionality here has been included already, so I think we can close this.