RuntimeError: function_attributes(): after cudaFuncGetAttributes: invalid device function

[BigFishMaster]

when running se-resnext-50 networking to train imagenet-1000 dataset, I got an ERROR as below.

Traceback (most recent call last):
  File "se_resnext50_parallel.py", line 237, in <module>
    train(learning_rate=0.1, batch_size=256, num_passes=120, init_model=None)
  File "se_resnext50_parallel.py", line 174, in train
    exe.run(fluid.default_startup_program())
  File "/ssd1/yancanxiang/se_resnext50_parallel_MSRA_imagenet1000/python-gcc482-paddle/lib/python2.7/site-packages/paddle/fluid/executor.py", line 373, in run
    self.executor.run(program.desc, scope, 0, True, True)
RuntimeError: function_attributes(): after cudaFuncGetAttributes: invalid device function

I use the paddle-wheel from folows:
paddle url：https://paddleci.ngrok.io/viewType.html?buildTypeId=Manylinux1_Cuda80cudnn5cp27cp27mu
wheel：https://paddleci.ngrok.io/repository/download/Manylinux1_Cuda80cudnn5cp27cp27mu/31050:id/paddlepaddle_gpu-0.11.0-cp27-cp27mu-linux_x86_64.whl

[typhoonzero]

cuda error "invalid device function" usually means your CUDA library does not fit your hardware and driver.

dup: #7820 #6524 #5674 #5629 #34

[BigFishMaster]

Ok, I will check this. @typhoonzero

[BigFishMaster]

@typhoonzero I tried (cuda8 + cudnn5/cudnn5.1 + driver384.81/384.66) 4 combination, but failed.

[qingqing01]

@BigFishMaster From doc http://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

Tesla V100 should use CUDA 9.0 or later. There is no problem when I run Fluid model with CUDA 9.0 + cuDNN V7.

[BigFishMaster]

@qingqing01 I meet a gcc problem:

ImportError: /opt/compiler/gcc-4.8.2/lib/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /ssd1/yancanxiang/se_resnext50_parallel_MSRA_imagenet1000/python-gcc482-paddle/lib/python2.7/site-packages/paddle/fluid/core.so)

[BigFishMaster]

I tried jumbo with gcc5.3 and paddle is ok. Thanks @qingqing01