[Build] Unable to build ONNX Runtime against CUDA 12.5

[mc-nv]

Describe the issue

Unable to build ONNX Runtime against CUDA 12.5.

Urgency

It's quiet important as it may impact Triton 24.06 scope.

Target platform

Windows

Build branch

rel-1.18.0

Build script

./build.sh --config Release --skip_submodule_sync --parallel --build_shared_lib     --build_dir /workspace/build --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES='60;61;70;75;80;86;90'  --update --build --use_cuda --cuda_home "/usr/local/cuda" --cudnn_home "/usr/local/cudnn-9.1/cuda" --use_tensorrt --use_tensorrt_builtin_parser --tensorrt_home "/usr/src/tensorrt" --allow_running_as_root

Error / output

#16 98.94 [ 73%] Building CXX object CMakeFiles/onnxruntime_providers.dir/workspace/onnxruntime/onnxruntime/core/providers/cpu/ml/scaler.cc.o
#16 99.04 /workspace/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu(68): error: identifier "FLT_MAX" is undefined
#16 99.04     float threadData(-FLT_MAX);
#16 99.04                       ^
#16 99.04 
#16 99.12 1 error detected in the compilation of "/workspace/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu".

Visual Studio Version

No response

GCC / Compiler Version

No response

[mc-nv]

cc: @pranavsharma

[chilo-ms]

+@tianleiwu

[yf711]

Hi @mc-nv Is your test branch including this commit? I tried building latest main with cuda 12.5 and didn't repro this issue

[mc-nv]

Is your test branch including this commit? I tried building latest main with cuda 12.5 and didn't repro this issue

@yf711 my build is based on rel-1.18.0

[yf711]

Is your test branch including this commit? I tried building latest main with cuda 12.5 and didn't repro this issue

@yf711 my build is based on rel-1.18.0

I see. rel-1.18.1 will include this fix and support cuda 12.5, which is planning to be released on 6/17

[mc-nv]

Is your test branch including this commit? I tried building latest main with cuda 12.5 and didn't repro this issue

@yf711 my build is based on rel-1.18.0

I see. rel-1.18.1 will include this fix and support cuda 12.5, which is planning to be released on 6/17

Can we have that branch available?
I don't see any 1.18.1 changes in the working tree to be able refer to.
https://github.com/microsoft/onnxruntime/branches/all?query=1.18

[yf711]

Is your test branch including this commit? I tried building latest main with cuda 12.5 and didn't repro this issue

@yf711 my build is based on rel-1.18.0

I see. rel-1.18.1 will include this fix and support cuda 12.5, which is planning to be released on 6/17

Can we have that branch available? I don't see any 1.18.1 changes in the working tree to be able refer to. https://github.com/microsoft/onnxruntime/branches/all?query=1.18

The branch will be available for testing by the end of next week (6/14)
and I just get noticed that the scheduled 6/17 release date will be delayed for a bit. Will give update once it's determined

[tianleiwu]

Use branch rel-1.18.1, and use --compile_no_warning_as_error in build.sh command line shall fix it.