Windows CI failures #1211

ptheywood · 2024-06-11T10:50:20Z

At some point between 2024-04-26 and 2024-06-11 our Windows CI has stopped working, for CUDA 11.8 and CUDA 12.3 builds (11.0 builds are fine).

In both cases, CMake could not find CUDA language support:

CMake Warning at cmake/CheckCompilerFunctionality.cmake:23 (message):
  CUDA Language Support Not Found

This is after CUDA was installed successfully via network installer at the appropriate location and path updated

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3

Probably worth debugging in a separate mcuh smaller repo to avoid CI spam / long CI times when it does work.
i.e. a fork of flamegpu with most CI disabled, or a fork of https://github.com/ptheywood/cuda-cmake-github-actions (the repo I used to develop the cuda install scripts in the first place).

Things to check could be:

nvcc is correctly on the path just before the cmake configuration phase
nvcc can manually compile a test program without CMake (which is what cmake does)
Check which versions of packages have changes in the github actions runner between those dates, try and reproduce on a local windows build
Output the CMake log files when the step fails, to see what error messages were provided during CUDA language support checking.

The text was updated successfully, but these errors were encountered:

ptheywood · 2024-07-09T15:12:50Z

Possibly just a bad CI image, failures were using 20240603.1.0 as in actions/runner-images#10004.

I've retriggered one of the failed workflows to see if the issues has magically gone away again.

https://github.com/FLAMEGPU/FLAMEGPU2/actions/runs/9463757119/job/27223363960

Not fixed by 20240630.1.0, so not the same issue. Probably still caused by some change in the base image.

Will be simpler to debug in a fork or less complex repository (i.e. ptheywood/cuda-cmake-github-actionsafter some updates)

ptheywood · 2024-07-12T16:22:43Z

From triggering a CI run on the above repo, it appears that CUDA is deciding the version of visual studio is incompatible when CMake is triggering the test build:

D:\a\cuda-cmake-github-actions\cuda-cmake-github-actions\build\CMakeFiles\CMakeScratch\TryCompile-um0ksd>"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\nvcc.exe"  --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64" -x cu    -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include"     --keep-dir cmTC_98de8\x64\Debug  -maxrregcount=0  --machine 64 --compile -cudart static -Xcompiler="/EHsc -Zi -Ob0" -g  -D_WINDOWS -D"CMAKE_INTDIR=\"Debug\"" -D_MBCS -D"CMAKE_INTDIR=\"Debug\"" -Xcompiler "/EHsc /W3 /nologo /Od /FdcmTC_98de8.dir\Debug\vc143.pdb /FS /Zi /RTC1 /MDd /GR" -o cmTC_98de8.dir\Debug\main.obj "D:\a\cuda-cmake-github-actions\cuda-cmake-github-actions\build\CMakeFiles\CMakeScratch\TryCompile-um0ksd\main.cu" 
    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include\crt/host_config.h(153): fatal error C1189: #error:  -- unsupported Microsoft Visual Studio version! Only the versions between 2017 and 2022 (inclusive) are supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk. [D:\a\cuda-cmake-github-actions\cuda-cmake-github-actions\build\CMakeFiles\CMakeScratch\TryCompile-um0ksd\cmTC_98de8.vcxproj]
      main.cu

However it says 2022 inclusive should be fine.

But it seems that this might be caused by Microsoft changing the numeric version to 19.40, but nvidia assumed that 1940 would be visual studio 202X.

CUDA 12.5 should not get angry, but older ones still will.

We need to tell CMake to use --allow-unsupported-compiler when testing CUDA detection, using -DCMAKE_CUDA_FLAGS="--allow-unsupported-compiler" might work to fix CI, but the issue will still be present for any windows users with the recent vs2022, and CUDA < 12.5. Can probably append conditionally set it in our CMake, but might be a bit awkward to fit in between C++ and CUDA language support checking.

ptheywood · 2024-07-12T16:39:37Z

Might also require a CMake >= 3.29.4 for compiler detection to use the user provided flags / 3.29.4 will fix the issue anyway (but users will need to specify allowing unsupported compilers too) if I'm understanding this correctly:

https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9546

So it might just be that on windows, if using MSVC >= whichever version this is, and CUDA < ~~12.5~~ 12.4 you must also use CMake >= 3.29.4 and specify allowing unsupported compilers.

We can probably emit an appropriate error message somewhere like we do for other compiler issues if all these conditions are met (and a compiler check fails).
Probably as an nested if in here.

FLAMEGPU2/cmake/CheckCompilerFunctionality.cmake

Lines 22 to 26 in 4e02604

    
           if(NOT CMAKE_CUDA_COMPILER) 
        
               message(WARNING "CUDA Language Support Not Found") 
        
               set(FLAMEGPU_CheckCompilerFunctionality_RESULT "NO" PARENT_SCOPE) 
        
               return() 
        
           endif()

Edit

Yep -DCMAKE_CUDA_FLAGS="--allow-unsupported-compiler" at cmake configuration time fixes my standalone test, so:

Add a warning if cmake language detection fails and OS is windows and MSVC versions is > the bad one and CUDA < ~~12.5~~ 12.4, emit a warning suggesting users reconfigure with -DCMAKE_CUDA_FLAGS="--allow-unsupported-compiler" due to an incorrect assumption in CUDA's compiler version checking

Not certain if this also requires newer CMake or not, would need to do a local test really. CMake in the windows action image is currently 3.30.

add -DCMAKE_CUDA_FLAGS="--allow-unsupported-compiler" to our CI workflows on windows (possibly just for specific CUDA versions, but could probably just allow it for all of them if we want to keep it simpler).

ptheywood · 2024-07-15T14:17:09Z

Final CI runs I'd set going on Friday confirm that CUDA <= 12.3 with -DCMAKE_CUDA_FLAGS="--allow-unsupported-compiler" does require CMake >= 3.29.4 (3.29.3 failes, 3.29.6 passes, which are the closest not yanked from pip)

So (ideally) the warning needs to be emitted when:

CUDA language support detection fails
And using MSVC
And MSVC_VERSION >= 1940 (we could err on the side of caution and also ensure MSVC_VERSION < 1950, but this might fail if VS2022 has many more versions
And the attempted CUDA is < 12.4 (if this is accessible at during finding, not sure it is).
And the user has not already set --allow-unsupported-compiler (i.e. the suggest fix of setting this would not work).
If CMAKE_VERSION < 3.29.4 then the warning will need to be different.

It should tell users that:

There is a known issue with CUDA < 12.4 and VS 2022 >= 1940, with several options
They must specify -DCMAKE_CUDA_FLAGS="--allow-unsupported-compiler"
- And update to CMake >= 3.29.4 if required
Update to CUDA >= 12.4 (probably not ideal with RTC perf)
Downgrade Visual Studio 2022 to whichever version is 1939 (not sure how trivial this is, prolly better to update cmake and set the flag)

Could also be worth adding to installation instructions for MSVC users or somewhere. It's not an us issue, it's a generic CUDA < 12.4 && MSVC_VERISON >= 1940 (but still VS 2022) issue.

It would be worth updating our CI to include CUDA 11.x, 12.0 & 12.5 builds at the same time to cover this.

ptheywood · 2024-08-14T17:40:47Z

Possible problem:

CMake's fix to pass -allow-unsupported-compiler to MSVC in 3.29.4 works when using project(main LANGUAGES CXX CUDA).

It does not appear to work when using check_language(CUDA), at least not in CMake 3.30.

This is how we optionally enable CUDA (to allow docs only builds on CI for instance) and is required if we want to even stand a chance of providing a helpful error message about this.

From looking at the generated build/CMakeFiles/CheckCUDA/CMakeCache.txt, when configuring the parent project with -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler" (which works when using project languages), the sub-project's cache contains:

CMAKE_CUDA_FLAGS:STRING=-D_WINDOWS -Xcompiler=" /EHsc"

CMakeFiles/CheckCUDA/CMakeFiles/CMakeConfigureLog.yaml includes the following sections, showing it encounters the unsupported compiler error during 2 checks which use try_compile

"Detecting CUDA compiler ABI info"
"Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3/bin/nvcc.exe"

because it is not passing -allow-unsupported-compiler during some calls to NVCC.

My previous standalone project CI runs which showed that would be a fix did not use check_language(CUDA), hence -DCMAKE_CUDA_FLAGS=-allow-unsupported-compiler worked. It won't fix CUDA <=12.3 on our CI.

I'll report this upstream when I have time.

For FLAME GPU 2 short term, we have quite limited options:

Only have CUDA 12.4+ Windows CI - but this will prevent building older wheels, and not help windows users.
Have a way to opt-out of our use of check_language(CUDA)

Either an extra cmake configuration option, or after check_language(CXX), if MSVC_VER is bad, just enable CUDA and hope for the best (or then manually check ourselves for a nice error message)
3 places this workaround would be needed
This could be amended to not be skipped for future CMake's which don't have this issue.

Wait for this to be fixed upstream, and then on windows heavily encourage people to use newest cmake

  -
    kind: "try_compile-v1"
    backtrace:
      - "C:/Program Files/CMake/share/cmake-3.30/Modules/CMakeDetermineCompilerABI.cmake:74 (try_compile)"
      - "C:/Program Files/CMake/share/cmake-3.30/Modules/CMakeTestCUDACompiler.cmake:19 (CMAKE_DETERMINE_COMPILER_ABI)"
      - "CMakeLists.txt:3 (project)"
    checks:
      - "Detecting CUDA compiler ABI info"
    directories:
      source: "C:/Users/ptheywood/code/ptheywood/cuda-cmake-github-actions/build/CMakeFiles/CheckCUDA/CMakeFiles/CMakeScratch/TryCompile-dqp1y3"
      binary: "C:/Users/ptheywood/code/ptheywood/cuda-cmake-github-actions/build/CMakeFiles/CheckCUDA/CMakeFiles/CMakeScratch/TryCompile-dqp1y3"
    cmakeVariables:
      CMAKE_CUDA_ARCHITECTURES: "52"
      CMAKE_CUDA_FLAGS: "-D_WINDOWS -Xcompiler=\" /EHsc\""
      CMAKE_CUDA_FLAGS_DEBUG: "-Xcompiler=\" -Ob0 -Od /RTC1\""
      CMAKE_CUDA_RUNTIME_LIBRARY: "Static"
      CMAKE_EXE_LINKER_FLAGS: ""
      CMAKE_MODULE_PATH: ""
    buildResult:
      variable: "CMAKE_CUDA_ABI_COMPILED"
      cached: true
      stdout: |
        Change Dir: 'C:/Users/ptheywood/code/ptheywood/cuda-cmake-github-actions/build/CMakeFiles/CheckCUDA/CMakeFiles/CMakeScratch/TryCompile-dqp1y3'
        
        Run Build Command(s): "C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuild.exe" cmTC_225fb.vcxproj /p:Configuration=Debug /p:Platform=x64 /p:VisualStudioVersion=17.0 /v:n
        MSBuild version 17.10.4+10fbfbf2e for .NET Framework
        Build started 14/08/2024 18:00:23.
        
        Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" on node 1 (default targets).
        PrepareForBuild:
          Creating directory "cmTC_225fb.dir\\Debug\\".
          Structured output is enabled. The formatting of compiler diagnostics will reflect the error hierarchy. See https://aka.ms/cpp/structured-output for more details.
          Creating directory "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\Debug\\".
          Creating directory "cmTC_225fb.dir\\Debug\\cmTC_225fb.tlog\\".
        InitializeBuildStatus:
          Creating "cmTC_225fb.dir\\Debug\\cmTC_225fb.tlog\\unsuccessfulbuild" because "AlwaysCreate" was specified.
          Touching "cmTC_225fb.dir\\Debug\\cmTC_225fb.tlog\\unsuccessfulbuild".
        AddCudaCompileDeps:
          C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64\\cl.exe /E /nologo /showIncludes /TP /D__CUDACC__ /D__CUDACC_VER_MAJOR__=12 /D__CUDACC_VER_MINOR__=3 /D_WINDOWS /DCMAKE_INTDIR="Debug" /D_MBCS /DCMAKE_INTDIR="Debug" /I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin" /I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include" /I. /FIcuda_runtime.h /c "C:\\Program Files\\CMake\\share\\cmake-3.30\\Modules\\CMakeCUDACompilerABI.cu" 
        Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (1) is building "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (1:2) on node 1 (CudaBuildCore target(s)).
        CudaBuildCore:
          Compiling CUDA source file ..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\Program Files\\CMake\\share\\cmake-3.30\\Modules\\CMakeCUDACompilerABI.cu...
          cmd.exe /C "C:\\Users\\ptheywood\\AppData\\Local\\Temp\\tmpbe8d517072574a7eaacbe4915d7bdafb.cmd"
          "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_225fb\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi -v" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_225fb.dir\\Debug\\vc143.pdb" -o cmTC_225fb.dir\\Debug\\CMakeCUDACompilerABI.obj "C:\\Program Files\\CMake\\share\\cmake-3.30\\Modules\\CMakeCUDACompilerABI.cu"
          
          C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3>"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_225fb\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi -v" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_225fb.dir\\Debug\\vc143.pdb" -o cmTC_225fb.dir\\Debug\\CMakeCUDACompilerABI.obj "C:\\Program Files\\CMake\\share\\cmake-3.30\\Modules\\CMakeCUDACompilerABI.cu" 
        cl : Command line warning D9002: ignoring unknown option '-v' [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj]
          CMakeCUDACompilerABI.cu
        C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include\\crt/host_config.h(164): fatal error C1189: #error:  -- unsupported Microsoft Visual Studio version! Only the versions between 2017 and 2022 (inclusive) are supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj]
          CMakeCUDACompilerABI.cu
        C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Microsoft\\VC\\v170\\BuildCustomizations\\CUDA 12.3.targets(799,9): error MSB3721: The command ""C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_225fb\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi -v" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_225fb.dir\\Debug\\vc143.pdb" -o cmTC_225fb.dir\\Debug\\CMakeCUDACompilerABI.obj "C:\\Program Files\\CMake\\share\\cmake-3.30\\Modules\\CMakeCUDACompilerABI.cu"" exited with code 2. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj]
        Done Building Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (CudaBuildCore target(s)) -- FAILED.
        Done Building Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (default targets) -- FAILED.
        
        Build FAILED.
        
        "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (default target) (1) ->
        "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (CudaBuildCore target) (1:2) ->
        (CudaBuildCore target) -> 
          cl : Command line warning D9002: ignoring unknown option '-v' [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj]
        
        
        "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (default target) (1) ->
        "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj" (CudaBuildCore target) (1:2) ->
        (CudaBuildCore target) -> 
          C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include\\crt/host_config.h(164): fatal error C1189: #error:  -- unsupported Microsoft Visual Studio version! Only the versions between 2017 and 2022 (inclusive) are supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj]
          C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Microsoft\\VC\\v170\\BuildCustomizations\\CUDA 12.3.targets(799,9): error MSB3721: The command ""C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_225fb\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi -v" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_225fb.dir\\Debug\\vc143.pdb" -o cmTC_225fb.dir\\Debug\\CMakeCUDACompilerABI.obj "C:\\Program Files\\CMake\\share\\cmake-3.30\\Modules\\CMakeCUDACompilerABI.cu"" exited with code 2. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-dqp1y3\\cmTC_225fb.vcxproj]
        
            1 Warning(s)
            2 Error(s)
        
        Time Elapsed 00:00:01.23
        
      exitCode: 1
  -
    kind: "try_compile-v1"
    backtrace:
      - "C:/Program Files/CMake/share/cmake-3.30/Modules/CMakeTestCUDACompiler.cmake:48 (try_compile)"
      - "CMakeLists.txt:3 (project)"
    checks:
      - "Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3/bin/nvcc.exe"
    directories:
      source: "C:/Users/ptheywood/code/ptheywood/cuda-cmake-github-actions/build/CMakeFiles/CheckCUDA/CMakeFiles/CMakeScratch/TryCompile-0pdlvw"
      binary: "C:/Users/ptheywood/code/ptheywood/cuda-cmake-github-actions/build/CMakeFiles/CheckCUDA/CMakeFiles/CMakeScratch/TryCompile-0pdlvw"
    cmakeVariables:
      CMAKE_CUDA_ARCHITECTURES: "52"
      CMAKE_CUDA_FLAGS: "-D_WINDOWS -Xcompiler=\" /EHsc\""
      CMAKE_CUDA_FLAGS_DEBUG: "-Xcompiler=\" -Ob0 -Od /RTC1\""
      CMAKE_EXE_LINKER_FLAGS: ""
      CMAKE_MODULE_PATH: ""
    buildResult:
      variable: "CMAKE_CUDA_COMPILER_WORKS"
      cached: true
      stdout: |
        Change Dir: 'C:/Users/ptheywood/code/ptheywood/cuda-cmake-github-actions/build/CMakeFiles/CheckCUDA/CMakeFiles/CMakeScratch/TryCompile-0pdlvw'
        
        Run Build Command(s): "C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuild.exe" cmTC_4752f.vcxproj /p:Configuration=Debug /p:Platform=x64 /p:VisualStudioVersion=17.0 /v:n
        MSBuild version 17.10.4+10fbfbf2e for .NET Framework
        Build started 14/08/2024 18:00:25.
        
        Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj" on node 1 (default targets).
        PrepareForBuild:
          Creating directory "cmTC_4752f.dir\\Debug\\".
          Structured output is enabled. The formatting of compiler diagnostics will reflect the error hierarchy. See https://aka.ms/cpp/structured-output for more details.
          Creating directory "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\Debug\\".
          Creating directory "cmTC_4752f.dir\\Debug\\cmTC_4752f.tlog\\".
        InitializeBuildStatus:
          Creating "cmTC_4752f.dir\\Debug\\cmTC_4752f.tlog\\unsuccessfulbuild" because "AlwaysCreate" was specified.
          Touching "cmTC_4752f.dir\\Debug\\cmTC_4752f.tlog\\unsuccessfulbuild".
        AddCudaCompileDeps:
          C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64\\cl.exe /E /nologo /showIncludes /TP /D__CUDACC__ /D__CUDACC_VER_MAJOR__=12 /D__CUDACC_VER_MINOR__=3 /D_WINDOWS /DCMAKE_INTDIR="Debug" /D_MBCS /DCMAKE_INTDIR="Debug" /I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin" /I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include" /I. /FIcuda_runtime.h /c C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\main.cu 
        Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj" (1) is building "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj" (1:2) on node 1 (CudaBuildCore target(s)).
        CudaBuildCore:
          Compiling CUDA source file main.cu...
          cmd.exe /C "C:\\Users\\ptheywood\\AppData\\Local\\Temp\\tmpbd4679752b9044a88fd473a03a6b39ac.cmd"
          "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_4752f\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_4752f.dir\\Debug\\vc143.pdb" -o cmTC_4752f.dir\\Debug\\main.obj "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\main.cu"
          
          C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw>"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_4752f\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_4752f.dir\\Debug\\vc143.pdb" -o cmTC_4752f.dir\\Debug\\main.obj "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\main.cu" 
        C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include\\crt/host_config.h(164): fatal error C1189: #error:  -- unsupported Microsoft Visual Studio version! Only the versions between 2017 and 2022 (inclusive) are supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj]
          main.cu
        C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Microsoft\\VC\\v170\\BuildCustomizations\\CUDA 12.3.targets(799,9): error MSB3721: The command ""C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_4752f\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_4752f.dir\\Debug\\vc143.pdb" -o cmTC_4752f.dir\\Debug\\main.obj "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\main.cu"" exited with code 2. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj]
        Done Building Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj" (CudaBuildCore target(s)) -- FAILED.
        Done Building Project "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj" (default targets) -- FAILED.
        
        Build FAILED.
        
        "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj" (default target) (1) ->
        "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj" (CudaBuildCore target) (1:2) ->
        (CudaBuildCore target) -> 
          C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include\\crt/host_config.h(164): fatal error C1189: #error:  -- unsupported Microsoft Visual Studio version! Only the versions between 2017 and 2022 (inclusive) are supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj]
          C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\MSBuild\\Microsoft\\VC\\v170\\BuildCustomizations\\CUDA 12.3.targets(799,9): error MSB3721: The command ""C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\bin\\nvcc.exe"  --use-local-env -ccbin "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX64\\x64" -x cu    -I"C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.3\\include"     --keep-dir cmTC_4752f\\x64\\Debug  -maxrregcount=0   --machine 64 --compile -cudart static --generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler="/EHsc -Ob0 -Zi" -g  -D_WINDOWS -D"CMAKE_INTDIR=\\"Debug\\"" -D_MBCS -D"CMAKE_INTDIR=\\"Debug\\"" -Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " -Xcompiler "/FdcmTC_4752f.dir\\Debug\\vc143.pdb" -o cmTC_4752f.dir\\Debug\\main.obj "C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\main.cu"" exited with code 2. [C:\\Users\\ptheywood\\code\\ptheywood\\cuda-cmake-github-actions\\build\\CMakeFiles\\CheckCUDA\\CMakeFiles\\CMakeScratch\\TryCompile-0pdlvw\\cmTC_4752f.vcxproj]
        
            0 Warning(s)
            2 Error(s)
        
        Time Elapsed 00:00:01.24
        
      exitCode: 1

These errors occur even if check_language(CUDA) is called on a project with CUDA language already enabled.

ptheywood · 2024-08-14T18:17:40Z

Although -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler" doesn't work, using the env var CUDAFLAGS instead does get used by check_language(CUDA)

So we can fix this in our CI (and suggest this to windows users). Although the warnings we can provide will not be as useful as I'd hoped (Can't find out which CUDA versions CMake tried and failed to use, without reading random files off disk which feels brittle and more effort than it is worth)

Per configuration it can also be set using CMAKE_CUDA_FLAGS_INIT, but this is documented as being intended for toolchain files.

So we can workaround this without needing any fixes upstream (although I might comment on the original issue that if using check_language(CUDA) must use CUDAFLAGS or CMAKE_CUDA_FLAGS_INIT, not CMAKE_CUDA_FLAGS)

Todo

Emit warning to users if MSVC and MSVC_VER VERSION_GREATER_EQUAL "1940" and not CMAKE_CUDA_COMPILER:
- if CMAKE_VERSION VERSION_LESS "3.29.4" tell them to MSVC 1940 might not agree with their CUDA if <= 12.3. They should update CMake and set CUDAFLAGS, (and/)or update CUDA.
- if CMAKE_VERSION VERSION_GREATER_EQUAL "3.29.4" tell them to MSVC 1940 might not agree with their CUDA if <= 12.3. They should set CUDAFLAGS, or update CUDA to 12.4+.
set CUDAFLAGS in .github/workflows/Windows.yml
set CUDAFLAGS in .github/workflows/Windows-test.yml (once the rest is done)
set CUDAFLAGS in .github/workflows/Draft-Release.yml (once the rest is done)
Update regular CI to include 11.0, 11.8, 12.0 and 12.6 (manylinux/draft release will still be 11.2 and 12.0 for widest cuda compat, although 11.8 would potentially be of use too for CUDA 11.x hopper support)
Test the above works for standalone projects too / update in the template repo if necessary (should be fine based on how the template cmake is structured I think)

ptheywood · 2024-08-29T15:34:50Z

My previous fix which worked last week appears to have stopped working on CI after an update 👍

I.e. PR on the vis repo worked, CI after merge failed.

MSVC has updated to 1941 from 1940, which stands out as a possible culprit.

CMake version may also have changed, but that is not obviously logged (we use the default in most CI places, perhaps we want to fix to a known version for stability, at the cost of hiding new problems)

Set a standalone workflow going to see if I can reproduce, otherwise will need to update a local install and see if I can reproduce and find a fix.

ptheywood · 2024-08-29T16:23:43Z

Microsofts standard library distributed with MSVC 1941 now issues a static assertion when CUDA is < 12.4, introduced in
microsoft/STL#4475.

This means that we need to ammend our user-facing error message if MSVC >= 1941 and cuda language support not found, then if they are attempting to use CUDA <= 12.3 then they need to also define _ALLOW_COMPILER_AND_STL_VERSION_MISMATCH.

However, this would need to be defined during CMake CheckLanguage testing (as well as for all targets produced by users).

ptheywood mentioned this issue Aug 22, 2024

Fixes for MSVC 1940 & 1941, CMake 3.30, Github Actions Node16 #1225

Merged

ptheywood closed this as completed in #1225 Sep 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows CI failures #1211

Windows CI failures #1211

ptheywood commented Jun 11, 2024

ptheywood commented Jul 9, 2024 •

edited

Loading

ptheywood commented Jul 12, 2024

ptheywood commented Jul 12, 2024 •

edited

Loading

ptheywood commented Jul 15, 2024 •

edited

Loading

ptheywood commented Aug 14, 2024

ptheywood commented Aug 14, 2024 •

edited

Loading

ptheywood commented Aug 29, 2024

ptheywood commented Aug 29, 2024

Windows CI failures #1211

Windows CI failures #1211

Comments

ptheywood commented Jun 11, 2024

ptheywood commented Jul 9, 2024 • edited Loading

ptheywood commented Jul 12, 2024

ptheywood commented Jul 12, 2024 • edited Loading

Edit

ptheywood commented Jul 15, 2024 • edited Loading

ptheywood commented Aug 14, 2024

ptheywood commented Aug 14, 2024 • edited Loading

Todo

ptheywood commented Aug 29, 2024

ptheywood commented Aug 29, 2024

ptheywood commented Jul 9, 2024 •

edited

Loading

ptheywood commented Jul 12, 2024 •

edited

Loading

ptheywood commented Jul 15, 2024 •

edited

Loading

ptheywood commented Aug 14, 2024 •

edited

Loading