GPUCompiler doesn't work on aarch64-apple-darwin

[giordano]

julia> using LLVMExtra_jll

julia> LLVMExtra_jll.is_available()
true

julia> LLVMExtra_jll.libLLVMExtra
"/Users/mose/.julia/artifacts/548b26fac5815e9981ab5c77077a1bd8db562747/lib/libLLVMExtra-12.dylib"

julia> LLVMExtra_jll.libLLVMExtra_handle
Ptr{Nothing} @0x000000012b03c590

julia> filter(l -> occursin("LLVMExtra", l), readlines(`nm $(LLVMExtra_jll.libLLVMExtra_path)`))
14-element Vector{String}:
 "000000000000b430 T _LLVMExtraAddGenericAnalysisPasses"
 "000000000000b7d0 T _LLVMExtraAddNamedMetadataOperand2"
 "000000000000b2f8 T _LLVMExtraAppendToCompilerUsed"
 "000000000000b1c0 T _LLVMExtraAppendToUsed"
 "000000000000b508 T _LLVMExtraDIScopeGetName"
 "000000000000b52c T _LLVMExtraDumpMetadata"
 "000000000000b6e8 T _LLVMExtraGetMDNodeNumOperands2"
 "000000000000b6f0 T _LLVMExtraGetMDNodeOperands2"
 "000000000000b6bc T _LLVMExtraGetMDString2"
 "000000000000b774 T _LLVMExtraGetNamedMetadataNumOperands2"
 "000000000000b778 T _LLVMExtraGetNamedMetadataOperands2"
 "000000000000b55c T _LLVMExtraPrintMetadataToString"
 "000000000000b7d4 T _LLVMExtraSetInitializer"
 "000000000000b7d8 T _LLVMExtraSetPersonalityFn"

julia> using GPUCompiler
ERROR: InitError: could not load symbol "LLVMExtraInitializeAllTargets":
dlsym(RTLD_DEFAULT, LLVMExtraInitializeAllTargets): symbol not found
Stacktrace:
 [1] LLVMInitializeAllTargets
   @ ~/.julia/packages/LLVM/srSVa/lib/libLLVM_extra.jl:10 [inlined]
 [2] InitializeAllTargets
   @ ~/.julia/packages/LLVM/srSVa/src/init.jl:58 [inlined]
 [3] __init__()
   @ GPUCompiler ~/.julia/packages/GPUCompiler/XwWPj/src/GPUCompiler.jl:50
 [4] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base ./loading.jl:768
 [5] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base ./loading.jl:854
 [6] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1097
 [7] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997
during initialization of module GPUCompiler

julia> versioninfo()
Julia Version 1.8.0-DEV.151
Commit 6240d352f1 (2021-07-07 19:24 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin20.3.0)
  CPU: Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.0 (ORCJIT, cyclone)

This means that also all packages depending on GPUCompiler.jl can't be loaded, including CUDA.jl.

The package loads when running Julia with Rosetta, but nm finds the same symbols in the x86_64 library.

[vchuravy]

Which versions? We haven't released a version of LLVM.jl that depennds on LLVMExtra_jll

[giordano]

(jl_ZRC3JZ) pkg> st
      Status `/private/var/folders/v2/hmy3kzgj4tb3xsy8qkltxd0r0000gn/T/jl_ZRC3JZ/Project.toml`
  [61eb1bfa] GPUCompiler v0.10.0
  [929cbde3] LLVM v3.9.0
  [dad2f222] LLVMExtra_jll v0.0.5+1

(jl_ZRC3JZ) pkg> st -m
      Status `/private/var/folders/v2/hmy3kzgj4tb3xsy8qkltxd0r0000gn/T/jl_ZRC3JZ/Manifest.toml`
  [fa961155] CEnum v0.4.1
  [34da2185] Compat v3.31.0
  [864edb3b] DataStructures v0.18.9
  [e2ba6199] ExprTools v0.1.6
  [61eb1bfa] GPUCompiler v0.10.0
  [692b3bcd] JLLWrappers v1.3.0
  [929cbde3] LLVM v3.9.0
  [bac558e1] OrderedCollections v1.4.1
  [21216c6a] Preferences v1.2.2
  [6c6a2e73] Scratch v1.1.0
  [a759f4b9] TimerOutputs v0.5.12
  [dad2f222] LLVMExtra_jll v0.0.5+1
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.4.0
  [b77e0a4c] InteractiveUtils
  [b27032c2] LibCURL v0.6.2
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.6.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [fa267f1f] TOML v1.0.0
  [a4e569a6] Tar v1.9.1
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [deac9b47] LibCURL_jll v7.73.0+4
  [29816b5a] LibSSH2_jll v1.9.1+2
  [c8ffd9c3] MbedTLS_jll v2.24.0+2
  [14a3606d] MozillaCACerts_jll v2020.7.22
  [83775a58] Zlib_jll v1.2.12+1
  [8e850b90] libblastrampoline_jll v3.0.4+0
  [8e850ede] nghttp2_jll v1.41.0+1
  [3f19e933] p7zip_jll v16.2.1+1

[vchuravy]

But... https://github.com/maleadt/LLVM.jl/blob/v3.9.0/Project.toml doesn't depend on LLVMExtra?

[vchuravy]

Oooh I see what is going on. JuliaLang/julia#41431 which of course breaks LLVM 3.9

[giordano]

Ok, I can confirm that on JuliaLang/julia@b617e8d, the parent of JuliaLang/julia#41431, I can load GPUCompiler without issues

[vchuravy]

Yeah it will probably take us a few days to get LLVM 4.0 out

[maleadt]

Yup, this is just LLVM.jl being broken on 1.7 and 1.8 due to the LLVMExtra API removal, and not aarch64-specific. We need a LLVM.jl release -- but I'm waiting for the ORC changes to land first. Should be OK in time for 1.7 :-)

[giordano]

and not aarch64-specific

I had this impression because it worked with the x86_64 build under Rosetta, but that was Julia v1.6

[maleadt]

GPUCompiler should work again. CUDA.jl will do too, soon: JuliaGPU/CUDA.jl#1037