PackageCompiler: LLVM ERROR

[rssdev10]

Hi, I’m getting the following error while compiling my package:

ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
LLVM ERROR: Broken function found, compilation aborted!
ERROR: LoadError: failed process: Process(`/Applications/Julia-1.3.app/Contents/Resources/julia/bin/julia --output-o=run.a --track-allocation=none --code-coverage=none --history-file=yes --inline=yes --math-mode=ieee --project=@. --compile=yes --track-allocation=none --sysimage-native-code=yes --sysimage=/Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib --compiled-modules=yes --optimize=2 /Users/rss/.julia/packages/PackageCompiler/CJQcs/sysimg/run_julia_code.jl`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] #run#565(::Bool, ::typeof(run), ::Cmd) at ./process.jl:440
 [3] run at ./process.jl:438 [inlined]
 [4] #run_julia#1 at /Users/rss/.julia/packages/PackageCompiler/CJQcs/src/compiler_flags.jl:225 [inlined]
 [5] #run_julia at ./none:0 [inlined]
 [6] (::PackageCompiler.var"#13#14"{Base.Iterators.Pairs{Symbol,Nothing,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),NTuple{14,Nothing}}},String})() at /Users/rss/.julia/packages/PackageCompiler/CJQcs/src/static_julia.jl:263
 [7] cd(::PackageCompiler.var"#13#14"{Base.Iterators.Pairs{Symbol,Nothing,NTuple{14,Symbol},NamedTuple{(:sysimage, :startup_file, :handle_signals, :sysimage_native_code, :compiled_modules, :depwarn, :warn_overwrite, :compile, :cpu_target, :optimize, :debug_level, :inline, :check_bounds, :math_mode),NTuple{14,Nothing}}},String}, ::String) at ./file.jl:104
...

Is any way to understand what exactly produces that error?

Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.0.0)
  CPU: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)

[terasakisatoshi]

It might as well submit issue to PackageCompiler.jl not here.

[KristofferC]

I think this is fixed in 1.3.1. Please comment if not and give instructions how to reproduce.

[IanButterworth]

I just got this issue on an aarch64 system on PackageCompiler master with julia v1.3.1

[ Info: PackageCompiler: creating system image object file, this might take a while...
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
LLVM ERROR: Broken function found, compilation aborted!

[IanButterworth]

It's on a private package, and PackageCompiler works with another private package on the same setup. Given it's on a private package with many deps, I can't easily post instructions to reproduce. Any ideas on what to test or look for?

[KristofferC]

That just seems like a julia bug... Getting a repro would be pretty important for it to be fixable I feel.

[DilumAluthge]

I would start by making a list of all of the dependencies of your private package. And try to compile each of those, to see if a dependency is the problem.

If all of the dependencies are able to compile, then you should remove source code files from your package one file at a time until it compiles, so we can identify what the problem is.

[IanButterworth]

I'm happy to try that out. Although, there is a reproducible example in JuliaLang/julia#31156 though, and @tkf 's been bumping that.

(I haven't tried that yet but will)

[IanButterworth]

Yeah, it's reproducible for me JuliaLang/julia#31156 (comment)

Is the example given still relevant for PackageCompiler 1.0.0?

[rssdev10]

@KristofferC , sorry, I cannot provide the code. Primary, I asked how to see what is happening. I can provide debug info or rebuild Julia in some specific mode to do it if it is implemented.

[KristofferC]

If the problem is with the permutedims specifically in JuliaLang/julia#31156 (comment) we could black list those.

[KristofferC]

Can you guys try with #329?

[IanButterworth]

It still errors for me, but it might just be that my package is trying to precompile permutedims for an array of higher dimensionality.. perhaps the blacklist should search for:

"precompile(Tuple{typeof(Base.permutedims), Array{Bool, "
"precompile(Tuple{typeof(Base.permutedims), Array{UInt8, "

or a regex of the full statement that allows for a variable array dimension?

[IanButterworth]

I'm trying hardcoding more dimensions into the blacklist for now

[KristofferC]

Just blacklist the whole permutedims then and try? Having it more narrow would of course be better though.

[IanButterworth]

I’ve blacklisted bool and uint8 arrays up to 10 dimensions. I’ll see how this goes. (Each attempt takes ~30 mins on this system)

[KristofferC]

Yep, I know it is a slow process :D

[IanButterworth]

Just hit the same error.. Time to try blacklisting permutedims entirely

[KristofferC]

If that doesn't work I guess you will have to print out all the precompile statements and start bisecting heh.

[rssdev10]

regarding the mentioned Base.precompile(Tuple{typeof(Base.permutedims), Array{Bool, 3}, Array{Int64, 1}}) with compile.bash/precompile_wrapper.jl I'm getting the same errors:

ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
LLVM ERROR: Broken function found, compilation aborted!

Regarding the own project, as the PackageCompiler was completely changed, I have some issues with compiling.... Trying to call PackageCompiler.create_sysimage with prepared by unit tests a precomplied file. But getting:

ERROR: LoadError: UndefVarError: Logging not defined
Stacktrace:
 [1] top-level scope at /Users/rss/projects/jdtp/job_precompile.jl:6
 [2] include at ./boot.jl:328 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1105
 [4] include(::Module, ::String) at ./Base.jl:31
 [5] exec_options(::Base.JLOptions) at ./client.jl:287
 [6] _start() at ./client.jl:460
in expression starting at /Users/rss/projects/jdtp/job_precompile.jl:6
ERROR: failed process: Process(`/Applications/Julia-1.3.app/Contents/Resources/julia/bin/julia --color=yes --startup-file=no --sysimage=/Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib --project=/Users/rss/projects/jdtp --compile=all --trace-compile=/var/folders/n_/h8tm56xd4q9_41rd7knts2r80000gn/T/jl_NJ6p9u job_precompile.jl`, ProcessExited(1)) [1]

And first 6 lines of that file are looking correct.

precompile(Tuple{typeof(Base.atreplinit), Function})
precompile(Tuple{typeof(Base.vcat), String, String, String})
precompile(Tuple{typeof(Base.convert), Type{Base.CoreLogging.LogLevel}, Base.CoreLogging.LogLevel})
precompile(Tuple{typeof(Base.getindex), Base.RefValue{Base.CoreLogging.LogLevel}})
precompile(Tuple{typeof(Base.:(>=)), Base.CoreLogging.LogLevel, Base.CoreLogging.LogLevel})
precompile(Tuple{typeof(Base.CoreLogging.shouldlog), Logging.ConsoleLogger, Base.CoreLogging.LogLevel, Module, String, Symbol})

[KristofferC]

What is the command you run? PackageCompiler should make Logging visible.

[rssdev10]

julia --project=@. --trace-compile=job_precompile.jl test/runtests.jl
julia --project=@. -e 'using PackageCompiler; PackageCompiler.create_sysimage(:Job; sysimage_path="JobSysimagePrecompile.so", precompile_execution_file="job_precompile.jl")'

[KristofferC]

Should be precompile_statements_file. The argument precompile_execution_file is if you want to have julia code to execute like @info "foo".

[rssdev10]

ok. Next error:

[ Info: PackageCompiler: creating system image object file, this might take a while...
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
LLVM ERROR: Broken function found, compilation aborted!
ERROR: failed process: Process(`/Applications/Julia-1.3.app/Contents/Resources/julia/bin/julia --color=yes --startup-file=no --cpu-target=native --sysimage=/Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib --project=/Users/rss/projects/jdtp --output-o=/var/folders/n_/h8tm56xd4q9_41rd7knts2r80000gn/T/jl_H3GuZ0.o -e 'Base.reinit_stdio()
@eval Sys BINDIR = ccall(:jl_get_julia_bindir, Any, ())::String
Base.init_load_path()
Base.init_depot_path()
using Job
# This @eval prevents symbols from being put into Main
@eval Module() begin
    PrecompileStagingArea = Module()
    for (_pkgid, _mod) in Base.loaded_modules
        if !(_pkgid.name in ("Main", "Core", "Base"))
            eval(PrecompileStagingArea, :(const $(Symbol(_mod)) = $_mod))
        end
    end
    precompile_statements = String[]
        append!(precompile_statements, readlines("/var/folders/n_/h8tm56xd4q9_41rd7knts2r80000gn/T/jl_g4trKH"))
    append!(precompile_statements, readlines("/Users/rss/projects/jdtp/job_precompile.jl"))

    precompile_blacklist = ["precompile(Tuple{typeof(Base.permutedims), Array{Bool, 2}, Array{Int64, 1}})", "precompile(Tuple{typeof(Base.permutedims), Array{Bool, 3}, Array{Int64, 1}})", "precompile(Tuple{typeof(Base.permutedims), Array{UInt8, 3}, Array{Int64, 1}})"]
    for statement in sort(precompile_statements)
        if strip(statement) in precompile_blacklist
            continue
        end
        # println(statement)
        try
            Base.include_string(PrecompileStagingArea, statement)
        catch
            # See julia issue #28808
            @debug "failed to execute $statement"
        end
    end
end # module
empty!(LOAD_PATH)
empty!(DEPOT_PATH)
'`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] #run#565(::Bool, ::typeof(run), ::Cmd) at ./process.jl:440
 [3] run at ./process.jl:438 [inlined]
 [4] #create_sysimg_object_file#6(::String, ::String, ::Array{String,1}, ::Array{String,1}, ::String, ::Nothing, ::Bool, ::typeof(PackageCompiler.create_sysimg_object_file), ::String, ::Array{String,1}) at /Users/rss/.julia/packages/PackageCompiler/K3UUo/src/PackageCompiler.jl:285
 [5] #create_sysimg_object_file at ./none:0 [inlined]
 [6] #create_sysimage#7(::String, ::String, ::Array{String,1}, ::String, ::Bool, ::Bool, ::Bool, ::String, ::Nothing, ::Nothing, ::Bool, ::typeof(create_sysimage), ::Symbol) at /Users/rss/.julia/packages/PackageCompiler/K3UUo/src/PackageCompiler.jl:415
 [7] (::PackageCompiler.var"#kw##create_sysimage")(::NamedTuple{(:sysimage_path, :precompile_statements_file),Tuple{String,String}}, ::typeof(create_sysimage), ::Symbol) at ./none:0
 [8] top-level scope at none:1

[IanButterworth]

That's the main issue in this thread

[rssdev10]

Am I understand right that the purpose is to localize precompile() statement which is producing that error? The file contains ~2200 statements. In theory, I can try binary split it and find the issue. But it will take too much time....

[KristofferC]

Yeah, we need to figure out a better way than judt bisecting (which is probably to make LLVM print out what julia function it has troubles with)

[rssdev10]

Is any way to run the compiler in a verbose mode line by line with that file and find all failing statements? In worst case, bisecting of the file by a script and keeping it working for a night, looks not so bad idea.....

[KristofferC]

Maybe if we run with an assertion build of llvm?

[rssdev10]

how to do it?

[KristofferC]

There is a setting that can be set in Make.user to enable assertions,I don't remember exactly the name right now.

[rssdev10]

https://github.com/JuliaLang/julia/blob/master/Make.inc#L296

Do you mean this group?

# LLVM Options
LLVMROOT := $(build_prefix)
LLVM_ASSERTIONS := 0
LLVM_DEBUG := 0
# set to 1 to get clang and compiler-rt
BUILD_LLVM_CLANG := 0
# set to 1 to get lldb (often does not work, no chance with llvm3.2 and earlier)
# see http://lldb.llvm.org/build.html for dependencies
BUILD_LLDB := 0

[IanButterworth]

I'm just giving this a go with Julia master and LLVM_ASSERTIONS := 1

[rssdev10]

These ones doesn't help:

LLVM_ASSERTIONS := 1
LLVM_DEBUG := 1

Completely nothing new comparing with the default build.

Julia v 1.3.1 (#34151) / 2d5741174ce3e6a394010d2e470e4269ca54607f.

[IanButterworth]

If you check the main julia issue, I found that the nightly reports the function when the LLVM fails. It doesn’t tell us much more than that it’s from permutedims though

[rssdev10]

Am I understand right that if I'm taking master branch, then all previously compiled packages from ~/.julia must be recompiled?

[IanButterworth]

I wrote a Multithreaded Randomized Precompile Statement Bisector™ to hunt out the precompile statements that are failing, and whittled down the 6406 precompile statements to these 2 which all individually fail (in 3 hrs across 6 parallel threads on an aarch64 machine)

precompile(Tuple{typeof(Base.copyto!), Array{UInt8, 2}, Base.IteratorsMD.CartesianIndices{2, Tuple{Base.UnitRange{Int64}, Base.UnitRange{Int64}}}, Array{UInt8, 2}, Base.IteratorsMD.CartesianIndices{2, Tuple{Base.UnitRange{Int64}, Base.UnitRange{Int64}}}}) 

precompile(Tuple{typeof(Base.circshift!), Array{UInt8, 2}, Array{UInt8, 2}, Tuple{Int64, Int64}})

For record, the number of precompile statements after each step of the bisector: 6406, 1067, 355, 118, 19, 6, 2

[DilumAluthge]

If you remove those 2 statements from the list of 6406, do all of the remaining 6404 statements run without error?

[IanButterworth]

Yeah! Just produced a sysimage on aarch64 using precompile_statements_file with those two lines removed, and it completed and I can use the sysimage 🎉🎉🎉🎉🎉

So, there's a viable work-around for me!

However, I also tried using precompile_execution_file and blacklisting

[r"typeof(Base.copyto!), Array{UInt8"i, 
r"typeof(Base.circshift!), Array{UInt8"i, 
r"typeof(Base.permutedims), Array{Bool"i]

but that didn't work. I still hit the error. Although maybe I got the regex wrong? or the precompile_execution_file somehow created another precomp that failed..
Perhaps if the blacklist finds a precomp statement it should report that it's excluding it. Otherwise I'm not certain it actually worked.

[DilumAluthge]

You need to escape special characters such as parentheses and periods when you make a regex.

You may find https://regex101.com/ helpful for testing out regexes.

And in the Julia REPL, you should try running occursin(r"my regex", "my statement"). That will return true if the given regex correctly matches the given statement.

For example, in your case, I would recommend trying this instead for the blacklist:

[r"typeof\(Base\.copyto!\), Array{UInt8"i, 
r"typeof\(Base\.circshift!\), Array{UInt8"i, 
r"typeof\(Base\.permutedims\), Array{Bool"i]

You see that I have escaped all periods, left parentheses, and right parentheses.

@ianshmean Can you try that blacklist and let me know if it works?

This is very exciting. I feel like we have almost solved this.

[IanButterworth]

I've always felt my regex skills were lacking.. Just running

[IanButterworth]

Argh... It still errored, and that was after I checked each of your regex's worked against the target statements, and even broadened the regex's to just the first half.

The complexity of the package I'm trying to compile, which is very multithreaded and asynchronous seems to be creating a rather chaotic precompile statements list.

My strategy for now is to stick with the list I bisected, and quash those listed above from it.

As for a proper fix, I'm shifting my hope to this being fixed upstream tbh, and backported. It just seems too hard to trace back to specific precompile statements

[rssdev10]

For me the issue was fixed with Julia 1.4.0 update

Julia Version 1.4.0
Commit b8e9a9ecc6 (2020-03-21 16:36 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, broadwell)

[IanButterworth]

That's surprising.. the LLVM patch isn't included in the LLVM build that ships with 1.4.0. You/we may just be lucky with some subtle change in 1.4.0 avoiding the issue.

But for a proper fix, there is a patch on the LLVM version used by julia master (9.0.1) and an upcoming backport of the patch for 8.0.1 that I hope will make it into 1.4.1 JuliaPackaging/Yggdrasil#619

[rssdev10]

Ok. I'm using your script on Julia 1.3.1 with few modifications - replaced threads number by fixed number of slices. Now it is 20. It should be really much more than number of functions which are failing the compilation.

And I see that number of messages:

ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers
ptrtoint not supported for non-integral pointers
inttoptr not supported for non-integral pointers

doesn't correspond to number of error functions.

And second, in most cases, the part with failing function contains one of our functions. It is still working. But one of samples - precompile(Tuple{typeof(Job.process), String, String, Job.Context}) where Job.Context is a struct with partially declared types inside.

[IanButterworth]

I'm not sure why you're still trying to bisect the issue.. It's been identified and patched in LLVM in https://github.com/JuliaLang/julia/pull/34860/files and worked through into master in the link above (and awaiting release in 1.4.1).

Sorry if you've been out of the loop! The issue has been discussed in a variety of places

[rssdev10]

I tried to clarify the reason, why it is working in Julia 1.4.0 without that patch and is not working up to 1.3.1. Sorry, the issue was really low priority for us. It is just an inconvenience for the service activation in k8s but is not a blocking issue. And time, when you shared your script, was overlapped for me by a business trip.

The final list of failing functions:

precompile(Tuple{typeof(Job.process_data), String, String, Job.Context})
precompile(Tuple{typeof(Job.process), String, String, Job.Context})
precompile(Tuple{typeof(Job.check_calc_cache_data), getfield(Job, Symbol("#40#50")){Base.Set{String}}, String, String})
precompile(Tuple{Type{Job.OnetCorpus}, DataFrames.DataFrame})
precompile(Tuple{Type{Job.OnetCorpus}, DataFrames.DataFrame, Base.Set{String}})
precompile(Tuple{typeof(Job.init)})
precompile(Tuple{Type{Job.OnetCorpus}, DataFrames.DataFrame})
precompile(Tuple{typeof(Job.count_tfidf), TextAnalysis.Corpus{TextAnalysis.TokenDocument{String}}, TextAnalysis.TokenDocument{String}})
precompile(Tuple{typeof(Job.create_context)})
precompile(Tuple{Type{Job.OnetCorpus}, DataFrames.DataFrame, Base.Set{String}})
precompile(Tuple{getfield(Job, Symbol("#40#50")){Base.Set{String}}, String})
precompile(Tuple{typeof(Job.check_calc_cache_data), getfield(Job, Symbol("#40#50")){Base.Set{String}}, String, String})
precompile(Tuple{typeof(Job.process), String, String, Job.Context})

Let me know please if that case is interesting for somebody for more investigation. Or, will think that it is finally fixed in Julia 1.4.0. Thank you for the help.

[KristofferC]

This should be fixed in 1.4.1