Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

illegal hardware instruction via precompiling CPU and CUDA KA kernels #580

Open
lcw opened this issue May 9, 2024 · 3 comments
Open

illegal hardware instruction via precompiling CPU and CUDA KA kernels #580

lcw opened this issue May 9, 2024 · 3 comments
Assignees
@vchuravy

Comments

@lcw
Copy link
Contributor

lcw commented May 9, 2024

I tried to add precompilation to my package (via PrecompileTools in HorribleSanity/Raven.jl@52e0cc1) and am now generating illegal hardware instruction errors. I have not been able to minimize it but @vchuravy asked me to submit an issue so here it is. I have been able to reproduce the issue on two different machines via

rm -rf ~/.julia
git clone -b lcw/precompile_tools git@github.com:HorribleSanity/Raven.jl
cd Raven.jl
julia -e "using Pkg; Pkg.add(\"CUDA\")"
julia --project -e "using Pkg; Pkg.instantiate()"
julia --project -e "using CUDA, Raven"
julia --project -e "using CUDA, Raven; generate(GridManager(LobattoCell{Float32,Array}(3,4,5),brick(3,3,2)))"

The stack trace that is produced is

Internal error: encountered unexpected error in runtime:
AssertionError(msg="malformed :new expression")
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2405
reprocess_instruction! at ./compiler/ssair/irinterp.jl:126
#_ir_abstract_constant_propagation#511 at ./compiler/ssair/irinterp.jl:251
_ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:206
ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:384
semi_concrete_eval_call at ./compiler/abstractinterpretation.jl:1168
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:812
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:788
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:103
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_apply at ./compiler/abstractinterpretation.jl:1612
abstract_call_known at ./compiler/abstractinterpretation.jl:2004
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_45276.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_type_infer at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2480 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2368
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2886 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
#_#16 at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:44
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
Kernel at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:37
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
materializemetrics at /tmp/lucas/Raven.jl/src/lobattocells.jl:2849
unknown function (ip: 0x7f114e845a22)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:422
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:127
unknown function (ip: 0x7f114e840015)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
top-level scope at none:1
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:552
jfptr__start_82738.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Internal error: encountered unexpected error in runtime:
AssertionError(msg="malformed :new expression")
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2405
reprocess_instruction! at ./compiler/ssair/irinterp.jl:126
#_ir_abstract_constant_propagation#511 at ./compiler/ssair/irinterp.jl:251
_ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:206
ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:384
semi_concrete_eval_call at ./compiler/abstractinterpretation.jl:1168
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:812
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:788
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:103
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_apply at ./compiler/abstractinterpretation.jl:1612
abstract_call_known at ./compiler/abstractinterpretation.jl:2004
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_45276.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_type_infer at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2480 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2368
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2886 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
__run at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:82
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
#_#16 at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:44
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
Kernel at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:37
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
materializemetrics at /tmp/lucas/Raven.jl/src/lobattocells.jl:2849
unknown function (ip: 0x7f114e845a22)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:422
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:127
unknown function (ip: 0x7f114e840015)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
top-level scope at none:1
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:552
jfptr__start_82738.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
warning: /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/extras/loopinfo.jl:26:0: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering
Unreachable reached at 0x7f114e849c5f

[666880] signal (4.2): Illegal instruction
in expression starting at none:1
NamedTuple at ./boot.jl:622
unknown function (ip: 0x7f114e849c99)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
macro expansion at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/extras/loopinfo.jl:26 [inlined]
macro expansion at /tmp/lucas/Raven.jl/src/lobattocells.jl:2729 [inlined]
cpu_hexvolumebrickmetrics! at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/macros.jl:287 [inlined]
cpu_hexvolumebrickmetrics! at ./none:0
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
__thread_run at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:115
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
__run at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:82
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
#_#16 at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:44
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
Kernel at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:37
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
materializemetrics at /tmp/lucas/Raven.jl/src/lobattocells.jl:2849
unknown function (ip: 0x7f114e845a22)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:422
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:127
unknown function (ip: 0x7f114e840015)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:552
jfptr__start_82738.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 6252595 (Pool: 6247471; Big: 5124); GC: 8

The julia version is

Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  uname: Linux 4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023 x86_64 x86_64
  CPU: AMD EPYC 7443P 24-Core Processor: 
                 speed         user         nice          sys         idle          irq
       #1-48  3478 MHz     182186 s      12777 s     469537 s  2330423177 s      41199 s
  Memory: 251.2397003173828 GB (252315.3515625 MB free)
  Uptime: 4.8571565e6 sec
  Load Avg:  0.44  2.3  2.71
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 48 virtual cores)
Environment:
  JULIA_CPU_TARGET = haswell,clone_all;broadwell,base(0);skylake-avx512,base(0);znver2,base(0);znver3,base(0)
  LD_LIBRARY_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib
  LD_LIBRARY_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib:1
  ACLOCAL_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/share/aclocal
  ACLOCAL_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/share/aclocal:1
  CMAKE_PREFIX_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/
  CMAKE_PREFIX_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/:1
  HOME = /home/lwilcox
  LIBRARY_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib
  LIBRARY_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib:1
  MANPATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/share/man:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/man:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/share/man::::/home/lwilcox/.fzf/man
  MANPATH_modshare = :1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/man:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/share/man:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/share/man:1
  MODULEPATH = /etc/scl/modulefiles:/usr/share/Modules/modulefiles:/share/modules/base
  MODULEPATH_modshare = /home/lwilcox/spack/share/spack/modules/linux-centos7-x86_64:1:/etc/scl/modulefiles:1:/share/modules/base:2:/usr/share/Modules/modulefiles:1
  MODULESHOME = /usr/share/Modules
  PATH = /home/lwilcox/.local/bin:/home/lwilcox/local/bin:/usr/share/Modules/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/environment-modules-4.6.1-zdvbiw6n4iwdghhmstuqqewwl4hjdfsg/bin:/home/lwilcox/.local/bin:/home/lwilcox/local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/lwilcox/.fzf/bin:/home/lwilcox/bin:/home/lwilcox/.config/zsh/plugins/fzf-zsh-plugin/bin
  PATH_modshare = /home/lwilcox/.local/bin:1:/usr/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/bin:1:/home/lwilcox/local/bin:1:/usr/local/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/environment-modules-4.6.1-zdvbiw6n4iwdghhmstuqqewwl4hjdfsg/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/bin:1:/usr/sbin:1:/home/lwilcox/.fzf/bin:1:/usr/local/sbin:1
  TERM = tmux-256color
  JAVA_HOME = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.402.b06-2.el8.x86_64/jre
  FPATH = /home/lwilcox/.config/zsh/plugins/fzf-zsh-plugin/completions:/usr/share/Modules/init/zsh-functions:/usr/local/share/zsh/site-functions:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/zsh/site-functions:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/zsh/5.8/functions:/home/lwilcox/.config/zsh/plugins/zsh-defer:/home/lwilcox/.config/zsh/plugins/zsh-completions:/home/lwilcox/.config/zsh/plugins/zsh-history-substring-search:/home/lwilcox/.config/zsh/plugins/zsh-vi-mode:/home/lwilcox/.config/zsh/plugins/fzf-zsh-plugin:/home/lwilcox/.config/zsh/plugins/fzf-tab:/home/lwilcox/.config/zsh/plugins/zsh-autosuggestions:/home/lwilcox/.config/zsh/plugins/fast-syntax-highlighting:/home/lwilcox/.config/zsh/plugins/zsh-completions/src:/home/lwilcox/.config/zsh/plugins/fzf-tab/lib

The global environment

Status `~/.julia/environments/v1.10/Manifest.toml`                                                                                                                                                                  
  [621f4979] AbstractFFTs v1.5.0                                                                                                                                                                                    
  [79e6a3ab] Adapt v4.0.4                                                                                                                                                                                           
  [a9b6321e] Atomix v0.1.0                                                                                                                                                                                          
  [ab4f0b2a] BFloat16s v0.5.0                                                                                                                                                                                       
  [fa961155] CEnum v0.5.0                                                                                                                                                                                           
  [052768ef] CUDA v5.3.3                                                                                                                                                                                            
⌅ [1af6417a] CUDA_Runtime_Discovery v0.2.4                                                                                                                                                                          
  [3da002f7] ColorTypes v0.11.5                                                                                                                                                                                     
  [5ae59095] Colors v0.12.10                                                                                                                                                                                        
  [34da2185] Compat v4.15.0                                                                                                                                                                                         
  [a8cc5b0e] Crayons v4.1.1                                                                                                                                                                                         
  [9a962f9c] DataAPI v1.16.0                                                                                                                                                                                        
  [a93c6f00] DataFrames v1.6.1                                                                                                                                                                                      
  [864edb3b] DataStructures v0.18.20                                                                                                                                                                                
  [e2d170a0] DataValueInterfaces v1.0.0                                                                                                                                                                             
  [e2ba6199] ExprTools v0.1.10                                                                                                                                                                                      
  [53c48c17] FixedPointNumbers v0.8.4                                                                                                                                                                               
  [0c68f7d7] GPUArrays v10.1.0                                                                                                                                                                                      
  [46192b85] GPUArraysCore v0.1.6                                                                                                                                                                                   
  [61eb1bfa] GPUCompiler v0.26.4                                                                                                                                                                                    
  [842dd82b] InlineStrings v1.4.0                                                                                                                                                                                   
  [41ab1584] InvertedIndices v1.3.0                                                                                                                                                                                 
  [82899510] IteratorInterfaceExtensions v1.0.0                                                                                                                                                                     
  [692b3bcd] JLLWrappers v1.5.0                                                                                                                                                                                     
  [63c18a36] KernelAbstractions v0.9.18                                                                                                                                                                             
⌅ [929cbde3] LLVM v6.6.3                                                                                                                                                                                            
  [8b046642] LLVMLoopInfo v1.0.0                                                                                                                                                                                    
  [b964fa9f] LaTeXStrings v1.3.1                                                                                                                                                                                    
  [1914dd2f] MacroTools v0.5.13                                                                                                                                                                                     
  [e1d29d7a] Missings v1.2.0                                                                                                                                                                                        
  [5da4648a] NVTX v0.3.4                                                                                                                                                                                            
  [bac558e1] OrderedCollections v1.6.3                                                                                                                                                                              
  [69de0a69] Parsers v2.8.1                                                                                                                                                                                         
  [2dfb63ee] PooledArrays v1.4.3                                                                                                                                                                                    
  [aea7be01] PrecompileTools v1.2.1                                                                                                                                                                                 
  [21216c6a] Preferences v1.4.3                                                                                                                                                                                     
  [08abe8d2] PrettyTables v2.3.1                                                                                                                                                                                    
  [74087812] Random123 v1.7.0                                                                                                                                                                                       
  [e6cf234a] RandomNumbers v1.5.3                                                                                                                                                                                   
  [189a3867] Reexport v1.2.2                                                                                                                                                                                        
  [ae029012] Requires v1.3.0                                                                                                                                                                                        
  [6c6a2e73] Scratch v1.2.1                                                                                                                                                                                         
  [91c51154] SentinelArrays v1.4.1                                                                                                                                                                                  
  [a2af1166] SortingAlgorithms v1.2.1                                                                                                                                                                               
  [90137ffa] StaticArrays v1.9.3                                                                                                                                                                                    
  [1e83bf80] StaticArraysCore v1.4.2
  [892a3eda] StringManipulation v0.3.4
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.11.1
  [a759f4b9] TimerOutputs v0.5.23
  [013be700] UnsafeAtomics v0.2.1
  [d80eeb9a] UnsafeAtomicsLLVM v0.1.3
  [4ee394cb] CUDA_Driver_jll v0.8.1+0
⌅ [76a88914] CUDA_Runtime_jll v0.12.1+0
  [9c1d0b0a] JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] LLVMExtra_jll v0.0.29+0
  [e98f9f5b] NVTX_jll v3.1.0+2
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [10745b16] Statistics v1.10.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2

And the project environment is

Project Raven v0.1.0-pre                                                                                                                                                                                            
Status `/tmp/lucas/Raven.jl/Manifest.toml`                                                                                                                                                                          
  [1520ce14] AbstractTrees v0.4.5                                                                                                                                                                                   
  [79e6a3ab] Adapt v4.0.4                                                                                                                                                                                           
  [a9b6321e] Atomix v0.1.0                                                                                                                                                                                          
  [fa961155] CEnum v0.5.0                                                                                                                                                                                           
  [34da2185] Compat v4.15.0                                                                                                                                                                                         
  [ffbed154] DocStringExtensions v0.9.3                                                                                                                                                                             
  [46192b85] GPUArraysCore v0.1.6                                                                                                                                                                                   
  [692b3bcd] JLLWrappers v1.5.0                                                                                                                                                                                     
  [63c18a36] KernelAbstractions v0.9.18                                                                                                                                                                             
⌅ [929cbde3] LLVM v6.6.3                                                                                                                                                                                            
  [da04e1cc] MPI v0.20.19                                                                                                                                                                                           
  [3da0fdf6] MPIPreferences v0.1.11                                                                                                                                                                                 
  [1914dd2f] MacroTools v0.5.13                                                                                                                                                                                     
  [c5182250] OneDimensionalNodes v1.1.0                                                                                                                                                                             
  [7d669430] P4est v0.4.13                                                                                                                                                                                          
  [f636fe8e] P4estTypes v0.1.3                                                                                                                                                                                      
  [eebad327] PkgVersion v0.3.3                                                                                                                                                                                      
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [3cdcf5f2] RecipesBase v1.3.4
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [90137ffa] StaticArrays v1.9.3
  [1e83bf80] StaticArraysCore v1.4.2
  [013be700] UnsafeAtomics v0.2.1
  [d80eeb9a] UnsafeAtomicsLLVM v0.1.3
  [e33a78d0] Hwloc_jll v2.10.0+0
  [dad2f222] LLVMExtra_jll v0.0.29+0
  [7cb0a576] MPICH_jll v4.2.1+1
  [f1f71cc9] MPItrampoline_jll v5.3.3+1
  [9237b28f] MicrosoftMPI_jll v10.1.4+2
⌅ [fe0851c0] OpenMPI_jll v4.1.6+0
  [6b5a15aa] P4est_jll v2.8.1+2
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2
@lcw lcw mentioned this issue May 9, 2024
Open
@vchuravy
Copy link
Member

Ah shoot. I had expected it to be related to JuliaGPU/CUDA.jl#2336

But this looks much more like a core Julia bug.

@vchuravy
Copy link
Member

I am going to try and creduce this.

@vchuravy vchuravy self-assigned this May 10, 2024
@vchuravy
Copy link
Member

So it does belong here since on a machine without CUDA it is not reproducible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vchuravy vchuravy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lcw @vchuravy