Precompilation code causes deadlock

[andreasnoack]

I've had issues with this package hanging during precompilation. My colleague @vtjnash said about

Term.jl/src/Term.jl

Lines 175 to 192 in e58f28b

	@precompile_setup begin
	originalSTDOUT = stdout
	(outRead, outWrite) = redirect_stdout()
	@precompile_all_calls begin
	Panel()
	Panel("test") * Panel(Panel()) / hLine(20) |> tprint
	termshow(Panel)
	termshow(Dict(:x => 1))
	termshow(zeros(4))
	termshow(zeros(4, 4))
	end
	close(outRead)
	redirect_stdout(originalSTDOUT)
	end

that

I am not sure you are allowed to alter stdout during precompile. But regardless, making it unread creates a deadlock

and that similar issues apply to the version on master.

[FedeClaudi]

Thanks for getting in touch. This hasn't been a problem in my experience, including in CI on various OS. Could you give me more details about your setup and what's happening please?

[ronisbr]

Hi! I saw this problem before when something caused an unexpected behavior inside the precompilation calls, which is not seen due to the stdout redirection. It can be some error that is not printed to stderr or some request for user input. This assumption can be verified by removing all lines related to redirection and trying to recompile the package.

[FedeClaudi]

thanks!

[andreasnoack]

Precompilation simply hangs at

julia> using Term
[ Info: Precompiling Term [22787eb5-b846-44ae-b979-8e399b8463ab]

and when interrupted, it gives

julia> using Term
[ Info: Precompiling Term [22787eb5-b846-44ae-b979-8e399b8463ab]
^C
[3960231] signal (2): Interrupt
in expression starting at /home/andreasnoack/.julia/dev/Term/src/Term.jl:175
epoll_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv__io_poll at /workspace/srcdir/libuv/src/unix/epoll.c:236
uv_run at /workspace/srcdir/libuv/src/unix/core.c:400
ijl_task_get_next at /home/andreasnoack/julia/src/partr.c:381
poptask at ./task.jl:963
wait at ./task.jl:972
uv_write at ./stream.jl:1048
unsafe_write at ./stream.jl:1120
write at ./strings/io.jl:244 [inlined]
print at ./strings/io.jl:246 [inlined]
#tprint#4 at /home/andreasnoack/.julia/dev/Term/src/tprint.jl:43
tprint at /home/andreasnoack/.julia/dev/Term/src/tprint.jl:37 [inlined]
#termshow#36 at /home/andreasnoack/.julia/dev/Term/src/repr.jl:308
termshow at /home/andreasnoack/.julia/dev/Term/src/repr.jl:261
unknown function (ip: 0x7fcc5284668e)
_jl_invoke at /home/andreasnoack/julia/src/gf.c:2647
ijl_apply_generic at /home/andreasnoack/julia/src/gf.c:2829
#termshow#25 at /home/andreasnoack/.julia/dev/Term/src/repr.jl:64
termshow at /home/andreasnoack/.julia/dev/Term/src/repr.jl:64 [inlined]
macro expansion at /home/andreasnoack/.julia/dev/Term/src/Term.jl:183 [inlined]
macro expansion at /home/andreasnoack/.julia/packages/SnoopPrecompile/UWvXF/src/SnoopPrecompile.jl:51 [inlined]
macro expansion at /home/andreasnoack/.julia/dev/Term/src/Term.jl:180 [inlined]
top-level scope at /home/andreasnoack/.julia/packages/SnoopPrecompile/UWvXF/src/SnoopPrecompile.jl:107 [inlined]
top-level scope at /home/andreasnoack/.julia/dev/Term/src/Term.jl:0
jl_fptr_args at /home/andreasnoack/julia/src/gf.c:2291
_jl_invoke at /home/andreasnoack/julia/src/gf.c:2647
ijl_invoke at /home/andreasnoack/julia/src/gf.c:2654
jl_toplevel_eval_flex at /home/andreasnoack/julia/src/toplevel.c:903
jl_eval_module_expr at /home/andreasnoack/julia/src/toplevel.c:203
jl_toplevel_eval_flex at /home/andreasnoack/julia/src/toplevel.c:715
jl_toplevel_eval_flex at /home/andreasnoack/julia/src/toplevel.c:856
jl_toplevel_eval_flex at /home/andreasnoack/julia/src/toplevel.c:856
ijl_toplevel_eval at /home/andreasnoack/julia/src/toplevel.c:921
ijl_toplevel_eval_in at /home/andreasnoack/julia/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1754
jl_fptr_args at /home/andreasnoack/julia/src/gf.c:2291
_jl_invoke at /home/andreasnoack/julia/src/gf.c:2628
ijl_apply_generic at /home/andreasnoack/julia/src/gf.c:2829
_include at ./loading.jl:1814
include at ./Base.jl:452 [inlined]
include_package_for_output at ./loading.jl:1900
jfptr_include_package_for_output_26637 at /home/andreasnoack/julia/usr/lib/julia/sys-debug.so (unknown line)
_jl_invoke at /home/andreasnoack/julia/src/gf.c:2628
ijl_apply_generic at /home/andreasnoack/julia/src/gf.c:2829
jl_apply at /home/andreasnoack/julia/src/julia.h:1869
do_call at /home/andreasnoack/julia/src/interpreter.c:126
eval_value at /home/andreasnoack/julia/src/interpreter.c:228
eval_stmt_value at /home/andreasnoack/julia/src/interpreter.c:177
eval_body at /home/andreasnoack/julia/src/interpreter.c:608
jl_interpret_toplevel_thunk at /home/andreasnoack/julia/src/interpreter.c:764
jl_toplevel_eval_flex at /home/andreasnoack/julia/src/toplevel.c:912
jl_toplevel_eval_flex at /home/andreasnoack/julia/src/toplevel.c:856
ijl_toplevel_eval at /home/andreasnoack/julia/src/toplevel.c:921
ijl_toplevel_eval_in at /home/andreasnoack/julia/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1754
include_string at ./loading.jl:1764 [inlined]
exec_options at ./client.jl:305
_start at ./client.jl:522
jfptr__start_48497 at /home/andreasnoack/julia/usr/lib/julia/sys-debug.so (unknown line)
_jl_invoke at /home/andreasnoack/julia/src/gf.c:2628
ijl_apply_generic at /home/andreasnoack/julia/src/gf.c:2829
jl_apply at /home/andreasnoack/julia/src/julia.h:1869
true_main at /home/andreasnoack/julia/src/jlapi.c:573
jl_repl_entrypoint at /home/andreasnoack/julia/src/jlapi.c:717
jl_load_repl at /home/andreasnoack/julia/cli/loader_lib.c:507
main at /home/andreasnoack/julia/cli/loader_exe.c:58
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /home/andreasnoack/julia/usr/bin/julia-debug (unknown line)
unknown function (ip: (nil))
Allocations: 36085026 (Pool: 36066956; Big: 18070); GC: 48
ERROR: InterruptException:
Stacktrace:
  [1] poptask(W::Base.IntrusiveLinkedListSynchronized{Task})
    @ Base ./task.jl:963
  [2] wait()
    @ Base ./task.jl:972
  [3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] wait(x::Base.Process)
    @ Base ./process.jl:661
  [6] success(x::Base.Process)
    @ Base ./process.jl:523
  [7] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2052
  [8] compilecache
    @ ./loading.jl:2016 [inlined]
  [9] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1660
 [10] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1515
 [11] macro expansion
    @ ./loading.jl:1503 [inlined]
 [12] macro expansion
    @ ./lock.jl:267 [inlined]
 [13] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1466

I was able to reproduce the issue consistently on a Ubuntu server with five different versions of Julia. Precompilation goes through if I remove the output redirection.

The issue initially showed up on CI for a larger project when we tried to upgrade CI to 1.9.0-beta2. The same Manifest works on CI with 1.8.4 and 1.7.3. My understanding from the comment mentioned in the top post is that redirection during precompilation isn't something that is expected to work generally even if it sometimes work.

[FedeClaudi]

Thank you both for the input. I will edit the precompilation code to remove the need for this.