Random.init() and Base.PCRE.init() may be required in Julia >= 1.3 #277

tkf · 2019-09-22T21:50:30Z

FYI:

PyJulia needed Base.PCRE.__init__() Fix system image builder for Julia >= 1.3 JuliaPy/pyjulia#326
PyCall needed Random.__init__() Call Random.__init__() manually to fix AOT with Julia 1.3 JuliaPy/PyCall.jl#713

The text was updated successfully, but these errors were encountered:

asinghvi17 · 2019-09-22T22:38:17Z

Interesting. Do they use any of the stdlib's functionality that other packages don't? Makie, for example, while depending on Random, gets by without needing Random.__init__().

tkf · 2019-09-22T22:46:32Z

If you only require Random and don't call its functions in the --output-o mode, it should be fine. Pkg seems to call shuffle!.

asinghvi17 · 2019-09-23T01:28:40Z

https://github.com/JuliaLang/julia/blob/784eb57a06aaab95bc4c22f288bd0c9f2b1687ca/stdlib/Random/src/RNGs.jl#L307-L309
is Random.__init__(). It seems like it would fix your issue...not sure what the fix would be on master, though.

PackageCompiler.jl/src/snooping.jl

Line 4 in 06a49ca

const packages_needing_initialization = [:GR, :Unitful]

is what you could edit in sd-notomls.

tkf · 2019-09-23T01:50:54Z

Thanks, I know why it's causing the trouble and how to work-around the problem (it's already implemented). I should have clarified in the OP that I'm just reporting it because I thought that PackageCompiler.jl maintainers may be interested in incorporating the fix. Like Base.__init__ and Sys.__init__, it is probably a good idea to call them as early as possible.

KristofferC · 2019-10-17T13:22:19Z

Out of curiosity, what ends up calling shuffle! in Pkg when output-o is active?

tkf · 2019-10-17T23:16:03Z

Here is the traceback of the error
https://travis-ci.org/JuliaPy/PyCall.jl/jobs/585892454#L803-L808

Note that PyCall is not using PackageCompiler's snoop-based API because of JuliaLang/julia#31156. It has a custom script for testing sysimage compatibility: https://github.com/JuliaPy/PyCall.jl/blob/9f4387535936f275e3582ebdc87cc2ef1f210030/aot/precompile.jl

Actually, looking at the traceback, it looks like shuffle! is not called via Pkg.jl since the signature is shuffle!(::Array{Symbol,1}). It looks like it is happening via jl_load_ C API so I'm not sure what is going on. Sorry about blaming Pkg.jl without enough investigation.

KristofferC · 2019-10-18T05:14:44Z

But that should run in a normal process (without --output-o) but with the --trace-compile flag active. The collected precompile statements are then what is run in the process doing output, but that doesn't run any code.

tkf · 2019-10-18T05:32:39Z

I know that's the ideal/correct way but as I said snooping generated a precompile statement that broke --output-o (at least in Julia <= 1.1).

But that's what I wanted to look into. Provided that PackageCompiler.jl is invoking __init__ of packages, I thought that was necessary for invoking all top-level statements of downstream packages. I have to admit I was puzzled why that's necessary since in principle the evaluated result of the top level code can be de-serialized from the precompilation cache. I knew I had to look into what's going on inside --output-o process but at the same time I needed a working solution... So, the ad-hoc (maybe incorrect) solution I've been using is to execute code in --output-o process and try to cleanup non-serializable states via atexit_hooks.

tkf · 2019-10-18T05:40:25Z

JuliaLang/julia#31156 is still reproducible with Julia 1.4-DEV.

IanButterworth · 2021-11-01T14:56:54Z

PackageCompiler is highly reliable for me, but I just hit a glitch with this that looks exactly like JuliaLang/julia#33471, without changing anything in the setup

julia 1.6.3

ERROR: LoadError: LoadError: AssertionError: 0 < tid <= length(THREAD_RNGs)
Stacktrace:
  [1] _rng_length_assert()
    @ Random /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:381
  [2] default_rng(tid::Int64)
    @ Random /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:372
  [3] default_rng
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:370 [inlined]
  [4] rand! (repeats 2 times)
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Random/src/Random.jl:264 [inlined]
  [5] _rand(#unused#::Type{ColorTypes.Gray{FixedPointNumbers.N0f8}}, sz::Tuple{Int64, Int64})
    @ ColorTypes ~/.julia/packages/ColorTypes/6m8P7/src/operations.jl:97
  [6] rand
    @ ~/.julia/packages/ColorTypes/6m8P7/src/operations.jl:120 [inlined]
  [7] rand(#unused#::Type{ColorTypes.Gray{FixedPointNumbers.N0f8}}, d::Int64, dims::Int64)
    @ Random /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Random/src/Random.jl:292
  [8] (::TiffImages.var"#14#15")(fpath::String, #unused#::IOStream)
    @ TiffImages ~/.julia/packages/TiffImages/XzftO/src/TiffImages.jl:40
  [9] mktemp(fn::TiffImages.var"#14#15", parent::String)
    @ Base.Filesystem ./file.jl:703
 [10] mktemp(fn::Function)
    @ Base.Filesystem ./file.jl:701
 [11] top-level scope
    @ ~/.julia/packages/TiffImages/XzftO/src/TiffImages.jl:37
 [12] include
    @ ./Base.jl:386 [inlined]
 [13] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1072
 [14] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:936

KristofferC · 2021-11-01T20:19:47Z

This happens when for some reason the package fails to get precompiled before the sysimage is attempted to be created. If the package is not precompiled, Julia will try to run all toplevel code and if that includes a call to rand it will fail because the process generating the sysimage does not run __init__ methods and Random.__init__ is required for a functional RNG.

PackageCompiler.jl/src/PackageCompiler.jl

Lines 219 to 227 in 0b31703

    
           function ensurecompiled(project, packages, sysimage) 
        
               length(packages) == 0 && return 
        
               # TODO: Only precompile `packages` (should be available in Pkg 1.8) 
        
               cmd = `$(get_julia_cmd()) --sysimage=$sysimage -e 'using Pkg; Pkg.precompile()'` 
        
               splitter = Sys.iswindows() ? ';' : ':' 
        
               cmd = addenv(cmd, "JULIA_LOAD_PATH" => "$project$(splitter)@stdlib") 
        
               run(cmd) 
        
               return 
        
           end

is meant to ensure everything gets precompiled.

IanButterworth · 2021-11-01T21:11:38Z

The error I saw was repeatable on that setup after I first hit it, until I wiped the depot compiled and packages dirs.
Perhaps there was something somehow precompiled with a glitch that was too subtle for stalecache to detect it, but loading to consider it unusable..?

KristofferC · 2021-11-01T21:20:20Z

Running with ENV["JULIA_DEBIG"] = "loading" might have given some clue?

IanButterworth · 2021-11-01T21:24:07Z

If I come across it again I will. This was a device in the field so didn't have much debug time

tkf mentioned this issue Sep 22, 2019

Call Random.__init__() manually to fix AOT with Julia 1.3 JuliaPy/PyCall.jl#713

Closed

This was referenced Oct 4, 2019

AssertionError: 0 < tid <= length(THREAD_RNGs) compiling Random.shuffle! (e.g. MacroTools) fails on 1.3 JuliaLang/julia#33471

Closed

ERROR: LoadError: LoadError: LoadError: LoadError: LoadError: AssertionError: 0 < tid <= length(THREAD_RNGs) #268

Closed

tkf mentioned this issue Oct 23, 2019

Modules' __init__ should be called in imported order #192

Closed

KristofferC mentioned this issue Feb 9, 2020

Total rewrite of the package (aka make PackageCompilerX the new PackageCompiler) #304

Merged

SimonDanisch closed this as completed in #304 Feb 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Random.init() and Base.PCRE.init() may be required in Julia >= 1.3 #277

Random.init() and Base.PCRE.init() may be required in Julia >= 1.3 #277

tkf commented Sep 22, 2019

asinghvi17 commented Sep 22, 2019

tkf commented Sep 22, 2019

asinghvi17 commented Sep 23, 2019 •

edited

Loading

tkf commented Sep 23, 2019

KristofferC commented Oct 17, 2019

tkf commented Oct 17, 2019

KristofferC commented Oct 18, 2019

tkf commented Oct 18, 2019 •

edited

Loading

tkf commented Oct 18, 2019

IanButterworth commented Nov 1, 2021

KristofferC commented Nov 1, 2021

IanButterworth commented Nov 1, 2021

KristofferC commented Nov 1, 2021

IanButterworth commented Nov 1, 2021

Random.__init__() and Base.PCRE.__init__() may be required in Julia >= 1.3 #277

Random.__init__() and Base.PCRE.__init__() may be required in Julia >= 1.3 #277

Comments

tkf commented Sep 22, 2019

asinghvi17 commented Sep 22, 2019

tkf commented Sep 22, 2019

asinghvi17 commented Sep 23, 2019 • edited Loading

tkf commented Sep 23, 2019

KristofferC commented Oct 17, 2019

tkf commented Oct 17, 2019

KristofferC commented Oct 18, 2019

tkf commented Oct 18, 2019 • edited Loading

tkf commented Oct 18, 2019

IanButterworth commented Nov 1, 2021

KristofferC commented Nov 1, 2021

IanButterworth commented Nov 1, 2021

KristofferC commented Nov 1, 2021

IanButterworth commented Nov 1, 2021

Random.init() and Base.PCRE.init() may be required in Julia >= 1.3 #277

Random.init() and Base.PCRE.init() may be required in Julia >= 1.3 #277

asinghvi17 commented Sep 23, 2019 •

edited

Loading

tkf commented Oct 18, 2019 •

edited

Loading