Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittently broken precompile #13200

Closed
yuyichao opened this issue Sep 18, 2015 · 18 comments
Closed

Intermittently broken precompile #13200

yuyichao opened this issue Sep 18, 2015 · 18 comments
Labels
bug Indicates an unexpected problem or unintended behavior compiler:precompilation Precompilation of modules packages Package management and loading

Comments

@yuyichao
Copy link
Contributor

Update: Just to be clear, this issue doesn't seem to be OSX specific althought it does happen more often on OSX travis.

Original post:

This started to happen on the OSX Travis CI since a few days ago for both 0.4 and master. (Also possible that the OSX travis was just not enabled before that). This is a typical failed build (output attached in case someone restarted that build).

The corresponding Linux build looks fine.

INFO: Package database updated
INFO: Recompiling stale cache file /Users/travis/.julia/lib/v0.4/Requests.ji for module Requests.
WARNING: Module JSON uuid did not match cache file
ERROR: LoadError: LoadError: __precompile__(true) but require failed to create a precompiled cache file
 in require at /Users/travis/julia/lib/julia/sys.dylib
 in include at /Users/travis/julia/lib/julia/sys.dylib
 in include_from_node1 at /Users/travis/julia/lib/julia/sys.dylib
 in include at /Users/travis/julia/lib/julia/sys.dylib
 in include_from_node1 at /Users/travis/julia/lib/julia/sys.dylib
 in require at /Users/travis/julia/lib/julia/sys.dylib
 in process_options at /Users/travis/julia/lib/julia/sys.dylib
 in _start at /Users/travis/julia/lib/julia/sys.dylib
while loading /Users/travis/.julia/v0.4/Coverage/src/coveralls.jl, in expression starting on line 5
while loading /Users/travis/.julia/v0.4/Coverage/src/Coverage.jl, in expression starting on line 185
@yuyichao yuyichao added the compiler:precompilation Precompilation of modules label Sep 18, 2015
@Keno
Copy link
Member

Keno commented Sep 18, 2015

I've seen this kind of thing several times (on OS X). Usually it's fine the next time I start julia though.

@yuyichao
Copy link
Contributor Author

Ref #13182 (comment) This might not be OSX specific (but it only happens on Travis OSX and not Travis Linux AFAICT)

@yuyichao
Copy link
Contributor Author

I've just seen this locally as well. It happend when I run Pkg.test("Cairo") after removing all the cache files.

@jiahao
Copy link
Member

jiahao commented Sep 19, 2015

It can be reproduced ~50% of the time running the World of Julia script.

@tkelman tkelman changed the title Broken precompile on OSX? Intermittently broken precompile Sep 24, 2015
@tkelman
Copy link
Contributor

tkelman commented Sep 24, 2015

This happens 100% of the time if you run vagrant up nightlyMZ in PackageEvaluator with 0.4.0-rc1 or newer. Badly needs fixing. Happens after running tests for MacroTools.jl. Unrelated to that package afaict:

==> nightlyMZ: ERROR: LoadError: ArgumentError: DataStructures not found in path
==> nightlyMZ:  in require at ./loading.jl:252
==> nightlyMZ:  in stale_cachefile at loading.jl:439
==> nightlyMZ:  in recompile_stale at loading.jl:457
==> nightlyMZ:  in _require_from_serialized at loading.jl:83
==> nightlyMZ:  in _require_from_serialized at ./loading.jl:109
==> nightlyMZ:  in require at ./loading.jl:219
==> nightlyMZ:  in include at ./boot.jl:261
==> nightlyMZ:  in include_from_node1 at ./loading.jl:304
==> nightlyMZ:  in process_options at ./client.jl:308
==> nightlyMZ:  in _start at ./client.jl:411
==> nightlyMZ: while loading /home/vagrant/pkgeval/src/prepjson.jl, in expression starting on line 11
==> nightlyMZ: INFO: Removing MacroTools v0.2.0
==> nightlyMZ: INFO: Package database updated

edit: this is actually probably a JSON.jl issue, not a base issue: JuliaIO/JSON.jl#123 (comment)

@Keno
Copy link
Member

Keno commented Sep 26, 2015

I just saw this, not sure if it's related, but seems bad as well:

julia> using Plots
[Plots.jl] Default backend: pyplot

julia> using GtkInteract
ERROR: LoadError: MethodError: `module_uuid` has no method matching module_uuid(::Type{ColorTypes.Color{T,N}})
 in stale_cachefile at ./loading.jl:441
 in recompile_stale at ./loading.jl:457
 in _require_from_serialized at ./loading.jl:83
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:219
 in stale_cachefile at ./loading.jl:439
 in recompile_stale at ./loading.jl:457
 in _require_from_serialized at ./loading.jl:83
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:219
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in require at ./loading.jl:243
while loading /Users/kfischer/.julia/v0.5/GtkInteract/src/GtkInteract.jl, in expression starting on line 12

julia>

@timholy
Copy link
Member

timholy commented Sep 26, 2015

That error means that you've got some conflict between Colors and Color somewhere on your system: something is expecting a module named Color, but in reality it's a type.

@tkelman
Copy link
Contributor

tkelman commented Oct 3, 2015

I got this to happen fairly reliably when doing Pkg.add("GeoIP"); Pkg.test("GeoIP") starting from a clean no-packages state on linux64

@mdcfrancis
Copy link

My issue which was closed as a dupe includes a reproducible (simple) example which depends on no external packages.

#14663

The problem occurs when a dependent module 'uses' a package which does not have precompile set and then you use this module. This only occurs first time round with an empty lib directory.

@vtjnash
Copy link
Member

vtjnash commented Jan 13, 2016

that is simply a usage error. __precompile__(true) is a user-annotation stating that all of the preconditions for compilation are going to be met. one of those conditions is that all modules that it uses also has that flag set. the system doesn't have a way to check many of the preconditions, but they will cause various other parts of the system to fail. documentation for this should be added to http://docs.julialang.org/en/latest/manual/modules/#module-initialization-and-precompilation if something is missing

@mdcfrancis
Copy link

@vtjnash as a user error I'd expect the top level package to error on precompile, it does not and causes the crash outlined here. I'd happily see the the precompile error out if all child modules are not so annotated, but I suspect it will break current usage. Perhaps a warning ?

The reproduction supplied boils the issue down to something which does not require a lot of moving parts, but is based on the real problem -

The manifestation of this issue in iJulia etc is due to BinDeps.jl not having precompile set and Conda.jl does (which calls import BinDeps). In isolation this causes no issue (e.g. when you call these on their own), but when you install a package through REQUIRE the 'build step' breaks (this is due to the reuse of the build child process in the package manager.) Specifically you can reproduce the problem by having LibCURL and PyCall in a parent REQUIRE file.

The reason that people see this as intermittent is that it depends on the order of the build and the set of packages specified in the build. Additionally once you have successfully precompiled you are good to go from then on. It is only when you try and build a package with many dependencies in a clean area that this rears its head.

I hope that helps.

@tkelman
Copy link
Contributor

tkelman commented Jan 13, 2016

If BinDeps is safe to precompile, we could fix that case by adding the precompile directive to it. Better checking, if possible, would be worth trying to do.

@mdcfrancis
Copy link

It certainly seems safe since any users of Conda are doing this today.

yuyichao added a commit that referenced this issue Jan 23, 2016
yuyichao added a commit that referenced this issue Jan 23, 2016
@JeffBezanson JeffBezanson added the packages Package management and loading label Feb 12, 2016
@JeffBezanson
Copy link
Member

I think we should reopen this. If we don't support precompiling a module whose dependencies can't be precompiled, we should throw an error as soon as possible to prevent unpredictable breakage later.

@vtjnash
Copy link
Member

vtjnash commented Feb 12, 2016

that's not really something the system can test for

@tkelman
Copy link
Contributor

tkelman commented Feb 12, 2016

We can't check whether dependencies have marked __precompile__(true) ?

tkelman pushed a commit that referenced this issue Mar 7, 2016
Fixes #13200

(cherry picked from commit 59ca442)
ref #14773
@sbromberger
Copy link
Contributor

FWIW, I'm running into this today with using Gadfly. Not quite sure how to fix.

Actually, I'm getting the ERROR: LoadError: MethodError: 'module_uuid' has no method matching module_uuid(::Type{ColorTypes.Color{T,N}}) message.

Edit: This was fixed by nuking ~/.julia/lib/0.4 and forcing a rebuild of Gadfly.

@skanskan
Copy link

I've just installed VS Code and Julia v0.7 and i'm having this same problem:

C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\Nullables\src\nullable.jl:392
in expression starting at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\Nullables\src\Nullables.jl:4
ERROR: LoadError: LoadError: Failed to precompile Nullables to C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\julia_pkgdir\lib\v0.7\Nullables.ji.
Stacktrace:
[1] error at .\error.jl:33 [inlined]
[2] compilecache(::String) at .\loading.jl:648
[3] compilecache at .\loading.jl:605 [inlined]
[4] _require(::Symbol) at .\loading.jl:440
[5] require(::Symbol) at .\loading.jl:333
[6] include(::Module, ::String) at .\boot.jl:292
[7] include_relative(::Module, ::String) at .\loading.jl:521
[8] include at .\sysimg.jl:26 [inlined]
[9] include(::String) at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\JSON\src\JSON.jl:3
[10] top-level scope
[11] include(::Module, ::String) at .\boot.jl:292
[12] include_relative(::Module, ::String) at .\loading.jl:521
[13] include(::Module, ::String) at .\sysimg.jl:26
[14] top-level scope
[15] eval at .\boot.jl:295 [inlined]
[16] top-level scope at .<missing>:2
in expression starting at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\JSON\src\Parser.jl:5
in expression starting at C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\packages\JSON\src\JSON.jl:10
ERROR: LoadError: Failed to precompile JSON to C:\Users\joe.vscode-insiders\extensions\julialang.language-julia-0.9.1\scripts\languageserver\julia_pkgdir\lib\v0.7\JSON.ji.
Stacktrace:
[1] error at .\error.jl:33 [inlined]

and so on

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Indicates an unexpected problem or unintended behavior compiler:precompilation Precompilation of modules packages Package management and loading
Projects
None yet
Development

No branches or pull requests

10 participants