Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extension module is missing from the cache #3557

Open
jishnub opened this issue Jul 27, 2023 · 6 comments
Open

Extension module is missing from the cache #3557

jishnub opened this issue Jul 27, 2023 · 6 comments

Comments

@jishnub
Copy link
Contributor

jishnub commented Jul 27, 2023 •
edited
Loading

In JuliaApproximation/ApproxFunBase.jl#506, I am trying to move Statistics and SparseArrays to weak deps loaded through extensions, and I test downstream packages against this PR. However, I obtained the following warning during precompilation of the downstream package ApproxFunOrthogonalPolynomials:

┌ ApproxFunBaseSparseArraysExt [2448fcf3-777e-5e57-899e-a100cfbe46a1]
│  ┌ Warning: Module ApproxFunBaseSparseArraysExt with build ID ffffffff-ffff-ffff-0000-0056a0f5e447 is missing from the cache.
│  │ This may mean ApproxFunBaseSparseArraysExt [2448fcf3-777e-5e57-899e-a100cfbe46a1] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1793
│  ┌ Error: Error during loading of extension ApproxFunBaseSparseArraysExt of ApproxFunBase, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1797
│  │      [2] _require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1660
│  │      [3] _require_prelocked(uuidkey::Base.PkgId)
│  │        @ Base ./loading.jl:1658
│  │      [4] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1255
│  │      [5] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1290
│  │      [6] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1124
│  │      [7] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1667
│  │      [8] macro expansion
│  │        @ ./loading.jl:1648 [inlined]
│  │      [9] macro expansion
│  │        @ ./lock.jl:267 [inlined]
│  │     [10] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1611
│  │     [11] include
│  │        @ ./Base.jl:457 [inlined]
│  │     [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2049
│  │     [13] top-level scope
│  │        @ stdin:3
│  │     [14] eval
│  │        @ ./boot.jl:370 [inlined]
│  │     [15] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:1903
│  │     [16] include_string
│  │        @ ./loading.jl:1913 [inlined]
│  │     [17] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:305
│  │     [18] _start()
│  │        @ Base ./client.jl:522
│  └ @ Base loading.jl:1261
└

and a couple of other similar warnings (one for Statistics and one for ApproxFunOrthogonalPolynomials). I am unsure about what leads to this warning, although it might be related to the fact that the package is deved while testing.

In case it matters, this is the CI script that is used to initiate the test
@IanButterworth
Copy link
Sponsor Member

IanButterworth commented Aug 25, 2023 •
edited
Loading

Seems like #3571 which is fixed on master and should be fixed in 1.9.4 (it's in the Pkg bump in JuliaLang/julia#50977)

@jishnub
Copy link
Contributor Author

jishnub commented Jan 10, 2024

This is not fixed on v1.9.4 or v1.10.0, but will possibly be resolved by JuliaLang/julia#52841. Reopening until that is merged.

@jishnub jishnub reopened this Jan 10, 2024
@IanButterworth
Copy link
Sponsor Member

That is a different issue to the one above, which doesn't appear have multiple exts that each depend on packages in the sysimage.

If the issue is only happening once you add the above packages to the sysimage, then the other issue is also enough to track this

@jishnub
Copy link
Contributor Author

jishnub commented Jan 11, 2024

Can confirm that this is fixed by JuliaLang/julia#52841

@jishnub
Copy link
Contributor Author

jishnub commented Apr 11, 2024 •
edited
Loading

Oddly, this doesn't seem to be resolved yet, and I am being able to replicate the issue on v1.10.2. The message is

┌ ApproxFunBase → ApproxFunBaseSparseArraysExt
│  ┌ Warning: Module ApproxFunBaseSparseArraysExt with build ID ffffffff-ffff-ffff-0000-003639e1cc73 is missing from the cache.
│  │ This may mean ApproxFunBaseSparseArraysExt [2448fcf3-777e-5e57-899e-a100cfbe46a1] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1948
│  ┌ Error: Error during loading of extension ApproxFunBaseSparseArraysExt of ApproxFunBase, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1952
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1812
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:1803 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:1802 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1295
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1330
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1164
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1819
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1803
│  │     [14] macro expansion
│  │        @ ./loading.jl:1790 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:267 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1753
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1746
│  │     [20] include
│  │        @ ./Base.jl:495 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2222
│  │     [22] top-level scope
│  │        @ stdin:3
│  │     [23] eval
│  │        @ ./boot.jl:385 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2076
│  │     [25] include_string
│  │        @ ./loading.jl:2086 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:316
│  │     [27] _start()
│  │        @ Base ./client.jl:552
│  └ @ Base loading.jl:1301
└  
┌ ApproxFunBase → ApproxFunBaseStatisticsExt
│  ┌ Warning: Module ApproxFunBaseStatisticsExt with build ID ffffffff-ffff-ffff-0000-00377536ceb1 is missing from the cache.
│  │ This may mean ApproxFunBaseStatisticsExt [06f40816-ae97-5315-8130-9e560ff8d968] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1948
│  ┌ Error: Error during loading of extension ApproxFunBaseStatisticsExt of ApproxFunBase, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1952
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1812
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:1803 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:1802 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1295
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1330
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1164
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1819
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1803
│  │     [14] macro expansion
│  │        @ ./loading.jl:1790 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:267 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1753
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1746
│  │     [20] include
│  │        @ ./Base.jl:495 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2222
│  │     [22] top-level scope
│  │        @ stdin:3
│  │     [23] eval
│  │        @ ./boot.jl:385 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2076
│  │     [25] include_string
│  │        @ ./loading.jl:2086 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:316
│  │     [27] _start()
│  │        @ Base ./client.jl:552
│  └ @ Base loading.jl:1301
└

I obtain a similar warning on v1.11.0-beta1.

I have uploaded the project and manifest files at https://github.com/jishnub/ApproxFunBase.jl/tree/extensionserror, so it should hopefully be easy to reproduce.

@jishnub jishnub reopened this Apr 11, 2024
@emstoudenmire emstoudenmire mentioned this issue Apr 12, 2024
Merged
@jishnub
Copy link
Contributor Author

jishnub commented Aug 19, 2024

Update: using julia v1.11.0-rc2, I find

┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│   Base.PkgId(Base.UUID("2448fcf3-777e-5e57-899e-a100cfbe46a1"), "ApproxFunBaseSparseArraysExt")
│   Base.PkgId(Base.UUID("06f40816-ae97-5315-8130-9e560ff8d968"), "ApproxFunBaseStatisticsExt")
└ @ Base.Precompilation precompilation.jl:552
Precompiling project...
  56 dependencies successfully precompiled in 52 seconds. 37 already precompiled. 2 skipped due to circular dependency.

I wonder if this is because some of the dependencies have SparseArrays and Statistics as hard dependencies? So the chain might go ApproxFunBase -> DSP -> Statistics -> ApproxFunBaseStatisticsExt -> ApproxFunBase, therefore creating a cycle?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@IanButterworth @jishnub