Extensions: make loading of extensions independent of what packages are in the sysimage

[KristofferC]

When triggers of extension are in the sysimage it is easy to end up with cycles in package loading. Say we have a package A with exts BExt and CExt and say that B and C is in the sysimage.

• Upon loading A, we will immidiately start to precompile BExt (because the trigger B is "loaded" by virtue of being in the sysimage).
• BExt will load A which will cause CExt to start precompiling (again because C is in the sysimage).
• CExt will load A which will now cause BExt to start loading and we get a cycle.

This is fixed in this PR by instead of looking at what modules are loaded, we look at what modules are actually required and only use that to drive the loading of extensions.

Fixes #52132.

[KristofferC]

@nanosoldier runtests()

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

[KristofferC]

Okay, this PR, as is, is incomplete because it also has to count the dependencies that are loaded. Right now it only considers the package that is explicitly loaded by the user.

[vtjnash]

Oh yeah, this is a very good observation. I think a better solution might be a bit different though--we should make sure instead that all of these extensions are included in the original precompile of the package that resolved them all (rather than trying to split them into separate files). The precompile code should already be perfectly happy for them to live inside the .ji file of the parent, we just need to teach loading.jl to put them there.

[KristofferC]

@nanosoldier runtests(["TranscodingStreams", "EulerAngles", "CodecBase", "GeneralizedSylvesterSolver", "BufferedStreams", "CodecBzip2", "CodecLz4", "CodecZlib", "CodecXz", "CodecZstd", "Sinograms", "Pseudospectra", "ImplicitDifferentiation", "StatsDiscretizations", "StipplePlotly", "MRIOperators", "CategoryData", "GR", "MRISimulation"])

[KristofferC]

@nanosoldier runtests(["TranscodingStreams", "EulerAngles", "CodecBase", "GeneralizedSylvesterSolver", "BufferedStreams", "CodecBzip2", "CodecLz4", "CodecZlib", "CodecXz", "CodecZstd", "Sinograms", "Pseudospectra", "ImplicitDifferentiation", "StatsDiscretizations", "StipplePlotly", "MRIOperators", "CategoryData", "GR", "MRISimulation"])

[KristofferC]

we should make sure instead that all of these extensions are included in the original precompile of the package that resolved them all (rather than trying to split them into separate files). The precompile code should already be perfectly happy for them to live inside the .ji file of the parent, we just need to teach loading.jl to put them there.

I don't understand this really. Let's say you have the extension BCExt for A that requires B and C. You load A and then B and then C. Now BCExt will load. Where do you want to put that cache file? In C? What happens next time when you load things in the order C, A, B? Do you reopen C to look for BCExt? What's the advantage of this?

[vtjnash]

it belongs in A, since B and C are already loaded (in the case where B and C are already loaded when A finishes precompiling)

[KristofferC]

And if they aren't and you just load A by itself (and then B and C)?

[vtjnash]

Then no change is needed, since there is no circularity problem?

[nanosoldier]

Your job failed. Consult the server logs for more details (cc @maleadt).

[maleadt]

This broke Nanosoldier: It hung during build (PkgEval should use a timeout) at the generate_precompile.jl stage:

Collecting and executing precompile statements
...
InitError(mod=:Base, error=UndefVarError(var=:required_modules, scope=Base))
ijl_undefined_var_error at /source/src/rtutils.c:152
ijl_get_binding_or_error at /source/src/module.c:464
__init__ at ./Base.jl:630
unknown function (ip: 0x7f6323b56cbf)
_jl_invoke at /source/src/gf.c:2921 [inlined]
ijl_apply_generic at /source/src/gf.c:3098
jl_apply at /source/src/julia.h:2153 [inlined]
jl_module_run_initializer at /source/src/toplevel.c:76
_finish_julia_init at /source/src/init.c:902
julia_init at /source/src/init.c:843
jl_repl_entrypoint at /source/src/jlapi.c:1087
main at /source/cli/loader_exe.c:58
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)

[KristofferC]

Yeah, I was a bit eager to submit that job...

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

[KristofferC]

@nanosoldier runtests(["CodecZstd", "TranscodingStreams", "CodecBase", "CodecLz4", "CodecBzip2", "CodecZlib", "CodecXz"])

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

[KristofferC]

I realize now that the failing packages are actually failing due to the bugfix here. They never explicitly load Random in their tests but they have an extension that requires both Random and Test to trigger. Their extension used to load due to Random being in the sysimage which isn't enough anymore.