make extension not load twice on workers

[KristofferC]

Fixes #49437

The issue is the following:

• Distributed wants to load packages on workers when someone loads a package on the master.
• Therefore it inserts a package callback to load the package whenever the master loads a package.
• Upon all the triggers for a package getting loaded on the master, this causes two copies of the extension to try to get loaded:
  • One from the callback (fired from the master)
  • And one due to all the triggers being loaded on the worker.

An alternative to this PR could be to never trigger the extension loading on workers and always let the callback handle it but I don't know how to identify a worker (myid) in Base.

[KristofferC]

The alternative would look something like

 function run_extension_callbacks(pkgid::PkgId)
+    distributed_id = PkgId(UUID((0x8ba89e20_285c_5b6f, 0x9357_94700520ee1b)), "
Distributed")
+    Distributed = get(loaded_modules, distributed_id, nothing)
+    if Distributed !== nothing && isdefined(Distributed, :myid)
+        id = @invokelatest Distributed.myid()
+        # Do not trigger extensions to load on workers, they will be loaded by the callback Distributed inserts
+        id != 1 && return
+    end

[marius311]

Thanks for this quick fix. The solution in the PR seems right to me because at least based on your description / my basic understanding (ie I havent actually tested), it will handle this correctly:

using Foo, Distributed
addprocs(2)
using Bar

which, based on current rules, should load Foo, Bar, and BarFooExt on master, and only Bar on workers. Conversely, letting master trigger the worker-loads I don't know if it could really handle that easily.

[vtjnash]

Shouldn't Base.toplevel_load[] be false already, since the extension is not the toplevel load?

[KristofferC]

Shouldn't Base.toplevel_load[] be false already, since the extension is not the toplevel load?

I don't know the exact definition of that but adding

        @show Base.toplevel_load[]
        @show mod

to

julia/stdlib/Distributed/src/Distributed.jl

Lines 74 to 85 in f84fb5b

	function _require_callback(mod::Base.PkgId)
	if Base.toplevel_load[] && myid() == 1 && nprocs() > 1
	# broadcast top-level (e.g. from Main) import/using from node 1 (only)
	@sync for p in procs()
	p == 1 && continue
	@async_unwrap remotecall_wait(p) do
	Base.require(mod)
	nothing
	end
	end
	end
	end

shows:

Base.toplevel_load[] = true
mod = BarFooExt [4863ffa8-0e90-52b0-a292-a281b01911ec]

[KristofferC]

I think an extension loading like this is arguably a top level load, no; it doesn't get loaded into any other package. The "extension callback" here gets run after we have loaded all the triggers and at that point we are at toplevel again calling require_prelocked.

[KristofferC]

I'll go with this for now, if someone has a cleaner solution it can always be tweaked.