add support for package extensions (#3264)

This is a feature that allows conditional loading of code based on other packages being loaded into the Julia session (similar to Requires.jl). See the added docs for a more detailed description.
JuliaLang · Dec 7, 2022 · 5d8b9dd · 5d8b9dd
1 parent d5ac7ca
commit 5d8b9dd
Show file tree

Hide file tree

Showing 33 changed files with 776 additions and 101 deletions.
diff --git a/docs/src/creating-packages.md b/docs/src/creating-packages.md
@@ -245,6 +245,171 @@ using Test
 Every dependency should in general have a compatibility constraint on it.
 This is an important topic so there is a separate chapter about it: [Compatibility](@ref Compatibility).
 
+## Weak dependencies
+
+!!! note
+    This is a somewhat advanced usage of Pkg which can be skipped for people new to Julia and Julia packages.
+
+A weak dependency is a dependency that will not automatically install when the package is installed but
+you can still control what versions of that package are allowed to be installed by setting compatibility on it.
+These are listed in the project file under the `[weakdeps]` section:
+
+```toml
+[weakdeps]
+SomePackage = "b3785f31-9d33-4cdf-bc73-f646780f1739"
+
+[compat]
+SomePackage = "1.2"
+```
+
+The current usage of this is almost solely limited to "extensions" which is described in the next section.
+
+## Conditional loading of code in packages (Extensions)
+
+!!! note
+    This is a somewhat advanced usage of Pkg which can be skipped for people new to Julia and Julia packages.
+
+It is sometimes desirable to be able to extend some functionality of a package without having to
+unconditionally take on the cost (in terms of e.g. load time) of adding a full dependency on that package.
+A package *extension* is a module in a file (similar to a package) that is automatically loaded when *some other set of packages* are
+loaded into the Julia session. This is very similar to functionality that the external package
+Requires.jl provides, but which is now available directly through Julia.
+
+A useful application of extensions could be for a plotting package that should be able to plot
+objects from a wide variety of different Julia packages.
+Adding all those different Julia packages as dependencies
+could be expensive since they would end up getting loaded even if they were never used.
+Instead, the code required to plot objects for specific packages can be put into separate files
+(extensions) and these are loaded only when the packages that defines the type we want to plot
+are loaded.
+
+Below is an example of how the code can be structured for a use case outlined above:
+
+ `Project.toml`:
+ ```toml
+name = "Plotting"
+version = "0.1.0"
+uuid = "..."
+
+[weakdeps]
+Contour = "d38c429a-6771-53c6-b99e-75d170b6e991"
+
+[extensions]
+# name of extension to the left
+# extension dependencies required to load the extension to the right
+# use a list for multiple extension dependencies
+ContourExt = "Contour" 
+
+[compat]
+Contour = "0.6.2"
+```
+
+`src/Plotting.jl`:
+```julia
+module Plotting
+
+function plot(x::Vector)
+    # Some functionality for plotting a vector here
+end
+
+end # module
+```
+
+`ext/ContourExt.jl` (can also be in `ext/ContourExt/ContourExt.jl`):
+```julia
+module ContourExt # Should be same name as the file (just like a normal package)
+
+using Plotting, Contour
+
+function Plotting.plot(c::Contour.ContourCollection)
+    # Some functionality for plotting a contour here
+end
+
+end # module
+```
+
+A user that depends on `Plotting` will not pay the cost of the "extension" inside the `ContourExt` module.
+It is only when the `Contour` package actually gets loaded that the `ContourExt` extension is loaded
+and provides the new functionality.
+
+If one considers `ContourExt` as a completely separate package, it could be argued that defining `Plotting.plot(c::Contour.ContourCollection)` is
+type piracy since `ContourExt` does not "own" neigher the method `Plotting.plot` nor the type `Contour.ContourCollection`.
+However, for extensions, it is ok to assume that the extension owns the methods in its parent package.
+In fact, this type of "type piracies" is one of the most standard use cases for extensions.
+
+An extension will only be loaded if the extension dependencies are loaded from the same environment or environments higher in the environment stack than the package itself.
+
+!!! compat
+    Often you will put the extension dependencies into the `test` target so they are loaded when running e.g. `Pkg.test()`. On earlier Julia versions
+    this requires you to also put the package in the `[extras]` section. This is unfortunate but the project verifier on older Julia versions will
+    complain if this is not done.
+
+!!! note
+    If you use a manifest generated by a Julia version that does not know about extensions with a Julia version that does
+    know about them, the extensions will not load. This is because the manifest lacks some information that tells Julia
+    when it should load these packages. So make sure you use a manifest generated at least the Julia version you are using.
+
+### Backwards compatibility
+
+This section discusses various methods for using extensions on Julia versions that support them,
+while simultaneously providing similar functionality on older Julia versions.
+#### Requires.jl
+
+This section is relevant if you are currently using Requires.jl but want to transition to using extensions (while still having Requires be used on Julia versions that do not support extensions).
+This is done by making the following changes (using the example above):
+
+- Add the following to the package file. This makes it so that Requires.jl loads and inserts the
+  callback only when extensions are not supported
+  ```julia
+  # This symbol is only defined on Julia versions that support extensions
+  if !isdefined(Base, :get_extension)
+  using Requires
+  function __init__()
+      @require Contour = "d38c429a-6771-53c6-b99e-75d170b6e991" include("../ext/ContourExt.jl")
+  end
+  end
+  ```
+  or if you have other things in your `__init__()` function:
+  ```julia
+  if !isdefined(Base, :get_extension)
+  using Requires
+  end
+
+  function __init__()
+      # Other init functionality here
+
+      @static if !isdefined(Base, :get_extension)
+          @require Contour = "d38c429a-6771-53c6-b99e-75d170b6e991" include("../ext/ContourExt.jl)
+      end
+  end
+  ```
+- Do the following change in the extensions for loading the extension dependency:
+ ```julia
+ isdefined(Base, :get_extension) ? (using Contour) : (using ..Contour)
+ ```
+
+The package should now work with Requires.jl on Julia versions before extensions were introduced
+and with extensions afterward.
+
+####  Transition from normal dependency to extension
+
+This section is relevant if you have a normal dependency that you want to transition be an extension (while still having the dependency be a normal dependency on Julia versions that do not support extensions).
+This is done by making the following changes (using the example above):
+
+- Make sure that the package is **both** in the `[deps]` and `[weakdeps]` section. Newer Julia versions will ignore dependencis in `[deps]` that are also in `[weakdeps]`.
+- Add the following to your main package file (typically at the bottom):
+  ```julia
+  if !isdefined(Base, :get_extension)
+    include("../ext/ContourExt.jl")
+  end
+  ```
+
+#### Using an extension while supporting older Julia version
+
+If you want to use use an extension with compatibility constraints while supporting earlier Julia
+versions you have to duplicate the packages under `[weakdeps]` into `[extras]`. This is an unfortunate
+duplication but without doing this the project verifier under older Julia versions will complain (error).
+
 ## Package naming guidelines
 
 Package names should be sensible to most Julia users, *even to those who are not domain experts*.

diff --git a/src/API.jl b/src/API.jl
@@ -301,9 +301,11 @@ function rm(ctx::Context, pkgs::Vector{PackageSpec}; mode=PKGMODE_PROJECT, all_p
     ensure_resolved(ctx, ctx.env.manifest, pkgs)
 
     Operations.rm(ctx, pkgs; mode)
+
     return
 end
 
+
 function append_all_pkgs!(pkgs, ctx, mode)
     if mode == PKGMODE_PROJECT || mode == PKGMODE_COMBINED
         for (name::String, uuid::UUID) in ctx.env.project.deps
@@ -1094,15 +1096,41 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
         for (name, uuid) in ctx.env.project.deps if !Base.in_sysimage(Base.PkgId(uuid, name))
     ]
 
-    man = ctx.env.manifest
-    deps_pair_or_nothing = Iterators.map(man) do dep
+    exts = Dict{Base.PkgId, String}() # ext -> parent
+    # make a flat map of each dep and its deps
+    depsmap = Dict{Base.PkgId, Vector{Base.PkgId}}()
+    pkg_specs = PackageSpec[]
+    for dep in ctx.env.manifest
         pkg = Base.PkgId(first(dep), last(dep).name)
-        Base.in_sysimage(pkg) && return nothing
+        Base.in_sysimage(pkg) && continue
         deps = [Base.PkgId(last(x), first(x)) for x in last(dep).deps]
-        return pkg => filter!(!Base.in_sysimage, deps)
+        depsmap[pkg] = filter!(!Base.in_sysimage, deps)
+        # add any extensions
+        weakdeps = last(dep).weakdeps
+        for (ext_name, extdep_names) in last(dep).exts
+            ext_deps = Base.PkgId[]
+            push!(ext_deps, pkg) # depends on parent package
+            all_extdeps_available = true
+            extdep_names = extdep_names isa String ? String[extdep_names] : extdep_names
+            for extdep_name in extdep_names
+                extdep_uuid = weakdeps[extdep_name]
+                if extdep_uuid in keys(ctx.env.manifest.deps)
+                    push!(ext_deps, Base.PkgId(extdep_uuid, extdep_name))
+                else
+                    all_extdeps_available = false
+                    break
+                end
+            end
+            all_extdeps_available || continue
+            ext_uuid = Base.uuid5(pkg.uuid, ext_name)
+            ext = Base.PkgId(ext_uuid, ext_name)
+            push!(pkg_specs, PackageSpec(uuid = ext_uuid, name = ext_name)) # create this here as the name cannot be looked up easily later via the uuid
+            depsmap[ext] = filter!(!Base.in_sysimage, ext_deps)
+            exts[ext] = pkg.name
+        end
     end
-    depsmap = Dict{Base.PkgId, Vector{Base.PkgId}}(Iterators.filter(!isnothing, deps_pair_or_nothing)) #flat map of each dep and its deps
 
+    # if the active environment is a package, add that
     ctx_env_pkg = ctx.env.pkg
     if ctx_env_pkg !== nothing && isfile( joinpath( dirname(ctx.env.project_file), "src", "$(ctx_env_pkg.name).jl") )
         depsmap[Base.PkgId(ctx_env_pkg.uuid, ctx_env_pkg.name)] = [
@@ -1112,21 +1140,25 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
         push!(direct_deps, Base.PkgId(ctx_env_pkg.uuid, ctx_env_pkg.name))
     end
 
+    # return early if no deps
     isempty(depsmap) && return
 
+    # initialize signalling
     started = Dict{Base.PkgId,Bool}()
     was_processed = Dict{Base.PkgId,Base.Event}()
     was_recompiled = Dict{Base.PkgId,Bool}()
-    pkg_specs = PackageSpec[]
     for pkgid in keys(depsmap)
         started[pkgid] = false
         was_processed[pkgid] = Base.Event()
         was_recompiled[pkgid] = false
         push!(pkg_specs, get_or_make_pkgspec(pkg_specs, ctx, pkgid.uuid))
     end
+
+    # remove packages that are suspended because they errored before
+    # note that when `Pkg.precompile` is manually called, all suspended packages are unsuspended
     precomp_prune_suspended!(pkg_specs)
 
-    # guarding against circular deps
+    # find and guard against circular deps
     circular_deps = Base.PkgId[]
     function in_deps(_pkgs, deps, dmap)
         isempty(deps) && return false
@@ -1143,8 +1175,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
         @warn """Circular dependency detected. Precompilation will be skipped for:\n  $(join(string.(circular_deps), "\n  "))"""
     end
 
+    # if a list of packages is given, restrict to dependencies of given packages
     if !isempty(pkgs)
-        # if a list of packages is given, restrict to dependencies of given packages
         function collect_all_deps(depsmap, dep, alldeps=Base.PkgId[])
             append!(alldeps, depsmap[dep])
             for _dep in depsmap[dep]
@@ -1162,6 +1194,7 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
         filter!(d->in(first(d), keep), depsmap)
         isempty(depsmap) && pkgerror("No direct dependencies found matching $(repr(pkgs))")
     end
+    target = string(isempty(pkgs) ? "project" : join(pkgs, ", "), "...")
 
     pkg_queue = Base.PkgId[]
     failed_deps = Dict{Base.PkgId, String}()
@@ -1199,7 +1232,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
         end
     end
 
-    t_print = @async begin # fancy print loop
+    ## fancy print loop
+    t_print = @async begin
         try
             wait(first_started)
             (isempty(pkg_queue) || interrupted_or_done.set) && return
@@ -1234,7 +1268,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
                         final_loop || print(iostr, sprint(io -> show_progress(io, bar; termwidth = displaysize(ctx.io)[2]); context=io), "\n")
                         for dep in pkg_queue_show
                             loaded = warn_loaded && haskey(Base.loaded_modules, dep)
-                            name = dep in direct_deps ? dep.name : string(color_string(dep.name, :light_black))
+                            _name = haskey(exts, dep) ? string(exts[dep], " → ", dep.name) : dep.name
+                            name = dep in direct_deps ? _name : string(color_string(_name, :light_black))
                             if dep in precomperr_deps
                                 print(iostr, color_string("  ? ", Base.warn_color()), name, "\n")
                             elseif haskey(failed_deps, dep)
@@ -1274,7 +1309,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
     end
     tasks = Task[]
     Base.LOADING_CACHE[] = Base.LoadingCache()
-    for (pkg, deps) in depsmap # precompilation loop
+    ## precompilation loop
+    for (pkg, deps) in depsmap
         paths = Base.find_all_in_cache_path(pkg)
         sourcepath = Base.locate_package(pkg)
         if sourcepath === nothing
@@ -1307,9 +1343,10 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call::
                     Base.acquire(parallel_limiter)
                     is_direct_dep = pkg in direct_deps
                     iob = IOBuffer()
-                    name = is_direct_dep ? pkg.name : string(color_string(pkg.name, :light_black))
+                    _name = haskey(exts, pkg) ? string(exts[pkg], " → ", pkg.name) : pkg.name
+                    name = is_direct_dep ? _name : string(color_string(_name, :light_black))
                     !fancyprint && lock(print_lock) do
-                        isempty(pkg_queue) && printpkgstyle(io, :Precompiling, "environment...")
+                        isempty(pkg_queue) && printpkgstyle(io, :Precompiling, target)
                     end
                     push!(pkg_queue, pkg)
                     started[pkg] = true
@@ -1609,13 +1646,14 @@ end
 
 @deprecate status(mode::PackageMode) status(mode=mode)
 
-function status(ctx::Context, pkgs::Vector{PackageSpec}; diff::Bool=false, mode=PKGMODE_PROJECT, outdated::Bool=false, compat::Bool=false, io::IO=stdout_f(), kwargs...)
+function status(ctx::Context, pkgs::Vector{PackageSpec}; diff::Bool=false, mode=PKGMODE_PROJECT, outdated::Bool=false, compat::Bool=false, extensions::Bool=false, io::IO=stdout_f())
     if compat
         diff && pkgerror("Compat status has no `diff` mode")
         outdated && pkgerror("Compat status has no `outdated` mode")
+        extensions && pkgerror("Compat status has no `extensions` mode")
         Operations.print_compat(ctx, pkgs; io)
     else
-        Operations.status(ctx.env, ctx.registries, pkgs; mode, git_diff=diff, io, outdated)
+        Operations.status(ctx.env, ctx.registries, pkgs; mode, git_diff=diff, io, outdated, extensions)
     end
     return nothing
 end