diff --git a/docs/src/creating-packages.md b/docs/src/creating-packages.md index ebf86517b4..f6149a380d 100644 --- a/docs/src/creating-packages.md +++ b/docs/src/creating-packages.md @@ -245,6 +245,171 @@ using Test Every dependency should in general have a compatibility constraint on it. This is an important topic so there is a separate chapter about it: [Compatibility](@ref Compatibility). +## Weak dependencies + +!!! note + This is a somewhat advanced usage of Pkg which can be skipped for people new to Julia and Julia packages. + +A weak dependency is a dependency that will not automatically install when the package is installed but +you can still control what versions of that package are allowed to be installed by setting compatibility on it. +These are listed in the project file under the `[weakdeps]` section: + +```toml +[weakdeps] +SomePackage = "b3785f31-9d33-4cdf-bc73-f646780f1739" + +[compat] +SomePackage = "1.2" +``` + +The current usage of this is almost solely limited to "extensions" which is described in the next section. + +## Conditional loading of code in packages (Extensions) + +!!! note + This is a somewhat advanced usage of Pkg which can be skipped for people new to Julia and Julia packages. + +It is sometimes desirable to be able to extend some functionality of a package without having to +unconditionally take on the cost (in terms of e.g. load time) of adding a full dependency on that package. +A package *extension* is a module in a file (similar to a package) that is automatically loaded when *some other set of packages* are +loaded into the Julia session. This is very similar to functionality that the external package +Requires.jl provides, but which is now available directly through Julia. + +A useful application of extensions could be for a plotting package that should be able to plot +objects from a wide variety of different Julia packages. +Adding all those different Julia packages as dependencies +could be expensive since they would end up getting loaded even if they were never used. +Instead, the code required to plot objects for specific packages can be put into separate files +(extensions) and these are loaded only when the packages that defines the type we want to plot +are loaded. + +Below is an example of how the code can be structured for a use case outlined above: + + `Project.toml`: + ```toml +name = "Plotting" +version = "0.1.0" +uuid = "..." + +[weakdeps] +Contour = "d38c429a-6771-53c6-b99e-75d170b6e991" + +[extensions] +# name of extension to the left +# extension dependencies required to load the extension to the right +# use a list for multiple extension dependencies +ContourExt = "Contour" + +[compat] +Contour = "0.6.2" +``` + +`src/Plotting.jl`: +```julia +module Plotting + +function plot(x::Vector) + # Some functionality for plotting a vector here +end + +end # module +``` + +`ext/ContourExt.jl` (can also be in `ext/ContourExt/ContourExt.jl`): +```julia +module ContourExt # Should be same name as the file (just like a normal package) + +using Plotting, Contour + +function Plotting.plot(c::Contour.ContourCollection) + # Some functionality for plotting a contour here +end + +end # module +``` + +A user that depends on `Plotting` will not pay the cost of the "extension" inside the `ContourExt` module. +It is only when the `Contour` package actually gets loaded that the `ContourExt` extension is loaded +and provides the new functionality. + +If one considers `ContourExt` as a completely separate package, it could be argued that defining `Plotting.plot(c::Contour.ContourCollection)` is +type piracy since `ContourExt` does not "own" neigher the method `Plotting.plot` nor the type `Contour.ContourCollection`. +However, for extensions, it is ok to assume that the extension owns the methods in its parent package. +In fact, this type of "type piracies" is one of the most standard use cases for extensions. + +An extension will only be loaded if the extension dependencies are loaded from the same environment or environments higher in the environment stack than the package itself. + +!!! compat + Often you will put the extension dependencies into the `test` target so they are loaded when running e.g. `Pkg.test()`. On earlier Julia versions + this requires you to also put the package in the `[extras]` section. This is unfortunate but the project verifier on older Julia versions will + complain if this is not done. + +!!! note + If you use a manifest generated by a Julia version that does not know about extensions with a Julia version that does + know about them, the extensions will not load. This is because the manifest lacks some information that tells Julia + when it should load these packages. So make sure you use a manifest generated at least the Julia version you are using. + +### Backwards compatibility + +This section discusses various methods for using extensions on Julia versions that support them, +while simultaneously providing similar functionality on older Julia versions. +#### Requires.jl + +This section is relevant if you are currently using Requires.jl but want to transition to using extensions (while still having Requires be used on Julia versions that do not support extensions). +This is done by making the following changes (using the example above): + +- Add the following to the package file. This makes it so that Requires.jl loads and inserts the + callback only when extensions are not supported + ```julia + # This symbol is only defined on Julia versions that support extensions + if !isdefined(Base, :get_extension) + using Requires + function __init__() + @require Contour = "d38c429a-6771-53c6-b99e-75d170b6e991" include("../ext/ContourExt.jl") + end + end + ``` + or if you have other things in your `__init__()` function: + ```julia + if !isdefined(Base, :get_extension) + using Requires + end + + function __init__() + # Other init functionality here + + @static if !isdefined(Base, :get_extension) + @require Contour = "d38c429a-6771-53c6-b99e-75d170b6e991" include("../ext/ContourExt.jl) + end + end + ``` +- Do the following change in the extensions for loading the extension dependency: + ```julia + isdefined(Base, :get_extension) ? (using Contour) : (using ..Contour) + ``` + +The package should now work with Requires.jl on Julia versions before extensions were introduced +and with extensions afterward. + +#### Transition from normal dependency to extension + +This section is relevant if you have a normal dependency that you want to transition be an extension (while still having the dependency be a normal dependency on Julia versions that do not support extensions). +This is done by making the following changes (using the example above): + +- Make sure that the package is **both** in the `[deps]` and `[weakdeps]` section. Newer Julia versions will ignore dependencis in `[deps]` that are also in `[weakdeps]`. +- Add the following to your main package file (typically at the bottom): + ```julia + if !isdefined(Base, :get_extension) + include("../ext/ContourExt.jl") + end + ``` + +#### Using an extension while supporting older Julia version + +If you want to use use an extension with compatibility constraints while supporting earlier Julia +versions you have to duplicate the packages under `[weakdeps]` into `[extras]`. This is an unfortunate +duplication but without doing this the project verifier under older Julia versions will complain (error). + ## Package naming guidelines Package names should be sensible to most Julia users, *even to those who are not domain experts*. diff --git a/src/API.jl b/src/API.jl index 3cdffc9644..6abfbb64e0 100644 --- a/src/API.jl +++ b/src/API.jl @@ -301,9 +301,11 @@ function rm(ctx::Context, pkgs::Vector{PackageSpec}; mode=PKGMODE_PROJECT, all_p ensure_resolved(ctx, ctx.env.manifest, pkgs) Operations.rm(ctx, pkgs; mode) + return end + function append_all_pkgs!(pkgs, ctx, mode) if mode == PKGMODE_PROJECT || mode == PKGMODE_COMBINED for (name::String, uuid::UUID) in ctx.env.project.deps @@ -1094,15 +1096,41 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: for (name, uuid) in ctx.env.project.deps if !Base.in_sysimage(Base.PkgId(uuid, name)) ] - man = ctx.env.manifest - deps_pair_or_nothing = Iterators.map(man) do dep + exts = Dict{Base.PkgId, String}() # ext -> parent + # make a flat map of each dep and its deps + depsmap = Dict{Base.PkgId, Vector{Base.PkgId}}() + pkg_specs = PackageSpec[] + for dep in ctx.env.manifest pkg = Base.PkgId(first(dep), last(dep).name) - Base.in_sysimage(pkg) && return nothing + Base.in_sysimage(pkg) && continue deps = [Base.PkgId(last(x), first(x)) for x in last(dep).deps] - return pkg => filter!(!Base.in_sysimage, deps) + depsmap[pkg] = filter!(!Base.in_sysimage, deps) + # add any extensions + weakdeps = last(dep).weakdeps + for (ext_name, extdep_names) in last(dep).exts + ext_deps = Base.PkgId[] + push!(ext_deps, pkg) # depends on parent package + all_extdeps_available = true + extdep_names = extdep_names isa String ? String[extdep_names] : extdep_names + for extdep_name in extdep_names + extdep_uuid = weakdeps[extdep_name] + if extdep_uuid in keys(ctx.env.manifest.deps) + push!(ext_deps, Base.PkgId(extdep_uuid, extdep_name)) + else + all_extdeps_available = false + break + end + end + all_extdeps_available || continue + ext_uuid = Base.uuid5(pkg.uuid, ext_name) + ext = Base.PkgId(ext_uuid, ext_name) + push!(pkg_specs, PackageSpec(uuid = ext_uuid, name = ext_name)) # create this here as the name cannot be looked up easily later via the uuid + depsmap[ext] = filter!(!Base.in_sysimage, ext_deps) + exts[ext] = pkg.name + end end - depsmap = Dict{Base.PkgId, Vector{Base.PkgId}}(Iterators.filter(!isnothing, deps_pair_or_nothing)) #flat map of each dep and its deps + # if the active environment is a package, add that ctx_env_pkg = ctx.env.pkg if ctx_env_pkg !== nothing && isfile( joinpath( dirname(ctx.env.project_file), "src", "$(ctx_env_pkg.name).jl") ) depsmap[Base.PkgId(ctx_env_pkg.uuid, ctx_env_pkg.name)] = [ @@ -1112,21 +1140,25 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: push!(direct_deps, Base.PkgId(ctx_env_pkg.uuid, ctx_env_pkg.name)) end + # return early if no deps isempty(depsmap) && return + # initialize signalling started = Dict{Base.PkgId,Bool}() was_processed = Dict{Base.PkgId,Base.Event}() was_recompiled = Dict{Base.PkgId,Bool}() - pkg_specs = PackageSpec[] for pkgid in keys(depsmap) started[pkgid] = false was_processed[pkgid] = Base.Event() was_recompiled[pkgid] = false push!(pkg_specs, get_or_make_pkgspec(pkg_specs, ctx, pkgid.uuid)) end + + # remove packages that are suspended because they errored before + # note that when `Pkg.precompile` is manually called, all suspended packages are unsuspended precomp_prune_suspended!(pkg_specs) - # guarding against circular deps + # find and guard against circular deps circular_deps = Base.PkgId[] function in_deps(_pkgs, deps, dmap) isempty(deps) && return false @@ -1143,8 +1175,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: @warn """Circular dependency detected. Precompilation will be skipped for:\n $(join(string.(circular_deps), "\n "))""" end + # if a list of packages is given, restrict to dependencies of given packages if !isempty(pkgs) - # if a list of packages is given, restrict to dependencies of given packages function collect_all_deps(depsmap, dep, alldeps=Base.PkgId[]) append!(alldeps, depsmap[dep]) for _dep in depsmap[dep] @@ -1162,6 +1194,7 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: filter!(d->in(first(d), keep), depsmap) isempty(depsmap) && pkgerror("No direct dependencies found matching $(repr(pkgs))") end + target = string(isempty(pkgs) ? "project" : join(pkgs, ", "), "...") pkg_queue = Base.PkgId[] failed_deps = Dict{Base.PkgId, String}() @@ -1199,7 +1232,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: end end - t_print = @async begin # fancy print loop + ## fancy print loop + t_print = @async begin try wait(first_started) (isempty(pkg_queue) || interrupted_or_done.set) && return @@ -1234,7 +1268,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: final_loop || print(iostr, sprint(io -> show_progress(io, bar; termwidth = displaysize(ctx.io)[2]); context=io), "\n") for dep in pkg_queue_show loaded = warn_loaded && haskey(Base.loaded_modules, dep) - name = dep in direct_deps ? dep.name : string(color_string(dep.name, :light_black)) + _name = haskey(exts, dep) ? string(exts[dep], " → ", dep.name) : dep.name + name = dep in direct_deps ? _name : string(color_string(_name, :light_black)) if dep in precomperr_deps print(iostr, color_string(" ? ", Base.warn_color()), name, "\n") elseif haskey(failed_deps, dep) @@ -1274,7 +1309,8 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: end tasks = Task[] Base.LOADING_CACHE[] = Base.LoadingCache() - for (pkg, deps) in depsmap # precompilation loop + ## precompilation loop + for (pkg, deps) in depsmap paths = Base.find_all_in_cache_path(pkg) sourcepath = Base.locate_package(pkg) if sourcepath === nothing @@ -1307,9 +1343,10 @@ function precompile(ctx::Context, pkgs::Vector{String}=String[]; internal_call:: Base.acquire(parallel_limiter) is_direct_dep = pkg in direct_deps iob = IOBuffer() - name = is_direct_dep ? pkg.name : string(color_string(pkg.name, :light_black)) + _name = haskey(exts, pkg) ? string(exts[pkg], " → ", pkg.name) : pkg.name + name = is_direct_dep ? _name : string(color_string(_name, :light_black)) !fancyprint && lock(print_lock) do - isempty(pkg_queue) && printpkgstyle(io, :Precompiling, "environment...") + isempty(pkg_queue) && printpkgstyle(io, :Precompiling, target) end push!(pkg_queue, pkg) started[pkg] = true @@ -1609,13 +1646,14 @@ end @deprecate status(mode::PackageMode) status(mode=mode) -function status(ctx::Context, pkgs::Vector{PackageSpec}; diff::Bool=false, mode=PKGMODE_PROJECT, outdated::Bool=false, compat::Bool=false, io::IO=stdout_f(), kwargs...) +function status(ctx::Context, pkgs::Vector{PackageSpec}; diff::Bool=false, mode=PKGMODE_PROJECT, outdated::Bool=false, compat::Bool=false, extensions::Bool=false, io::IO=stdout_f()) if compat diff && pkgerror("Compat status has no `diff` mode") outdated && pkgerror("Compat status has no `outdated` mode") + extensions && pkgerror("Compat status has no `extensions` mode") Operations.print_compat(ctx, pkgs; io) else - Operations.status(ctx.env, ctx.registries, pkgs; mode, git_diff=diff, io, outdated) + Operations.status(ctx.env, ctx.registries, pkgs; mode, git_diff=diff, io, outdated, extensions) end return nothing end diff --git a/src/Operations.jl b/src/Operations.jl index 53178462d8..8bf0ba8211 100644 --- a/src/Operations.jl +++ b/src/Operations.jl @@ -148,6 +148,27 @@ function update_manifest!(env::EnvCache, pkgs::Vector{PackageSpec}, deps_map, ju record_project_hash(env) end +# This has to be done after the packages have been downloaded +# since we need access to the Project file to read the information +# about extensions +function fixup_ext!(env, pkgs) + for pkg in pkgs + v = joinpath(source_path(env.project_file, pkg), "Project.toml") + if haskey(env.manifest, pkg.uuid) + entry = env.manifest[pkg.uuid] + if isfile(v) + p = Types.read_project(v) + entry.weakdeps = p.weakdeps + entry.exts = p.exts + for (name, _) in p.weakdeps + if !haskey(p.deps, name) + delete!(entry.deps, name) + end + end + end + end + end +end #################### # Registry Loading # @@ -198,24 +219,29 @@ function reset_all_compat!(proj::Project) return nothing end -function collect_project!(pkg::PackageSpec, path::String, - deps_map::Dict{UUID,Vector{PackageSpec}}) - deps_map[pkg.uuid] = PackageSpec[] +function collect_project(pkg::PackageSpec, path::String) + deps = PackageSpec[] + weakdeps = Set{UUID}() project_file = projectfile_path(path; strict=true) if project_file === nothing pkgerror("could not find project file for package $(err_rep(pkg)) at `$path`") end project = read_package(project_file) - julia_compat = get_compat(project, "julia") #= # TODO, this should either error or be quiet + julia_compat = get_compat(project, "julia") if julia_compat !== nothing && !(VERSION in julia_compat) println(io, "julia version requirement for package $(err_rep(pkg)) not satisfied") end =# for (name, uuid) in project.deps vspec = get_compat(project, name) - push!(deps_map[pkg.uuid], PackageSpec(name, uuid, vspec)) + push!(deps, PackageSpec(name, uuid, vspec)) + end + for (name, uuid) in project.weakdeps + vspec = get_compat(project, name) + push!(deps, PackageSpec(name, uuid, vspec)) + push!(weakdeps, uuid) end if project.version !== nothing pkg.version = project.version @@ -223,7 +249,7 @@ function collect_project!(pkg::PackageSpec, path::String, # @warn("project file for $(pkg.name) is missing a `version` entry") pkg.version = VersionNumber(0) end - return + return deps, weakdeps end is_tracking_path(pkg) = pkg.path !== nothing @@ -258,9 +284,12 @@ end function collect_fixed!(env::EnvCache, pkgs::Vector{PackageSpec}, names::Dict{UUID, String}) deps_map = Dict{UUID,Vector{PackageSpec}}() + weak_map = Dict{UUID,Set{UUID}}() if env.pkg !== nothing pkg = env.pkg - collect_project!(pkg, dirname(env.project_file), deps_map) + deps, weakdeps = collect_project(pkg, dirname(env.project_file)) + deps_map[pkg.uuid] = deps + weak_map[pkg.uuid] = weakdeps names[pkg.uuid] = pkg.name end for pkg in pkgs @@ -268,7 +297,9 @@ function collect_fixed!(env::EnvCache, pkgs::Vector{PackageSpec}, names::Dict{UU if !isdir(path) pkgerror("expected package $(err_rep(pkg)) to exist at path `$path`") end - collect_project!(pkg, path, deps_map) + deps, weakdeps = collect_project(pkg, path) + deps_map[pkg.uuid] = deps + weak_map[pkg.uuid] = weakdeps end fixed = Dict{UUID,Resolve.Fixed}() @@ -285,7 +316,7 @@ function collect_fixed!(env::EnvCache, pkgs::Vector{PackageSpec}, names::Dict{UU idx = findfirst(pkg -> pkg.uuid == uuid, pkgs) fix_pkg = pkgs[idx] end - fixed[uuid] = Resolve.Fixed(fix_pkg.version, q) + fixed[uuid] = Resolve.Fixed(fix_pkg.version, q, weak_map[uuid]) end return fixed end @@ -407,6 +438,7 @@ function deps_graph(env::EnvCache, registries::Vector{Registry.RegistryInstance} # pkg -> version -> (dependency => compat): all_compat = Dict{UUID,Dict{VersionNumber,Dict{UUID,VersionSpec}}}() + weak_compat = Dict{UUID,Dict{VersionNumber,Set{UUID}}}() for (fp, fx) in fixed all_compat[fp] = Dict(fx.version => Dict{UUID,VersionSpec}()) @@ -418,7 +450,8 @@ function deps_graph(env::EnvCache, registries::Vector{Registry.RegistryInstance} for uuid in unseen push!(seen, uuid) uuid in keys(fixed) && continue - all_compat_u = get_or_make!(all_compat, uuid) + all_compat_u = get_or_make!(all_compat, uuid) + weak_compat_u = get_or_make!(weak_compat, uuid) uuid_is_stdlib = false stdlib_name = "" @@ -446,35 +479,56 @@ function deps_graph(env::EnvCache, registries::Vector{Registry.RegistryInstance} push!(uuids, other_uuid) all_compat_u_vr[other_uuid] = VersionSpec() end + + if !isempty(proj.weakdeps) + weak_all_compat_u_vr = get_or_make!(weak_compat_u, v) + for (_, other_uuid) in proj.weakdeps + push!(uuids, other_uuid) + all_compat_u_vr[other_uuid] = VersionSpec() + push!(weak_all_compat_u_vr, other_uuid) + end + end else for reg in registries pkg = get(reg, uuid, nothing) pkg === nothing && continue info = Registry.registry_info(pkg) - for (v, compat_info) in Registry.compat_info(info) - # Filter yanked and if we are in offline mode also downloaded packages - # TODO, pull this into a function - Registry.isyanked(info, v) && continue - if Pkg.OFFLINE_MODE[] - pkg_spec = PackageSpec(name=pkg.name, uuid=pkg.uuid, version=v, tree_hash=Registry.treehash(info, v)) - is_package_downloaded(env.project_file, pkg_spec) || continue - end - # Skip package version that are not the same as external packages in sysimage - if PKGORIGIN_HAVE_VERSION && RESPECT_SYSIMAGE_VERSIONS[] && julia_version == VERSION - pkgid = Base.PkgId(uuid, pkg.name) - if Base.in_sysimage(pkgid) - pkgorigin = get(Base.pkgorigins, pkgid, nothing) - if pkgorigin !== nothing && pkgorigin.version !== nothing - if v != pkgorigin.version - continue + function add_compat!(d, cinfo) + for (v, compat_info) in cinfo + # Filter yanked and if we are in offline mode also downloaded packages + # TODO, pull this into a function + Registry.isyanked(info, v) && continue + if Pkg.OFFLINE_MODE[] + pkg_spec = PackageSpec(name=pkg.name, uuid=pkg.uuid, version=v, tree_hash=Registry.treehash(info, v)) + is_package_downloaded(env.project_file, pkg_spec) || continue + end + + # Skip package version that are not the same as external packages in sysimage + if PKGORIGIN_HAVE_VERSION && RESPECT_SYSIMAGE_VERSIONS[] && julia_version == VERSION + pkgid = Base.PkgId(uuid, pkg.name) + if Base.in_sysimage(pkgid) + pkgorigin = get(Base.pkgorigins, pkgid, nothing) + if pkgorigin !== nothing && pkgorigin.version !== nothing + if v != pkgorigin.version + continue + end end end end + dv = get_or_make!(d, v) + merge!(dv, compat_info) + union!(uuids, keys(compat_info)) + end + end + add_compat!(all_compat_u, Registry.compat_info(info)) + weak_compat_info = Registry.weak_compat_info(info) + if weak_compat_info !== nothing + add_compat!(all_compat_u, weak_compat_info) + # Version to Set + for (v, compat_info) in weak_compat_info + weak_compat_u[v] = keys(compat_info) end - - all_compat_u[v] = compat_info - union!(uuids, keys(compat_info)) end end end @@ -493,7 +547,7 @@ function deps_graph(env::EnvCache, registries::Vector{Registry.RegistryInstance} end end - return Resolve.Graph(all_compat, uuid_to_name, reqs, fixed, false, julia_version), + return Resolve.Graph(all_compat, weak_compat, uuid_to_name, reqs, fixed, false, julia_version), all_compat end @@ -1127,7 +1181,7 @@ function rm(ctx::Context, pkgs::Vector{PackageSpec}; mode::PackageMode) # only declare `compat` for remaining direct or `extra` dependencies # `julia` is always an implicit direct dependency filter!(ctx.env.project.compat) do (name, _) - name == "julia" || name in keys(ctx.env.project.deps) || name in keys(ctx.env.project.extras) + name == "julia" || name in keys(ctx.env.project.deps) || name in keys(ctx.env.project.extras) || name in keys(ctx.env.project.weakdeps) end deps_names = union(keys(ctx.env.project.deps), keys(ctx.env.project.extras)) filter!(ctx.env.project.targets) do (target, deps) @@ -1272,6 +1326,7 @@ function add(ctx::Context, pkgs::Vector{PackageSpec}, new_git=Set{UUID}(); pkgs, deps_map = _resolve(ctx.io, ctx.env, ctx.registries, pkgs, preserve, ctx.julia_version) update_manifest!(ctx.env, pkgs, deps_map, ctx.julia_version) new_apply = download_source(ctx) + fixup_ext!(ctx.env, pkgs) # After downloading resolutionary packages, search for (Julia)Artifacts.toml files # and ensure they are all downloaded and unpacked as well: @@ -1294,6 +1349,7 @@ function develop(ctx::Context, pkgs::Vector{PackageSpec}, new_git::Set{UUID}; pkgs, deps_map = _resolve(ctx.io, ctx.env, ctx.registries, pkgs, preserve, ctx.julia_version) update_manifest!(ctx.env, pkgs, deps_map, ctx.julia_version) new_apply = download_source(ctx) + fixup_ext!(ctx.env, pkgs) download_artifacts(ctx.env; platform=platform, julia_version=ctx.julia_version, io=ctx.io) write_env(ctx.env) # write env before building show_update(ctx.env, ctx.registries; io=ctx.io) @@ -1421,6 +1477,7 @@ function up(ctx::Context, pkgs::Vector{PackageSpec}, level::UpgradeLevel; end update_manifest!(ctx.env, pkgs, deps_map, ctx.julia_version) new_apply = download_source(ctx) + fixup_ext!(ctx.env, pkgs) download_artifacts(ctx.env, julia_version=ctx.julia_version, io=ctx.io) write_env(ctx.env; skip_writing_project) # write env before building show_update(ctx.env, ctx.registries; io=ctx.io, hidden_upgrades_info = true) @@ -1461,8 +1518,8 @@ function pin(ctx::Context, pkgs::Vector{PackageSpec}) pkgs, deps_map = _resolve(ctx.io, ctx.env, ctx.registries, pkgs, PRESERVE_TIERED, ctx.julia_version) update_manifest!(ctx.env, pkgs, deps_map, ctx.julia_version) - new = download_source(ctx) + fixup_ext!(ctx.env, pkgs) download_artifacts(ctx.env; julia_version=ctx.julia_version, io=ctx.io) write_env(ctx.env) # write env before building show_update(ctx.env, ctx.registries; io=ctx.io) @@ -1504,6 +1561,7 @@ function free(ctx::Context, pkgs::Vector{PackageSpec}; err_if_free=true) update_manifest!(ctx.env, pkgs, deps_map, ctx.julia_version) new = download_source(ctx) + fixup_ext!(ctx.env, pkgs) download_artifacts(ctx.env, io=ctx.io) write_env(ctx.env) # write env before building show_update(ctx.env, ctx.registries; io=ctx.io) @@ -1735,9 +1793,13 @@ function gen_target_project(ctx::Context, pkg::PackageSpec, source_path::String, test_project.deps = source_env.project.deps # collect test dependencies for name in get(source_env.project.targets, target, String[]) - uuid = get(source_env.project.extras, name, nothing) + uuid = nothing + for list in [source_env.project.extras, source_env.project.weakdeps] + uuid = get(list, name, nothing) + uuid === nothing || break + end if uuid === nothing - pkgerror("`$name` declared as a `$target` dependency, but no such entry in `extras`") + pkgerror("`$name` declared as a `$target` dependency, but no such entry in `extras` or `weakdeps`") end test_project.deps[name] = uuid end @@ -2034,6 +2096,35 @@ function is_package_downloaded(project_file::String, pkg::PackageSpec; platform= return true end +function status_ext_info(pkg::PackageSpec, env::EnvCache) + manifest = env.manifest + manifest_info = get(manifest, pkg.uuid, nothing) + manifest_info === nothing && return nothing + weakdepses = manifest_info.weakdeps + exts = manifest_info.exts + if !isempty(weakdepses) && !isempty(exts) + v = ExtInfo[] + for (ext, extdeps) in exts + extdeps isa String && (extdeps = String[extdeps]) + ext_loaded = (Base.get_extension(Base.PkgId(pkg.uuid, pkg.name), Symbol(ext)) !== nothing) + # Check if deps are loaded + extdeps_info= Tuple{String, Bool}[] + for extdep in extdeps + uuid = weakdepses[extdep] + loaded = haskey(Base.loaded_modules, Base.PkgId(uuid, extdep)) + push!(extdeps_info, (extdep, loaded)) + end + push!(v, ExtInfo((ext, ext_loaded), extdeps_info)) + end + return v + end + return nothing +end + +struct ExtInfo + ext::Tuple{String, Bool} # name, loaded + weakdeps::Vector{Tuple{String, Bool}} # name, loaded +end struct PackageStatusData uuid::UUID old::Union{Nothing, PackageSpec} @@ -2043,10 +2134,11 @@ struct PackageStatusData heldback::Bool compat_data::Union{Nothing, Tuple{Vector{String}, VersionNumber, VersionNumber}} changed::Bool + extinfo::Union{Nothing, Vector{ExtInfo}} end function print_status(env::EnvCache, old_env::Union{Nothing,EnvCache}, registries::Vector{Registry.RegistryInstance}, header::Symbol, - uuids::Vector, names::Vector; manifest=true, diff=false, ignore_indent::Bool, outdated::Bool, io::IO, + uuids::Vector, names::Vector; manifest=true, diff=false, ignore_indent::Bool, outdated::Bool, extensions::Bool, io::IO, mode::PackageMode, hidden_upgrades_info::Bool, show_usagetips::Bool=true) not_installed_indicator = sprint((io, args) -> printstyled(io, args...; color=Base.error_color()), "→", context=io) upgradable_indicator = sprint((io, args) -> printstyled(io, args...; color=:green), "⌃", context=io) @@ -2090,6 +2182,7 @@ function print_status(env::EnvCache, old_env::Union{Nothing,EnvCache}, registrie latest_version = true # Outdated info cinfo = nothing + ext_info = nothing if !isnothing(new) && !is_stdlib(new.uuid) cinfo = status_compat_info(new, env, registries) if cinfo !== nothing @@ -2101,6 +2194,17 @@ function print_status(env::EnvCache, old_env::Union{Nothing,EnvCache}, registrie continue end + if !isnothing(new) && !is_stdlib(new.uuid) + ext_info = status_ext_info(new, env) + end + + if extensions && ext_info === nothing + continue + end + + + # TODO: Show extension deps for project as well? + pkg_downloaded = !is_instantiated(new) || is_package_downloaded(env.project_file, new) new_ver_avail = !latest_version && !Operations.is_tracking_repo(new) && !Operations.is_tracking_path(new) @@ -2116,7 +2220,8 @@ function print_status(env::EnvCache, old_env::Union{Nothing,EnvCache}, registrie no_packages_upgradable &= (!changed || !pkg_upgradable) no_visible_packages_heldback &= (!changed || !pkg_heldback) no_packages_heldback &= !pkg_heldback - push!(package_statuses, PackageStatusData(uuid, old, new, pkg_downloaded, pkg_upgradable, pkg_heldback, cinfo, changed)) + + push!(package_statuses, PackageStatusData(uuid, old, new, pkg_downloaded, pkg_upgradable, pkg_heldback, cinfo, changed, ext_info)) end for pkg in package_statuses @@ -2161,6 +2266,27 @@ function print_status(env::EnvCache, old_env::Union{Nothing,EnvCache}, registrie printstyled(io, pkg_str; color=Base.warn_color()) end end + + if extensions && !diff && pkg.extinfo !== nothing + println(io) + for (i, ext) in enumerate(pkg.extinfo) + sym = i == length(pkg.extinfo) ? '└' : '├' + function print_ext_entry(io, (name, installed)) + color = installed ? :light_green : :light_black + printstyled(io, name, ;color) + end + print(io, " ", sym, "─ ") + print_ext_entry(io, ext.ext) + + print(io, " [") + join(io,sprint.(print_ext_entry, ext.weakdeps; context=io), ", ") + print(io, "]") + if i != length(pkg.extinfo) + println(io) + end + end + end + println(io) end @@ -2215,7 +2341,7 @@ end function status(env::EnvCache, registries::Vector{Registry.RegistryInstance}, pkgs::Vector{PackageSpec}=PackageSpec[]; header=nothing, mode::PackageMode=PKGMODE_PROJECT, git_diff::Bool=false, env_diff=nothing, ignore_indent=true, - io::IO, outdated::Bool=false, hidden_upgrades_info::Bool=false, show_usagetips::Bool=true) + io::IO, outdated::Bool=false, extensions::Bool=false, hidden_upgrades_info::Bool=false, show_usagetips::Bool=true) io == Base.devnull && return # if a package, print header if header === nothing && env.pkg !== nothing @@ -2242,10 +2368,10 @@ function status(env::EnvCache, registries::Vector{Registry.RegistryInstance}, pk diff = old_env !== nothing header = something(header, diff ? :Diff : :Status) if mode == PKGMODE_PROJECT || mode == PKGMODE_COMBINED - print_status(env, old_env, registries, header, filter_uuids, filter_names; manifest=false, diff, ignore_indent, io, outdated, mode, hidden_upgrades_info, show_usagetips) + print_status(env, old_env, registries, header, filter_uuids, filter_names; manifest=false, diff, ignore_indent, io, outdated, extensions, mode, hidden_upgrades_info, show_usagetips) end if mode == PKGMODE_MANIFEST || mode == PKGMODE_COMBINED - print_status(env, old_env, registries, header, filter_uuids, filter_names; diff, ignore_indent, io, outdated, mode, hidden_upgrades_info, show_usagetips) + print_status(env, old_env, registries, header, filter_uuids, filter_names; diff, ignore_indent, io, outdated, extensions, mode, hidden_upgrades_info, show_usagetips) end if is_manifest_current(env) === false tip = show_usagetips ? " It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary." : "" diff --git a/src/Pkg.jl b/src/Pkg.jl index cf30482305..451154db06 100644 --- a/src/Pkg.jl +++ b/src/Pkg.jl @@ -419,7 +419,8 @@ from packages that are tracking a path. const resolve = API.resolve """ - Pkg.status([pkgs...]; outdated::Bool=false, mode::PackageMode=PKGMODE_PROJECT, diff::Bool=false, compat::Bool=false, io::IO=stdout) + Pkg.status([pkgs...]; outdated::Bool=false, mode::PackageMode=PKGMODE_PROJECT, diff::Bool=false, compat::Bool=false, extensions::Bool=false, io::IO=stdout) + Print out the status of the project/manifest. @@ -450,6 +451,9 @@ that are in the project (explicitly added). If `mode` is `PKGMODE_MANIFEST`, print status also about those in the manifest (recursive dependencies). If there are any packages listed as arguments, the output will be limited to those packages. +Setting `ext=true` will show dependencies with extensions and what extension dependencies +of those that are currently loaded. + Setting `diff=true` will, if the environment is in a git repository, limit the output to the difference as compared to the last git commit. @@ -735,9 +739,9 @@ end # Precompilation # ################## -function _auto_precompile(ctx::Types.Context; warn_loaded = true, already_instantiated = false) +function _auto_precompile(ctx::Types.Context, pkgs::Vector{String}=String[]; warn_loaded = true, already_instantiated = false) if Base.JLOptions().use_compiled_modules == 1 && get_bool_env("JULIA_PKG_PRECOMPILE_AUTO"; default="true") - Pkg.precompile(ctx; internal_call=true, warn_loaded = warn_loaded, already_instantiated = already_instantiated) + Pkg.precompile(ctx, pkgs; internal_call=true, warn_loaded = warn_loaded, already_instantiated = already_instantiated) end end diff --git a/src/REPLMode/command_declarations.jl b/src/REPLMode/command_declarations.jl index 1c29bd6437..06faf59055 100644 --- a/src/REPLMode/command_declarations.jl +++ b/src/REPLMode/command_declarations.jl @@ -376,6 +376,7 @@ PSA[:name => "status", PSA[:name => "diff", :short_name => "d", :api => :diff => true], PSA[:name => "outdated", :short_name => "o", :api => :outdated => true], PSA[:name => "compat", :short_name => "c", :api => :compat => true], + PSA[:name => "extensions", :short_name => "e", :api => :extensions => true], ], :completions => complete_installed_packages, :description => "summarize contents of and changes to environment", @@ -383,6 +384,8 @@ PSA[:name => "status", [st|status] [-d|--diff] [-o|--outdated] [pkgs...] [st|status] [-d|--diff] [-o|--outdated] [-p|--project] [pkgs...] [st|status] [-d|--diff] [-o|--outdated] [-m|--manifest] [pkgs...] + [st|status] [-d|--diff] [-g|--extensions] [-p|--project] [pkgs...] + [st|status] [-d|--diff] [-g|--extensions] [-m|--manifest] [pkgs...] [st|status] [-c|--compat] [pkgs...] Show the status of the current environment. Packages marked with `⌃` have new @@ -391,6 +394,9 @@ new versions available, but cannot be installed due to compatibility constraints. To see why use `pkg> status --outdated` which shows any packages that are not at their latest version and if any packages are holding them back. +Use `pkg> status --extensions` to show dependencies with extensions and what extension dependencies +of those that are currently loaded. + In `--project` mode (default), the status of the project file is summarized. In `--manifest` mode the output also includes the recursive dependencies of added packages given in the manifest. If there are any packages listed as arguments the output will be limited to those packages. diff --git a/src/Registry/registry_instance.jl b/src/Registry/registry_instance.jl index 8b59dec914..bf4d9fba1e 100644 --- a/src/Registry/registry_instance.jl +++ b/src/Registry/registry_instance.jl @@ -36,8 +36,9 @@ custom_isfile(in_memory_registry::Union{Dict, Nothing}, folder::AbstractString, git_tree_sha1::Base.SHA1 yanked::Bool @lazy uncompressed_compat::Union{Dict{UUID, VersionSpec}} + @lazy weak_uncompressed_compat::Union{Dict{UUID, VersionSpec}} end -VersionInfo(git_tree_sha1::Base.SHA1, yanked::Bool) = VersionInfo(git_tree_sha1, yanked, uninit) +VersionInfo(git_tree_sha1::Base.SHA1, yanked::Bool) = VersionInfo(git_tree_sha1, yanked, uninit, uninit) # This is the information that exists in e.g. General/A/ACME struct PkgInfo @@ -53,6 +54,12 @@ struct PkgInfo # Deps.toml deps::Dict{VersionRange, Dict{String, UUID}} + + # WeakCompat.toml + weak_compat::Dict{VersionRange, Dict{String, VersionSpec}} + + # WeakDeps.toml + weak_deps::Dict{VersionRange, Dict{String, UUID}} end isyanked(pkg::PkgInfo, v::VersionNumber) = pkg.version_info[v].yanked @@ -100,8 +107,8 @@ function initialize_uncompressed!(pkg::PkgInfo, versions = keys(pkg.version_info sort!(versions) - uncompressed_compat = uncompress(pkg.compat, versions) - uncompressed_deps = uncompress(pkg.deps, versions) + uncompressed_compat = uncompress(pkg.compat, versions) + uncompressed_deps = uncompress(pkg.deps, versions) for v in versions vinfo = pkg.version_info[v] @@ -119,11 +126,43 @@ function initialize_uncompressed!(pkg::PkgInfo, versions = keys(pkg.version_info return pkg end +function initialize_weak_uncompressed!(pkg::PkgInfo, versions = keys(pkg.version_info)) + # Only valid to call this with existing versions of the package + # Remove all versions we have already uncompressed + versions = filter!(v -> !isinit(pkg.version_info[v], :weak_uncompressed_compat), collect(versions)) + + sort!(versions) + + weak_uncompressed_compat = uncompress(pkg.weak_compat, versions) + weak_uncompressed_deps = uncompress(pkg.weak_deps, versions) + + for v in versions + vinfo = pkg.version_info[v] + weak_compat = Dict{UUID, VersionSpec}() + weak_uncompressed_deps_v = weak_uncompressed_deps[v] + weak_uncompressed_compat_v = weak_uncompressed_compat[v] + for (pkg, uuid) in weak_uncompressed_deps_v + vspec = get(weak_uncompressed_compat_v, pkg, nothing) + weak_compat[uuid] = vspec === nothing ? VersionSpec() : vspec + end + @init! vinfo.weak_uncompressed_compat = weak_compat + end + return pkg +end + function compat_info(pkg::PkgInfo) initialize_uncompressed!(pkg) return Dict(v => info.uncompressed_compat for (v, info) in pkg.version_info) end +function weak_compat_info(pkg::PkgInfo) + if isempty(pkg.weak_deps) + return nothing + end + initialize_weak_uncompressed!(pkg) + return Dict(v => info.weak_uncompressed_compat for (v, info) in pkg.version_info) +end + @lazy struct PkgEntry # Registry.toml: path::String @@ -181,7 +220,29 @@ function init_package_info!(pkg::PkgEntry) # All packages depend on julia deps[VersionRange()] = Dict("julia" => JULIA_UUID) - @init! pkg.info = PkgInfo(repo, subdir, version_info, compat, deps) + # WeakCompat.toml + weak_compat_data_toml = custom_isfile(pkg.in_memory_registry, pkg.registry_path, joinpath(pkg.path, "WeakCompat.toml")) ? + parsefile(pkg.in_memory_registry, pkg.registry_path, joinpath(pkg.path, "WeakCompat.toml")) : Dict{String, Any}() + weak_compat_data_toml = convert(Dict{String, Dict{String, Union{String, Vector{String}}}}, weak_compat_data_toml) + weak_compat = Dict{VersionRange, Dict{String, VersionSpec}}() + for (v, data) in weak_compat_data_toml + vr = VersionRange(v) + d = Dict{String, VersionSpec}(dep => VersionSpec(vr_dep) for (dep, vr_dep) in data) + weak_compat[vr] = d + end + + # WeakDeps.toml + weak_deps_data_toml = custom_isfile(pkg.in_memory_registry, pkg.registry_path, joinpath(pkg.path, "WeakDeps.toml")) ? + parsefile(pkg.in_memory_registry, pkg.registry_path, joinpath(pkg.path, "WeakDeps.toml")) : Dict{String, Any}() + weak_deps_data_toml = convert(Dict{String, Dict{String, String}}, weak_deps_data_toml) + weak_deps = Dict{VersionRange, Dict{String, UUID}}() + for (v, data) in weak_deps_data_toml + vr = VersionRange(v) + d = Dict{String, UUID}(dep => UUID(uuid) for (dep, uuid) in data) + weak_deps[vr] = d + end + + @init! pkg.info = PkgInfo(repo, subdir, version_info, compat, deps, weak_compat, weak_deps) return pkg.info end diff --git a/src/Resolve/graphtype.jl b/src/Resolve/graphtype.jl index ecd1d92669..efe5593048 100644 --- a/src/Resolve/graphtype.jl +++ b/src/Resolve/graphtype.jl @@ -236,13 +236,12 @@ mutable struct Graph function Graph( compat::Dict{UUID,Dict{VersionNumber,Dict{UUID,VersionSpec}}}, + compat_weak::Dict{UUID,Dict{VersionNumber,Set{UUID}}}, uuid_to_name::Dict{UUID,String}, reqs::Requires, fixed::Dict{UUID,Fixed}, verbose::Bool = false, julia_version::Union{VersionNumber,Nothing} = VERSION - ; - compat_weak::Dict{UUID,Dict{VersionNumber,Set{UUID}}} = Dict{UUID,Dict{VersionNumber,Set{UUID}}}(), ) # Tell the resolver about julia itself @@ -256,7 +255,6 @@ mutable struct Graph data = GraphData(compat, uuid_to_name, verbose) pkgs, np, spp, pdict, pvers, vdict, rlog = data.pkgs, data.np, data.spp, data.pdict, data.pvers, data.vdict, data.rlog - extended_deps = let spp = spp # Due to https://github.com/JuliaLang/julia/issues/15276 [Vector{Dict{Int,BitVector}}(undef, spp[p0]-1) for p0 = 1:np] end diff --git a/src/Types.jl b/src/Types.jl index 2146149399..e65dda0beb 100644 --- a/src/Types.jl +++ b/src/Types.jl @@ -239,6 +239,12 @@ Base.@kwdef mutable struct Project manifest::Union{String, Nothing} = nothing # Sections deps::Dict{String,UUID} = Dict{String,UUID}() + # deps that are also in weakdeps for backwards compat + # we do not store them in deps because we want to ignore them + # but for writing out the project file we need to remember them: + _deps_weak::Dict{String,UUID} = Dict{String,UUID}() + weakdeps::Dict{String,UUID} = Dict{String,UUID}() + exts::Dict{String,Union{Vector{String}, String}} = Dict{String,String}() extras::Dict{String,UUID} = Dict{String,UUID}() targets::Dict{String,Vector{String}} = Dict{String,Vector{String}}() compat::Dict{String,Compat} = Dict{String,Compat}() @@ -262,6 +268,8 @@ Base.@kwdef mutable struct PackageEntry repo::GitRepo = GitRepo() tree_hash::Union{Nothing,SHA1} = nothing deps::Dict{String,UUID} = Dict{String,UUID}() + weakdeps::Dict{String,UUID} = Dict{String,UUID}() + exts::Dict{String,Union{Vector{String}, String}} = Dict{String,String}() uuid::Union{Nothing, UUID} = nothing other::Union{Dict,Nothing} = nothing end @@ -272,9 +280,11 @@ Base.:(==)(t1::PackageEntry, t2::PackageEntry) = t1.name == t2.name && t1.repo == t2.repo && t1.tree_hash == t2.tree_hash && t1.deps == t2.deps && + t1.weakdeps == t2.weakdeps && + t1.exts == t2.exts && t1.uuid == t2.uuid # omits `other` -Base.hash(x::PackageEntry, h::UInt) = foldr(hash, [x.name, x.version, x.path, x.pinned, x.repo, x.tree_hash, x.deps, x.uuid], init=h) # omits `other` +Base.hash(x::PackageEntry, h::UInt) = foldr(hash, [x.name, x.version, x.path, x.pinned, x.repo, x.tree_hash, x.deps, x.weakdeps, x.exts, x.uuid], init=h) # omits `other` Base.@kwdef mutable struct Manifest julia_version::Union{Nothing,VersionNumber} = nothing # only set to VERSION when resolving diff --git a/src/manifest.jl b/src/manifest.jl index 69e20e25d3..ff25782126 100644 --- a/src/manifest.jl +++ b/src/manifest.jl @@ -83,19 +83,22 @@ struct Stage1 uuid::UUID entry::PackageEntry deps::Union{Vector{String}, Dict{String,UUID}} + weakdeps::Union{Vector{String}, Dict{String,UUID}} end -normalize_deps(name, uuid, deps, manifest) = deps -function normalize_deps(name, uuid, deps::Vector{String}, manifest::Dict{String,Vector{Stage1}}) +normalize_deps(name, uuid, deps, manifest; isext=false) = deps +function normalize_deps(name, uuid, deps::Vector{String}, manifest::Dict{String,Vector{Stage1}}; isext=false) if length(deps) != length(unique(deps)) pkgerror("Duplicate entry in `$name=$uuid`'s `deps` field.") end final = Dict{String,UUID}() for dep in deps infos = get(manifest, dep, nothing) - if infos === nothing - pkgerror("`$name=$uuid` depends on `$dep`, ", - "but no such entry exists in the manifest.") + if !isext + if infos === nothing + pkgerror("`$name=$uuid` depends on `$dep`, ", + "but no such entry exists in the manifest.") + end end # should have used dict format instead of vector format length(infos) == 1 || pkgerror("Invalid manifest format. ", @@ -110,21 +113,30 @@ function validate_manifest(julia_version::Union{Nothing,VersionNumber}, manifest for (name, infos) in stage1, info in infos info.entry.deps = normalize_deps(name, info.uuid, info.deps, stage1) end + for (name, infos) in stage1, info in infos + info.entry.weakdeps = normalize_deps(name, info.uuid, info.weakdeps, stage1; isext=true) + end # invariant: all dependencies are now normalized to Dict{String,UUID} deps = Dict{UUID, PackageEntry}() for (name, infos) in stage1, info in infos deps[info.uuid] = info.entry end # now just verify the graph structure - for (entry_uuid, entry) in deps, (name, uuid) in entry.deps - dep_entry = get(deps, uuid, nothing) - if dep_entry === nothing - pkgerror("`$(entry.name)=$(entry_uuid)` depends on `$name=$uuid`, ", - "but no such entry exists in the manifest.") - end - if dep_entry.name != name - pkgerror("`$(entry.name)=$(entry_uuid)` depends on `$name=$uuid`, ", - "but entry with UUID `$uuid` has name `$(dep_entry.name)`.") + for (entry_uuid, entry) in deps + for (deptype, isext) in [(entry.deps, false), (entry.weakdeps, true)] + for (name, uuid) in deptype + dep_entry = get(deps, uuid, nothing) + if !isext + if dep_entry === nothing + pkgerror("`$(entry.name)=$(entry_uuid)` depends on `$name=$uuid`, ", + "but no such entry exists in the manifest.") + end + if dep_entry.name != name + pkgerror("`$(entry.name)=$(entry_uuid)` depends on `$name=$uuid`, ", + "but entry with UUID `$uuid` has name `$(dep_entry.name)`.") + end + end + end end end return Manifest(; julia_version, manifest_format, deps, other) @@ -147,6 +159,7 @@ function Manifest(raw::Dict, f_or_io::Union{String, IO})::Manifest entry.name = name uuid = nothing deps = nothing + weakdeps = nothing try entry.pinned = read_pinned(get(info, "pinned", nothing)) uuid = read_field("uuid", nothing, info, safe_uuid)::UUID @@ -158,13 +171,15 @@ function Manifest(raw::Dict, f_or_io::Union{String, IO})::Manifest entry.tree_hash = read_field("git-tree-sha1", nothing, info, safe_SHA1) entry.uuid = uuid deps = read_deps(get(info::Dict, "deps", nothing)) + weakdeps = read_deps(get(info::Dict, "weakdeps", nothing)) + entry.exts = get(Dict{String, String}, info::Dict, "extensions") catch # TODO: Should probably not unconditionally log something @debug "Could not parse manifest entry for `$name`" f_or_io rethrow() end entry.other = info::Union{Dict,Nothing} - stage1[name] = push!(get(stage1, name, Stage1[]), Stage1(uuid, entry, deps)) + stage1[name] = push!(get(stage1, name, Stage1[]), Stage1(uuid, entry, deps, weakdeps)) end # by this point, all the fields of the `PackageEntry`s have been type casted # but we have *not* verified the _graph_ structure of the manifest @@ -257,18 +272,25 @@ function destructure(manifest::Manifest)::Dict entry!(new_entry, "repo-url", repo_source) entry!(new_entry, "repo-rev", entry.repo.rev) entry!(new_entry, "repo-subdir", entry.repo.subdir) - if isempty(entry.deps) - delete!(new_entry, "deps") - else - if all(dep -> unique_name[first(dep)], entry.deps) - new_entry["deps"] = sort(collect(keys(entry.deps))) + for (deptype, depname) in [(entry.deps, "deps"), (entry.weakdeps, "weakdeps")] + if isempty(deptype) + delete!(new_entry, depname) else - new_entry["deps"] = Dict{String,String}() - for (name, uuid) in entry.deps - new_entry["deps"][name] = string(uuid) + if all(dep -> haskey(unique_name, first(dep)), deptype) && all(dep -> unique_name[first(dep)], deptype) + new_entry[depname] = sort(collect(keys(deptype))) + else + new_entry[depname] = Dict{String,String}() + for (name, uuid) in deptype + new_entry[depname][name] = string(uuid) + end end end end + + # TODO: Write this inline + if !isempty(entry.exts) + entry!(new_entry, "extensions", entry.exts) + end if manifest.manifest_format.major == 1 push!(get!(raw, entry.name, Dict{String,Any}[]), new_entry) elseif manifest.manifest_format.major == 2 diff --git a/src/project.jl b/src/project.jl index dfb0de7210..4febc6417f 100644 --- a/src/project.jl +++ b/src/project.jl @@ -2,7 +2,7 @@ # UTILS # ######### listed_deps(project::Project) = - append!(collect(keys(project.deps)), collect(keys(project.extras))) + append!(collect(keys(project.deps)), collect(keys(project.extras)), collect(keys(project.weakdeps))) ########### # READING # @@ -73,21 +73,26 @@ end read_project_compat(raw, project::Project) = pkgerror("Expected `compat` section to be a key-value list") -function validate(project::Project) +function validate(project::Project; file=nothing) # deps + location_string = file === nothing ? "" : " at $(repr(file))." dep_uuids = collect(values(project.deps)) if length(dep_uuids) != length(unique(dep_uuids)) - pkgerror("Two different dependencies can not have the same uuid") + pkgerror("Two different dependencies can not have the same uuid" * location_string) + end + weak_dep_uuids = collect(values(project.weakdeps)) + if length(weak_dep_uuids) != length(unique(weak_dep_uuids)) + pkgerror("Two different weak dependencies can not have the same uuid" * location_string) end # extras extra_uuids = collect(values(project.extras)) if length(extra_uuids) != length(unique(extra_uuids)) - pkgerror("Two different `extra` dependencies can not have the same uuid") + pkgerror("Two different `extra` dependencies can not have the same uuid" * location_string) end - dep_names = keys(project.deps) # TODO decide what to do in when `add`ing a dep that is already in `extras` # also, reintroduce test files for this #= + dep_names = keys(project.deps) for (name, uuid) in project.extras name in dep_names && pkgerror("name `$name` is listed in both `deps` and `extras`") uuid in dep_uuids && pkgerror("uuid `$uuid` is listed in both `deps` and `extras`") @@ -100,18 +105,18 @@ function validate(project::Project) pkgerror("A dependency was named twice in target `$target`") end dep in listed || pkgerror(""" - Dependency `$dep` in target `$target` not listed in `deps` or `extras` section. - """) + Dependency `$dep` in target `$target` not listed in `deps`, `weakdeps` or `extras` section + """ * location_string) end # compat for (name, version) in project.compat name == "julia" && continue name in listed || - pkgerror("Compat `$name` not listed in `deps` or `extras` section.") + pkgerror("Compat `$name` not listed in `deps`, `weakdeps` or `extras` section" * location_string) end end -function Project(raw::Dict) +function Project(raw::Dict; file=nothing) project = Project() project.other = raw project.name = get(raw, "name", nothing)::Union{String, Nothing} @@ -119,10 +124,16 @@ function Project(raw::Dict) project.uuid = read_project_uuid(get(raw, "uuid", nothing)) project.version = read_project_version(get(raw, "version", nothing)) project.deps = read_project_deps(get(raw, "deps", nothing), "deps") + project.weakdeps = read_project_deps(get(raw, "weakdeps", nothing), "weakdeps") + project.exts = get(Dict{String, String}, raw, "extensions") project.extras = read_project_deps(get(raw, "extras", nothing), "extras") project.compat = read_project_compat(get(raw, "compat", nothing), project) project.targets = read_project_targets(get(raw, "targets", nothing), project) - validate(project) + + # Handle deps in both [deps] and [weakdeps] + project._deps_weak = Dict(intersect(project.deps, project.weakdeps)) + filter!(p->!haskey(project._deps_weak, p.first), project.deps) + validate(project; file) return project end @@ -139,7 +150,7 @@ function read_project(f_or_io::Union{String, IO}) end rethrow() end - return Project(raw) + return Project(raw; file= f_or_io isa IO ? nothing : f_or_io) end @@ -167,14 +178,16 @@ function destructure(project::Project)::Dict entry!("uuid", project.uuid) entry!("version", project.version) entry!("manifest", project.manifest) + merge!(project.deps, project._deps_weak) entry!("deps", project.deps) + entry!("weakdeps", project.weakdeps) entry!("extras", project.extras) entry!("compat", Dict(name => x.str for (name, x) in project.compat)) entry!("targets", project.targets) return raw end -_project_key_order = ["name", "uuid", "keywords", "license", "desc", "deps", "compat"] +_project_key_order = ["name", "uuid", "keywords", "license", "desc", "deps", "weakdeps", "extensions", "compat"] project_key_order(key::String) = something(findfirst(x -> x == key, _project_key_order), length(_project_key_order) + 1) diff --git a/test/extensions.jl b/test/extensions.jl new file mode 100644 index 0000000000..a6bb7c2e38 --- /dev/null +++ b/test/extensions.jl @@ -0,0 +1,38 @@ +using .Utils +using Test + +@testset "weak deps" begin + isolate(loaded_depot=true) do + Pkg.activate(; temp=true) + Pkg.develop(path=joinpath(@__DIR__, "test_packages", "ExtensionExamples", "HasExtensions.jl")) + Pkg.test("HasExtensions", julia_args=`--depwarn=no`) # OffsetArrays errors from depwarn + end + isolate(loaded_depot=true) do + Pkg.activate(; temp=true) + Pkg.develop(path=joinpath(@__DIR__, "test_packages", "ExtensionExamples", "HasDepWithExtensions.jl")) + Pkg.test("HasDepWithExtensions", julia_args=`--depwarn=no`) # OffsetArrays errors from depwarn + io = IOBuffer() + Pkg.status(; extensions=true, mode=Pkg.PKGMODE_MANIFEST, io) + # TODO: Test output when ext deps are loaded etc. + str = String(take!(io)) + @test contains(str, "└─ OffsetArraysExt [OffsetArrays]" ) + end + + isolate(loaded_depot=true) do + Pkg.activate(; temp=true) + Pkg.develop(path=joinpath(@__DIR__, "test_packages", "ExtensionExamples", "HasExtensions.jl")) + @test_throws Pkg.Resolve.ResolverError Pkg.add(; name = "OffsetArrays", version = "0.9.0") + end + + isolate(loaded_depot=false) do + depot = mktempdir(); empty!(DEPOT_PATH); push!(DEPOT_PATH, depot) + Pkg.activate(; temp=true) + Pkg.Registry.add(Pkg.RegistrySpec(path=joinpath(@__DIR__, "test_packages", "ExtensionExamples", "ExtensionRegistry"))) + Pkg.Registry.add("General") + Pkg.add("HasExtensions") + Pkg.test("HasExtensions", julia_args=`--depwarn=no`) # OffsetArrays errors from depwarn + Pkg.add("HasDepWithExtensions") + Pkg.test("HasDepWithExtensions", julia_args=`--depwarn=no`) # OffsetArrays errors from depwarn + @test_throws Pkg.Resolve.ResolverError Pkg.add(; name = "OffsetArrays", version = "0.9.0") + end +end diff --git a/test/resolve_utils.jl b/test/resolve_utils.jl index 8763949e6c..981cb638b7 100644 --- a/test/resolve_utils.jl +++ b/test/resolve_utils.jl @@ -81,7 +81,7 @@ function graph_from_data(deps_data) end end end - return Graph(all_compat, uuid_to_name, Requires(), fixed, VERBOSE; compat_weak=all_compat_w) + return Graph(all_compat, all_compat_w, uuid_to_name, Requires(), fixed, VERBOSE) end function reqs_from_data(reqs_data, graph::Graph) reqs = Dict{UUID,VersionSpec}() diff --git a/test/runtests.jl b/test/runtests.jl index 7e3fedc819..97eb47e58e 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -66,6 +66,7 @@ Logging.with_logger(hide_logs ? Logging.NullLogger() : Logging.current_logger()) "api.jl", "registry.jl", "subdir.jl", + "extensions.jl", "artifacts.jl", "binaryplatforms.jl", "platformengines.jl", diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Compat.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Compat.toml new file mode 100644 index 0000000000..f66053e93f --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Compat.toml @@ -0,0 +1,3 @@ +[0] +HasExtensions = "0.2" +OffsetArrays = "1.12.0-1" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Deps.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Deps.toml new file mode 100644 index 0000000000..7df787b6bf --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Deps.toml @@ -0,0 +1,3 @@ +[0] +HasExtensions = "4d3288b3-3afc-4bb6-85f3-489fffe514c8" +OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Package.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Package.toml new file mode 100644 index 0000000000..d9c99cf3f4 --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Package.toml @@ -0,0 +1,4 @@ +name = "HasDepWithExtensions" +uuid = "d4ef3d4a-8e22-4710-85d8-c6cf2eb9efca" +repo = "https://github.com/KristofferC/GluePkgExamples.jl.git" +subdir = "HasDepWithExtensions.jl" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Versions.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Versions.toml new file mode 100644 index 0000000000..c5e7b2e77b --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasDepWithExtensions/Versions.toml @@ -0,0 +1,2 @@ +["0.2.0"] +git-tree-sha1 = "cafe9d753be789ca618c7951a653f2927f0f7bcf" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Compat.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Compat.toml new file mode 100644 index 0000000000..dc5e9a5b8c --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Compat.toml @@ -0,0 +1,3 @@ +[0] +Example = "0.5" +OffsetArrays = "1.12.0-1" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Deps.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Deps.toml new file mode 100644 index 0000000000..c6be33e2fc --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Deps.toml @@ -0,0 +1,3 @@ +[0] +Example = "7876af07-990d-54b4-ab0e-23690620f79a" +OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Package.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Package.toml new file mode 100644 index 0000000000..a19681dcee --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Package.toml @@ -0,0 +1,4 @@ +name = "HasExtensions" +uuid = "4d3288b3-3afc-4bb6-85f3-489fffe514c8" +repo = "https://github.com/KristofferC/GluePkgExamples.jl.git" +subdir = "HasExtensions.jl" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Versions.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Versions.toml new file mode 100644 index 0000000000..da794c5f1f --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/Versions.toml @@ -0,0 +1,2 @@ +["0.2.0"] +git-tree-sha1 = "f72a3690d75877ad5b6b45048dd5d1aad9264f52" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/WeakCompat.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/WeakCompat.toml new file mode 100644 index 0000000000..af8c7eba01 --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/WeakCompat.toml @@ -0,0 +1,2 @@ +[0] +OffsetArrays = "1.12.0-1" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/WeakDeps.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/WeakDeps.toml new file mode 100644 index 0000000000..59ec79f6c8 --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/H/HasExtensions/WeakDeps.toml @@ -0,0 +1,2 @@ +[0] +OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" diff --git a/test/test_packages/ExtensionExamples/ExtensionRegistry/Registry.toml b/test/test_packages/ExtensionExamples/ExtensionRegistry/Registry.toml new file mode 100644 index 0000000000..1a4492c2a8 --- /dev/null +++ b/test/test_packages/ExtensionExamples/ExtensionRegistry/Registry.toml @@ -0,0 +1,7 @@ +name = "ExtensionRegistry" +uuid = "e84ee987-529c-45f3-abc6-f0c1e5b43514" +repo = "" + +[packages] +4d3288b3-3afc-4bb6-85f3-489fffe514c8 = { name = "HasExtensions", path = "H/HasExtensions" } +d4ef3d4a-8e22-4710-85d8-c6cf2eb9efca = { name = "HasDepWithExtensions", path = "H/HasDepWithExtensions" } diff --git a/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/Manifest.toml b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/Manifest.toml new file mode 100644 index 0000000000..e7908cbab3 --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/Manifest.toml @@ -0,0 +1,57 @@ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.10.0-DEV" +manifest_format = "2.0" +project_hash = "51e63007fdb8e0cae2b36f0e7b2acebf7b7432a5" + +[[deps.Adapt]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "195c5505521008abea5aee4f96930717958eac6f" +uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e" +version = "3.4.0" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.0.1+0" + +[[deps.Example]] +git-tree-sha1 = "46e44e869b4d90b96bd8ed1fdcf32244fddfb6cc" +uuid = "7876af07-990d-54b4-ab0e-23690620f79a" +version = "0.5.3" + +[[deps.HasExtensions]] +deps = ["Example"] +path = "../HasExtensions.jl" +uuid = "4d3288b3-3afc-4bb6-85f3-489fffe514c8" +version = "0.1.0" +weakdeps = ["OffsetArrays"] + + [deps.HasExtensions.extensions] + OffsetArraysExt = "OffsetArrays" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[[deps.OffsetArrays]] +deps = ["Adapt"] +git-tree-sha1 = "f71d8950b724e9ff6110fc948dff5a329f901d64" +uuid = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" +version = "1.12.8" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.21+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.2.0+0" diff --git a/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/Project.toml b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/Project.toml new file mode 100644 index 0000000000..0e84d99671 --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/Project.toml @@ -0,0 +1,17 @@ +name = "HasDepWithExtensions" +uuid = "d4ef3d4a-8e22-4710-85d8-c6cf2eb9efca" +version = "0.1.0" + +[deps] +HasExtensions = "4d3288b3-3afc-4bb6-85f3-489fffe514c8" +OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" + +[compat] +HasExtensions = "0.1" +OffsetArrays = "1.12" + +[extras] +Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[targets] +test = ["Test"] diff --git a/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/src/HasDepWithExtensions.jl b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/src/HasDepWithExtensions.jl new file mode 100644 index 0000000000..ecf1a92cbc --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/src/HasDepWithExtensions.jl @@ -0,0 +1,15 @@ +module HasDepWithExtensions + +using HasExtensions +using OffsetArrays: OffsetArray +# Loading OffsetArrays makes the extesion "OffsetArraysExt" to load + +function do_something() + # @info "First do something with the basic array support in B" + HasExtensions.foo(rand(Float64, 2)) + + # @info "Now do something with extended OffsetArray support in B" + HasExtensions.foo(OffsetArray(rand(Float64, 2), 0:1)) +end + +end # module diff --git a/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/test/runtests.jl b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/test/runtests.jl new file mode 100644 index 0000000000..3830320b01 --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasDepWithExtensions.jl/test/runtests.jl @@ -0,0 +1,5 @@ +using HasDepWithExtensions +using Test + +@test HasDepWithExtensions.HasExtensions.offsetarrays_loaded +HasDepWithExtensions.do_something() diff --git a/test/test_packages/ExtensionExamples/HasExtensions.jl/Manifest.toml b/test/test_packages/ExtensionExamples/HasExtensions.jl/Manifest.toml new file mode 100644 index 0000000000..f9461df371 --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasExtensions.jl/Manifest.toml @@ -0,0 +1,10 @@ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.10.0-DEV" +manifest_format = "2.0" +project_hash = "549efd0c32255f4b8717f50f49778e183bcc0e36" + +[[deps.Example]] +git-tree-sha1 = "46e44e869b4d90b96bd8ed1fdcf32244fddfb6cc" +uuid = "7876af07-990d-54b4-ab0e-23690620f79a" +version = "0.5.3" diff --git a/test/test_packages/ExtensionExamples/HasExtensions.jl/Project.toml b/test/test_packages/ExtensionExamples/HasExtensions.jl/Project.toml new file mode 100644 index 0000000000..0687e1fed8 --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasExtensions.jl/Project.toml @@ -0,0 +1,22 @@ +name = "HasExtensions" +uuid = "4d3288b3-3afc-4bb6-85f3-489fffe514c8" +version = "0.1.0" + +[deps] +Example = "7876af07-990d-54b4-ab0e-23690620f79a" + +[weakdeps] +OffsetArrays = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" + +[extensions] +OffsetArraysExt = "OffsetArrays" + +[compat] +Example = "0.5" +OffsetArrays = "1.12" + +[extras] +Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[targets] +test = ["OffsetArrays", "Test"] diff --git a/test/test_packages/ExtensionExamples/HasExtensions.jl/ext/OffsetArraysExt.jl b/test/test_packages/ExtensionExamples/HasExtensions.jl/ext/OffsetArraysExt.jl new file mode 100644 index 0000000000..12e130a4f7 --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasExtensions.jl/ext/OffsetArraysExt.jl @@ -0,0 +1,13 @@ +module OffsetArraysExt + +using HasExtensions, OffsetArrays + +function foo(::OffsetArray) + return 2 +end + +function __init__() + HasExtensions.offsetarrays_loaded = true +end + +end diff --git a/test/test_packages/ExtensionExamples/HasExtensions.jl/src/HasExtensions.jl b/test/test_packages/ExtensionExamples/HasExtensions.jl/src/HasExtensions.jl new file mode 100644 index 0000000000..4ab13b1327 --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasExtensions.jl/src/HasExtensions.jl @@ -0,0 +1,11 @@ +module HasExtensions + +using Example + +function foo(::AbstractArray) + return 1 +end + +offsetarrays_loaded = false + +end # module diff --git a/test/test_packages/ExtensionExamples/HasExtensions.jl/test/runtests.jl b/test/test_packages/ExtensionExamples/HasExtensions.jl/test/runtests.jl new file mode 100644 index 0000000000..f0d4793edb --- /dev/null +++ b/test/test_packages/ExtensionExamples/HasExtensions.jl/test/runtests.jl @@ -0,0 +1,8 @@ +using HasExtensions +using Test + +@test !HasExtensions.offsetarrays_loaded + +using OffsetArrays + +@test HasExtensions.offsetarrays_loaded