Cache external CodeInstances

[timholy]

Builds on #43793 and #43881. Tested each package with a task for which it was precompiled (I've devved most of these, see diffs that force precompilation here; ModelingToolkit is here). TTFT = Time To Finish Task

EDIT: latest benchmarks are in #43990 (comment)

Package	Load (nightly)	Load (PR)	Load update (PR)	TTFT (nightly)	TTFT (PR)
CSV	2.6	2.0		15.8	12.9
DataFrames	1.3	1.5		12.5	10.4
Plots	4.1	6.7	4.6	8.9	9.5
GLMakie	7.3	15.4	11.4	60.4	42.1
OrdinaryDiffEq	6.8	8.8		1.4	1.3
ModelingToolkit	11.8	17.1		36.7	20.3
Flux	7.5	8.7	8.2	24.9	15.7
JuMP	4.8	5.0		2.5	1.6
ImageFiltering	2.5	3.0		1.2	1.2
LV (see below)	2.5	2.9		8.6	0.2

"Load update" includes the changes in #43990 (comment). (If not reported, they seemed unchanged from the previous measurement.)

Looks like a clear win. (It's not obvious the TTFT for Plots is a significant difference, I've never replicated that gap.)

Closes #42016
Fixes #35972 and possibly the other issues listed in JuliaGraphics/Colors.jl#496 (I haven't checked yet).

[simeonschaub]

The regression for Plots.jl looks quite serious. Have you looked into that?

[baggepinnen]

Is the header wrong? The current header does not make me think it's a clear win, almost the opposite. Time to finish task presumably includes loading the package as well, and all but one package takes longer to load, some by quite a lot as well. I can imagine that many Julia sessions include loading several packages that aren't used, and an increase in load time might then hurt more than the saving in TTFT.

[timholy]

This hasshould have little or no functional consequence if you don't add precompilation to the package definition, and indeed you get to choose specifically what you precompile and what you don't. So, add whatever helps, and don't add what hurts. This shows you can shave 20-30% off the time for some packages, which is a lot for big packages like Flux & Makie. And just don't add precompilation to the ones where it does more harm than good.

That's a pretty clear win @baggepinnen, no?

[timholy]

The regression for Plots.jl looks quite serious. Have you looked into that?

Not yet. It's fresh off the bus, there might indeed be some stuff that can be made better.

[timholy]

Here's a really fun one: with a package

module LV

export filter2davx

using LoopVectorization

function filter2davx!(out::AbstractMatrix, A::AbstractMatrix, kern::AbstractMatrix)
    @turbo for J in CartesianIndices(out)
        tmp = zero(eltype(out))
        for I ∈ CartesianIndices(kern)
            tmp += A[I + J] * kern[I]
        end
        out[J] = tmp
    end
    out
end

function filter2davx(A::AbstractMatrix, kern::AbstractMatrix)
    out = similar(A, size(A) .- size(kern) .+ 1)
    return filter2davx!(out, A, kern)
end

# precompilation
A = rand(Float64, 512, 512)
kern = [0.1 0.3 0.1;
        0.3 0.5 0.3;
        0.1 0.3 0.1];
filter2davx(A, kern)

end

and task (this is what the other tasks look like)

@time using LV
A = rand(Float64, 512, 512)
kern = [0.1 0.3 0.1;
        0.3 0.5 0.3;
        0.1 0.3 0.1];
tstart = time(); filter2davx(A, kern); println(time() - tstart)  # avoid @time to prevent compilation of the task

we get for nightly

julia> include("tasks_LV.jl")
  2.463157 seconds (6.24 M allocations: 408.848 MiB, 3.33% gc time, 18.92% compilation time)
8.572684049606323

and for this PR

julia> include("tasks_LV.jl")
  2.932197 seconds (6.09 M allocations: 408.457 MiB, 1.98% gc time, 18.04% compilation time)
0.23031187057495117

CC @chriselrod, @ChrisRackauckas.

[timholy]

I dug into the Plots regression a bit. For the loading, it can almost entirely be explained by the time to load GR_jll (2.7s on this PR vs 0.15s on nightly), which in turn depends on dozens of other JLL libraries. There are 169 freshly-compiled external MethodInstances when precompiling GR_jll, and as a consequence there are typically ~6000 backedges that need reinserting (when formerly there were 0).

My guess is we can prevent the compilation by doing a little more precompilation in, e.g., Base.BinaryPlatforms when we build Julia---if those methods are already compiled into Julia itself, they won't need to be compiled when precompiling the packages. Yet to be tested in practice.

[martinholters]

Just want to add a datapoint where this PR does really well: If have a branch of ACME laying around where I've added some more precompiles and tweaked the code for better inferability in a few places. On master, that increases the load time from 1.6s to 1.9s and reduces the run-time of some model set-up code (that mostly involves compilation) from 25s to 17s. I was underwhelmed by that result and put this aside (although even that mediocre improvement would probably have been worth including in the main branch). Now with this PR, load times don't change much (a slight improvement, but within measurement noise), and the example task using ACME main is just improved to 23s. But then using the ACME branch with more precompiles, it goes down to 6.5s! And given that I had previously abandoned work, there are probably more precompiles that could be added for even further improvement. So 💯 to this PR from that perspective!

EDIT: Darn, I did @time f() for my measurement. Doing @time @eval f() instead, the improvement due to this PR is much smaller. But it's still there.

[giordano]

For the loading, it can almost entirely be explained by the time to load GR_jll (2.7s on this PR vs 0.15s on nightly), which in turn depends on dozens of other JLL libraries

That's not good, I presume the loading time of GTK3_jll exploded as well.

[timholy]

@giordano, I've finally got most of them stomped (locally), but the annoying bit is compilations like these:

...
structdiff(NamedTuple{(:validate_strict, :libc, :call_abi, :libgfortran_version, :cxxstring_abi, :libstdcxx_version, :os_version), Tuple{Bool, Nothing, Nothing, Nothing, String, Nothing, Nothing}}, Type{NamedTuple{(:validate_strict, :compare_strategies), T} where T<:Tuple}) from structdiff(NamedTuple{an, T} where T<:Tuple, Union{Type{NamedTuple{bn, T} where T<:Tuple}, NamedTuple{bn, T} where T<:Tuple}) where {an, bn}
  from module Base

pairs(NamedTuple{(:libc, :cxxstring_abi), Tuple{String, String}}) from pairs(NamedTuple{names, T} where T<:Tuple where names)
  from module Base.Iterators

_pairs_elt(Base.Pairs{Symbol, String, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:libc, :cxxstring_abi, :call_abi), Tuple{String, String, String}}}, Symbol) from _pairs_elt(Base.Pairs{K, V, I, A} where A where I, Any) where {K, V}
  from module Base.Iterators

Type##kw(NamedTuple{(:cxxstring_abi,), Tuple{String}}, Type{Base.BinaryPlatforms.Platform}, String, String) from Type##kw(Any, Type{Base.BinaryPlatforms.Platform}, String, String)
  from module Base.BinaryPlatforms
...

I've added a block at the end of binaryplatforms.jl containing:

Platform("x86_64", "linux"; validate_strict=true, libc="glibc")
for nt in (
    (libc="glibc", cxxstring_abi=""),
    (libc="glibc", cxxstring_abi="", call_abi=""),
    (                       libc=nothing, call_abi=nothing, libgfortran_version=nothing, cxxstring_abi=nothing, libstdcxx_version=nothing, os_version=nothing),
    (                       libc="glibc", call_abi=nothing, libgfortran_version=nothing, cxxstring_abi=nothing, libstdcxx_version=nothing, os_version=nothing),
    (                       libc="glibc", call_abi="",      libgfortran_version=nothing, cxxstring_abi=nothing, libstdcxx_version=nothing, os_version=nothing),
    (                       libc=nothing, call_abi=nothing, libgfortran_version=nothing, cxxstring_abi="cxx11", libstdcxx_version=nothing, os_version=nothing),
    (                       libc="glibc", call_abi="",      libgfortran_version=nothing, cxxstring_abi="cxx11", libstdcxx_version=nothing, os_version=nothing),
    (validate_strict=false, libc=nothing, call_abi=nothing, libgfortran_version=nothing, cxxstring_abi=nothing, libstdcxx_version=nothing, os_version=nothing),
    (validate_strict=false, libc="glibc", call_abi=nothing, libgfortran_version=nothing, cxxstring_abi=nothing, libstdcxx_version=nothing, os_version=nothing),
    (validate_strict=false, libc="glibc", call_abi="",      libgfortran_version=nothing, cxxstring_abi=nothing, libstdcxx_version=nothing, os_version=nothing),
    (validate_strict=false, libc=nothing, call_abi=nothing, libgfortran_version=nothing, cxxstring_abi="cxx11", libstdcxx_version=nothing, os_version=nothing),
    (validate_strict=false, libc="glibc", call_abi="",      libgfortran_version=nothing, cxxstring_abi="cxx11", libstdcxx_version=nothing, os_version=nothing),
)
    pairs(nt)
    iterate(pairs(nt))
    iterate(pairs(nt), haskey(nt, :libgfortran_version) ? (nt.libgfortran_version === nothing)+1 : 1)
    merge(nt, NamedTuple())
    merge(nt, Pair{Symbol,String}[])
    Base.structdiff(nt, (validate_strict=false, compare_strategies=Dict{String,Function}()))
    Platform("x86_64", "linux"; nt...)
end

but it's annoying and so far isn't complete. Can you think of a better approach?

[KristofferC]

Is there anyway that can be prevented from compiling at all? It seems unlikely to be useful to compile?

[timholy]

OK, with:

• Precompile to reduce latency JuliaPackaging/Preferences.jl#28
• Reduce compilation of Base methods (NFC) JuliaPackaging/JLLWrappers.jl#38
• b4a4b31

I've got the time to load GR_jll back down to about where it was before.

[timholy]

Is there anyway that can be prevented from compiling at all?

I guess if we reordered the file order in Base.jl we could turn off compilation in BinaryPlatforms?

[timholy]

That seems to have helped the gap in load times substantially (updated numbers posted above), although still not zero. I'm not sure we should expect it to be: if the packages are using precompilation, you're storing more stuff in the file, so loading might be expected to take longer. It's not really so simple since the load time is actually dominated by backedge insertion and other method-table perusals, but certainly it wouldn't be surprising for a load-time regression for some packages.

[timholy]

Ok, except for a backwards-compatible shim that nothing calls, I just got rid of kwargs in the constructor for Platform. I've left the various NamedTuple specializations commented out because they were a ton of work to discover, and I'd hate to lose them to an accidental rebase. I'll strip garbage before merging.

With this and JuliaPackaging/Preferences.jl#28 we're basically tied with master wrt the time to load GR_jll.

[timholy]

With the latest changes,

Package	Load (nightly)	Load (PR)	TTFT (nightly)	TTFT (PR)
Plots	3.0	3.5	10.9	10.8

so we're definitely at parity. The time to actually plot seems to vary a lot, and of course some packages might have been updated since I first started running this, but this comparison here uses the same Manifest file and were run moments apart from one another so the comparison between them is valid.

[timholy]

And now with MakieOrg/Makie.jl#1634 we get

Package	Load (nightly)	Load (PR)	TTFT (nightly)	TTFT (PR)
GLMakie	6.6	9.9	52.0	33.9

With those changes, the majority of Makie's inference time has been wiped out (~27s is not inference).

[timholy]

I noticed one problem & fixed it (see the commit just above this). Assuming CI passes, I think this is ready to go? But in contrast with the two predecessor PRs, I don't feel comfortable merging this without a review from someone who knows more than me.

[Krastanov]

This pull request reduced time-to-first-X for QuantumClifford from 11 seconds to 0.5 seconds. This is amazing! Thank you!

Hopefully this is the correct venue to share this and I am not causing noise.

[timholy]

A key point is that for me as a developer, the metric to optimize is not TTFT but (precompile time)+TTFT (ie the cost of restarting a julia session). Working to precompile things in DFTK looks like hard and confusing work that will make TTFT go down but precompile time go up

Understood, but do keep in mind that typically many more people use a package than develop it, so there's a bit of a tradeoff between developer convenience and user convenience.

You may be able to have both, though: it should be possible to set up a compile-time Preference that will control whether you precompile locally with generated code. E.g., you might be able to set a local environment variable JULIA_PRECOMPILE_DFTK=0 and check for that during DFTK build.

That said, there are some things Julia could do better here. We recompile all dependents whenever a package changes, and perhaps we could be more selective (kind of like linking a bunch of .o files in a C compiler). But that's actually a pretty hard problem, so don't expect it to be solved anytime soon.

[baggepinnen]

I also recently thought about the issue about cost of precompilation for developers. Couldn't you choose whether or not to run precompilation code based on if the package is being checked out for development? Something like the following pseudo code

if !is_developed(@__MODULE__)
    include("precompile.jl")
end

My initial experiments with this appeared to work well. My implementation of is_developed simply checked the path to the package.

const __CONTROLSYSTEMS_SOURCE_DIR__ = dirname(Base.source_path())
is_developed() = !(occursin(joinpath(".julia", "dev"), __CONTROLSYSTEMS_SOURCE_DIR__))

[timholy]

That's a pretty clever approach. I'd want to shield that being a flag so that I could easily check the consequences of precompilation even if I have it devved, but there's a lot to like about your suggestion.