start docs section about loading/precomp/ttfx time

doc/src/manual/performance-tips.md

@@ -1394,6 +1394,105 @@ Prominent examples include [MKL.jl](https://github.com/JuliaLinearAlgebra/MKL.jl
 These are external packages, so we will not discuss them in detail here.
 Please refer to their respective documentations (especially because they have different behaviors than OpenBLAS with respect to multithreading).
 
+## Execution latency, loading and precompiling time
+
+### Reducing time to first plot etc.
+
+The first time a julia function is called it will be compiled. The [`@time`](@ref) macro family can illustrate this.
+
+```
+julia> foo() = rand(2,2) * rand(2,2)
+foo (generic function with 1 method)
+
+julia> @time @eval foo();
+  0.252395 seconds (1.12 M allocations: 56.178 MiB, 2.93% gc time, 98.12% compilation time)
+
+julia> @time @eval foo();
+  0.000156 seconds (63 allocations: 2.453 KiB)
+```
+
+Note that `@time @eval ...` form is explained in the [`@time`](@ref) documentation.
+
+When developing a package, it can be helpful to cache this compilation in the package caches that are "precompiled"
+during the `Pkg` installation process or before package load. To precompile package code effectively, it's recommended
+to use [`PrecompileTools.jl`](https://julialang.github.io/PrecompileTools.jl/stable/) to run a "precompile workload" during
+precompilation time, which will cache the native compiled code into the package `pkgimage` cache, greatly reducing
+"time to first ..." (often referred to as TTFX).
+
+
+### Reducing package loading time
+
+When developing a package keeping the time taken to load the package down is usually helpful.
+General good practice includes:
+
+1. Balancing the need to depending on packages for functionality vs. their load times, including stdlibs which on newer
+   julia versions may not be loaded already.
+2. Avoiding adding `__init__` functions to package modules unless unavoidable, especially those which might trigger a
+   lot of compilation, or just take a long time to execute.
+3. Watch out for invalidations in the dependency tree during load that could be causing significant recompilation.
+
+The `InteractiveUtils` tool [`@time_imports`](@ref) can be useful in the REPL to review the above factors.
+
+```julia-repl
+julia> @time @time_imports using Plots
+      0.5 ms  Printf
+     16.4 ms  Dates
+      0.7 ms  Statistics
+               ┌ 23.8 ms SuiteSparse_jll.__init__() 86.11% compilation time (100% recompilation)
+     90.1 ms  SuiteSparse_jll 91.57% compilation time (82% recompilation)
+      0.9 ms  Serialization
+               ┌ 39.8 ms SparseArrays.CHOLMOD.__init__() 99.47% compilation time (100% recompilation)
+    166.9 ms  SparseArrays 23.74% compilation time (100% recompilation)
+      0.4 ms  SparseArraysExt
+      0.5 ms  TOML
+      8.0 ms  Preferences
+      0.3 ms  PrecompileTools
+      0.2 ms  Reexport
+... many deps omitted for example ...
+      1.4 ms  Tar
+               ┌ 73.8 ms p7zip_jll.__init__() 99.93% compilation time (100% recompilation)
+     79.4 ms  p7zip_jll 92.91% compilation time (100% recompilation)
+               ┌ 27.7 ms GR.GRPreferences.__init__() 99.77% compilation time (100% recompilation)
+     43.0 ms  GR 64.26% compilation time (100% recompilation)
+               ┌ 2.1 ms Plots.__init__() 91.80% compilation time (100% recompilation)
+    300.9 ms  Plots 0.65% compilation time (100% recompilation)
+  1.795602 seconds (3.33 M allocations: 190.153 MiB, 7.91% gc time, 39.45% compilation time: 97% of which was recompilation)
+
+```
+
+Notice that in this example there are multiple packages loaded, some with `__init__()` functions, some of which cause
+compilation of which some is recompilation.
+
+This report gives a good opportunity to review whether the cost of dep load time is worth it for the functionality it brings.
+Also the `Pkg` utility `why` can be used to report why a an indirect dependency exists.
+
+```
+(CustomPackage) pkg> why FFMPEG_jll
+  Plots → FFMPEG → FFMPEG_jll
+  Plots → GR → GR_jll → FFMPEG_jll
+```
+
+or to see the indirect dependencies that a package brings in, you can `pkg> rm` the package, see the deps that are removed
+from the manifest, then revert the change with `pkg> undo`.
+
+If loading time is dominated by slow `__init__()` methods having compilation, one verbose way to identify what is being
+compiled is to use the julia args `--trace-compile=stderr --trace-compile-timing` which will report a [`precompile`](@ref)
+statement each time a method is compiled, along with how long compilation took. For instance, the full detail would be:
+
+```
+$ julia --trace-compile=stderr --trace-compile-timing
+julia> @time @time_imports using CustomPackage
+...
+```
+
+More analysis of the reasons for recompilation can be achieved with the
+[`SnoopCompile`](https://github.com/timholy/SnoopCompile.jl) package.
+
+### Reducing precompilation time
+
+TODO:
+Something about `Base.PRECOMPILE_TRACE_COMPILE[] = "stderr"`
+
 
 ## Miscellaneous