Move DelimitedFiles to a separate external package

[ViralBShah]

Can we move DelimitedFiles to an external package and also perhaps remove it from the system image?

Now that we are considering moving sparse out of the system image, I thought I would ask that about DelimitedFiles as well. #44247 (comment)

Is Dates.jl worth considering moving out? Any others?

[DilumAluthge]

The sharpest constraint we have is that we cannot remove any stdlibs that are direct or indirect dependencies of Pkg. (Because otherwise, you cannot install any external packages.)

In my opinion, everything else seems like fair game.

The full list of direct and indirect dependencies of Pkg is: (click to expand)

1. ArgTools
2. Artifacts
3. Base64
4. Dates
5. Downloads
6. InteractiveUtils
7. LibCURL
8. LibCURL_jll
9. LibGit2
10. LibSSH2_jll
11. Libdl
12. Logging
13. Markdown
14. MbedTLS_jll
15. MozillaCACerts_jll
16. NetworkOptions
17. Pkg
18. Printf
19. REPL
20. Random
21. SHA
22. Serialization
23. Sockets
24. TOML
25. Tar
26. UUIDs
27. Unicode
28. Zlib_jll
29. nghttp2_jll
30. p7zip_jll

[ViralBShah]

Yikes, will we really move LinearAlgebra or Random out? Technically, they will just work like regular packages, but the annoyance level (due to type piracy issues) may be quite high.

There's also a two step process: (1) move things out into separate packages while they continue to be in the system image first, and then (2) in a future release, stop having them be stdlibs.

[DilumAluthge]

Yikes, will we really move LinearAlgebra or Random out? Technically, they will just work like regular packages, but the annoyance level (due to type piracy issues) may be quite high.

Fair enough 😂.

In addition to the direct and indirect dependencies of Pkg, I suppose that we should make a list (a very short list) of stdlibs that (1) would be very annoying to move out and (2) would cause a lot of user backlash if we did move them out.

[giordano]

17. Pkg

Pkg indirectly depends on itself?

[DilumAluthge]

33. Pkg

Pkg indirectly depends on itself?

Serves me right for auto-generating the list and not proofreading it 😂

[adkabo]

I checked the usage of a few Pkg.jl dependencies

• Unicode afaict isn't used
• Serialization is barely used and could easily be replaced with TOML
• Dates is barely used and could easily be replaced with Libc

I would be in favor of separating these. Perhaps some of the others could be removed if Pkg.jl were split into pieces, where the dependencies needed for installing other packages can be retained.

[LilithHafner]

I suppose that we should make a list (a very short list) of stdlibs that (1) would be very annoying to move out and (2) would cause a lot of user backlash if we did move them out.

How frequently packages are used should be a factor in (2) because of the small but nonzero cost of pkg> add (e.g. LinearAlgebra, Test)

[ViralBShah]

Note that should be a non-issue if using Project.toml or Manifest.toml, and in any case, there's the autodetection of uninstalled packages and Julia offering to install it for you.

[adkabo]

To quantify the installation cost of a small package:

% JULIA_LOAD_PATH=@:@stdlib julia --project=. --startup-file=no --quiet
julia> import Pkg

julia> @time Pkg.add("IterTools")
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/tmp/tmp.gmAhldmGp9/Project.toml`
  [c8e1da08] + IterTools v1.4.0
    Updating `/tmp/tmp.gmAhldmGp9/Manifest.toml`
  [c8e1da08] + IterTools v1.4.0
  3.437964 seconds (2.66 M allocations: 163.276 MiB, 1.66% gc time, 33.38% compilation time)

This will depend on internet speed etc. It could be worth profiling to find out where the time is spent.

[LilithHafner]

In my currently running session

(@v1.9) pkg> activate --temp
  Activating new project at `/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_ZsugUQ`

julia> import Pkg

julia> @time Pkg.add("IterTools")
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_ZsugUQ/Project.toml`
  [c8e1da08] + IterTools v1.4.0
    Updating `/private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_ZsugUQ/Manifest.toml`
  [c8e1da08] + IterTools v1.4.0
Precompiling project...
  1 dependency successfully precompiled in 2 seconds
 16.597010 seconds (6.17 M allocations: 382.854 MiB, 0.82% gc time, 67.02% compilation time)

My internet speed is 218.9 Mbps download 23.2 Mbps upload 18 ms latency to google.com.

[JeffBezanson]

I say yes, let's move DelimitedFiles out of the stdlib and out of the system image (unless of course there is some kind of unforeseen breakage). The rest need to be taken on a case-by-case basis.

[ViralBShah]

Recipe for moving thing out while retaining history: JuliaAI/MLJOpenML.jl#1

[ViralBShah]

Making this issue be solely about moving DelimitedFiles.jl out of stdlib. Steps as recommended by @KristofferC:

• Move it out of the Julia repo - https://github.com/julialang/DelimitedFiles.jl
• Move it out of the sysimage
• Make sure it is registered in the General registry
• Delete it from Julia (but check how Pkg behaves before it)

In this period, we should avoid making any changes to this stdlib.

@KristofferC As I post your steps, I wonder if we should remove it from julia as step 2 (like the other stdlibs that live in external repos), and once that is stabilized, we do the registration and subsequent removal from system image.

[brenhinkeller]

Why?

[ViralBShah]

#45121 moves it to an external package and out of the sysimage. However, it is still an stdlib at the moment (and that will be addressed by #45540).

[A-childs-encyclopedia]

Why is it being removed?

[StefanKarpinski]

There's no reason for it to be built into every Julia binary. It makes the binary larger, slower to load, causes there to be more precompiled code to be potentially invalidated when other packages are loaded. It also makes it harder than necessary to develop and improve the DelimitedFiles package.