RFC: Per-project sysimage

[tkf]

With this patch, you can do

(@v1.x) pkg> activate .

julia> Base.set_sysimage_path("PATH/TO/sys.so")

julia> exit()

$ julia --project=.  # implicitly use -J=PATH/TO/sys.so

To use julia with project-specific system image.

Implementation/design

Given --project=$projectdir, julia binary tries to read the text file $projectdir/.julia/sysimages/$slug.path and use its content as the default --sysimage argument. The $slug is computed from the path of julia binary. This way, it is safe to use multiple julia binaries with the same project.

(Edit: it was pointed out that including VERSION in slug as well might be a better idea JuliaLang/Pkg.jl#2008 (comment))

This PR does not store the system image directly at $projectdir/.julia/sysimages/$slug.$dlext because the system image is rather large and it is nice to be able to use the same system image in multiple projects without copying the image file itself. Furthermore, it may make sense to distribute pre-compiled system image using the artifacts mechanism. It then is desirable to load system image directly from artifacts datastore rather than in $projectdir/.julia/sysimages. Note that symbolic link is not a good option for supporting Windows.

@staticfloat suggested to use triplet instead of $slug #33973 (comment) but this means to re-implement triplet detection in C. Furthermore, I don't think it is enough for supporting other variations of julia (e.g., debug build, different versions). Hashing the path of julia binary seems to be a simple robust solution.

---

If the design is good, I can add some tests. Also, since I'm not a C programmer, I may be doing something stupid. Let me know if there is a better way to implement this.

[fredrikekre]

Shouldn't this be recorded in the project file?

[tkf]

I don't think it's a good idea to add it to project or manifest file because these files have to be portable. Whether or not you have a particular system image at particular path is machine-dependent. Also, it'd be a problem if you use the same project with multiple julia versions.

I'd like to treat system image as an optimization. Project environment should work with and without the system image. (Of course, there is a gray zone because the package included in the image have to be consistent with what is in the manifest. I think it'd be better to handle it in a more generic mechanism like #32906.)

[Roger-luo]

I'm wondering how this should interact with binaries? should the system image binary also be shipped with the package manager instead of building them locally?

[tkf]

should the system image binary also be shipped with the package manager instead of building them locally?

I'm designing the API to make it easier (via, e.g., artifacts; see the OP) but I think making it actually work is challenging, as not all packages are relocatable.

[davidanthoff]

The existing implementation in VS Code for this kind of functionality compares the last modified date of the Manifest.toml with the sysimage, to make sure users don't end up loading a sysimage file that is out-of-date with their project definition. I think that is a useful and user friendly feature, does that somehow fit in here?

[tkf]

I think it'd be better to solve it with a more generic mechanism like #32906. Checking the modified time of Manifest.toml is not enough anyway since you can stack environments. It's also trivial for a frontend (e.g., VS code) to emit a warning if the sysimage and Manifest.toml are incompatible in some sense.

This design is about maximal flexibility and extensibility so that, e.g., you can distribute sysimage with the artifact system.

[oschulz]

Checking the modified time of Manifest.toml is not enough anyway since you can stack environments.

Would it be possible to use a hash of the content of the whole environment stack, or something similar?

[oschulz]

Or maybe better a hash of the part of the environment stack that forms the transitive dependencies of the packages included in the sysimage?

[tkf]

Frontends like VS code can implement such fancy checks on top of this PR. This PR is about maximal flexibility and composability. So, the checks like them are completely orthogonal to this PR. Also, note that everything has to be implemented in C (or C++ or scheme) because we can't use Julia before loading the system image [^1]. So, that's why I'm shooting for a very simple and flexible interface in this PR.

[^1] OK, technically, we can do this in a subprocess or maybe even re-initializing Julia runtime is possible? But it sounds very tricky and fragile to me.

[vchuravy]

As an aside this could allow for upgradable standard libaries, since those are baked into the system image. With a per-project dependent sysimage one could feasible update a standard library. We would need to give stdlibs independent version numbers, but that seems like the smallest issue.

[brenhinkeller]

Would fix #36840 I think

[oschulz]

Would fix #36840 I think

Yes, also @tkf's approach here is much more refined than my original suggestion in #36840 .

Closes #36840 .

[vchuravy]

I think pkgimages has mostly superseded this.