Store stdlibs dict at precompile time #2916
Conversation
|
👎 stdlib is currently the ONLY location one could install packages systemwise. This will break that. |
|
@yuyichao could using a layered depot path satisfy that for you? E.g. define |
|
ENV would not work since it's not going to work reliably systemwise. |
In lab environments, I think it is pretty normal to require users to setup some kind of systemwide environment variables. If this is your own setup, putting it into
See if the following works for your purposes: #!/bin/sh
# Disable automatic precompilation to showcase that when we load
# a package from the "outer" depot, its precompiled file written to
# the "inner" depot.
export JULIA_PKG_PRECOMPILE_AUTO=0
# Cleanup any previous runs of this script
sudo rm -rf depot_outer depot_inner
# Install Bzip2_jll to the "outer" depot
echo "Installing to outer depot..."
JULIA_DEPOT_PATH="depot_outer" julia -e 'using Pkg; Pkg.add("Bzip2_jll")'
# Remove environments, as we don't want Julia to try and modify them
# I consider this something of a bug; Julia should not be loading the `Project.toml`
# file of other depots as read/write, even if it never writes to it.
rm -rf depot_outer/environments
# Remove registries, as we want users to manage their own registries
rm -rf depot_outer/registries
# Make outer depot no longer writable by us
sudo chown root:root -R depot_outer
# Install CodecBzip2_jll to the "inner" depot, note that Bzip2_jll is not installed again
echo "Installing to inner depot..."
JULIA_DEPOT_PATH="depot_inner:depot_outer" julia -e 'using Pkg; Pkg.add("CodecBzip2")'
# Prove that we can load CodecBzip2, which will load Bzip2_jll from the outer depot
JULIA_DEPOT_PATH="depot_inner:depot_outer" julia -e 'using CodecBzip2; @show Base.pathof(CodecBzip2); @show CodecBzip2.libbzip2'When installing packages, Julia only ever writes to the first element in |
It's neither. A system package usually don't and shouldn't require env to work by default. And
AFAICT, after the environment is deleted, the depot_outer stops working. |
|
i.e. |
|
What version of Julia are you using? |
|
|
|
And to be clear, your command did produce the result you were claiming, but with it importing Bzip2_jll still doesn't work. |
|
Ah, I see. I thought you meant my script was giving that error. Yes, this is also expected, because Bzip2_jll isn't directly added to the user (inner) depot. If you |
|
Well, the desired behavior, and the one achieved by placing packages in stdlib, is that when it's installed in there, user can just do |
In general, you can't just load stdlibs. e.g. If you're writing a package, you can't load an stdlib unless it's been manually added. Interactive usage allows it for convenience, but it's a shortcut. The stdlib folder is meant for packages that ship with Julia, and we're planning on changing more than just this about them in the future. If you want this kind of functionality, I really do think it's going to require something more like multiple depots, or modifying You can probably re-create what you want by adding |
Unless you are talking about removing that shortcut altogether for scripts and REPL, then it is still inconvenience and undesired behavior for systemwise installed packages.
That's exactly what I was saying that can work. A different paths that basically behaves exactly like stdlib now that's in LOAD_PATH by default. i.e. I don't want to patch julia in this regard during compilation.
No, startup files can be disabled as well, it is often disabled when only USER startup.jl is meant to be bypassed. This will make it harder/impossible without patching, to run some package tests that has all the dependencies already installed systemwise. |
|
You're asking for a way to ensure that Julia behaves differently than a stock Julia, without modifying Julia's source, but still persisting even when a "don't allow user customizations" option is passed. Having all three of those things is impossible. Either don't expect Julia to behave differently (e.g. only be able to load the stdlibs that it shipped with), modify Julia's source (alter the default I've given you multiple workarounds for this issue; if you don't want to take them, that's fine, but don't expect future versions of Julia to treat stdlibs in a way that is not what they're meant for. I can't say for certainty that this PR will be merged or that stdlibs will cease to function for your usecases, but your usecase is not what stdlibs were meant for, and it's not the direction that we're headed, so I'm trying to give you options for how to improve your workflow. |
No, I'm asking for a way that behaves exactly the same as a stock julia right now and has always been possible since version 0.3-0.4. OTOH, you are saying you want to break that because now you decided that it shouldn't be supported.
And that's exactly what you've been doing. Whenever someone bring up a usecase that you don't like you simply decide that it's not a valid usecase and simply break it, no matter how easy it is to support that usecase (in this case an additional default in LOAD_PATH). |
IanButterworth
added
precompile
|
@yuyichao I was just wondering whether anything has changed here regarding your usage of |
Part of addressing #2771
@KristofferC what would be a simple PackageCompiler test for this?