DEPOT_PATH expansion from JULIA_DEPOT_PATH behavior change between 1.10 and 1.11

[Moelf]

$ echo $JULIA_LOAD_PATH, $JULIA_DEPOT_PATH
:/opt/julia/environments/v1.11, :/opt/julia

$ julia +1.10 -e 'println(DEPOT_PATH)'
["/home/jiling/.julia", "/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/local/share/julia", "/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia", "/opt/julia"]

$ julia +1.11 -e 'println(DEPOT_PATH)'
["/home/jiling/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/local/share/julia", "/home/jiling/.julia/juliaup/julia-1.11.0+0.x64.linux.gnu/share/julia", "/opt/julia"]

## this is casuing the problem in production
$ /opt/julia-1.11.0/bin/julia -e 'println(DEPOT_PATH)'
["/opt/julia-1.11.0/local/share/julia", "/opt/julia-1.11.0/share/julia", "/opt/julia"

this causes problem because on the HPC system, Julia and IJulia are pre-installed and pre-compiled under /opt which is read-only.

this used to work on 1.10: https://github.com/ivukotic/ml_platform/blob/julia/Dockerfile

[KristofferC]

#51448 maybe

[Moelf]

looks like it yea. so is this not considered a breaking change? if so what's the workaround for me (i.e. how to add user path without knwoing user path ahead of time?)

[StefanKarpinski]

It does seem like a breaking change to me. Didn't really follow the logic of the change.

[Moelf]

can we add this to 1.11.1 discussion or something? pretty breaking for the HPC users I think, and not clear how to mitigate from my side (what to insert so it expands to user dir at run time)

[IanButterworth]

What it expects is for you to set JULIA_DEPOT_PATH=/opt/julia: Note the trailing :

[Moelf]

that does not work:

[jiling@jiling-notebook-1 ~]$ JULIA_DEPOT_PATH=/opt/julia: /opt/julia-1.11.0/bin/julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0 (2024-10-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

ERROR: IOError: mkdir("/opt/julia/logs"; mode=0o777): permission denied (EACCES)
Stacktrace:
 [1] uv_error
   @ ./libuv.jl:106 [inlined]
 [2] mkdir(path::String; mode::UInt16)
   @ Base.Filesystem ./file.jl:185
 [3] mkdir
   @ ./file.jl:177 [inlined]
 [4] mkpath(path::String; mode::UInt16)
   @ Base.Filesystem ./file.jl:241
 [5] mkpath
   @ ./file.jl:235 [inlined]
 [6] setup_interface(repl::REPL.LineEditREPL, hascolor::Bool, extra_repl_keymap::Any)
   @ REPL /opt/julia-1.11.0/share/julia/stdlib/v1.11/REPL/src/REPL.jl:1166
 [7] setup_interface
   @ /opt/julia-1.11.0/share/julia/stdlib/v1.11/REPL/src/REPL.jl:1043 [inlined]
 [8] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
   @ REPL /opt/julia-1.11.0/share/julia/stdlib/v1.11/REPL/src/REPL.jl:1450
 [9] (::REPL.var"#75#81"{REPL.LineEditREPL, REPL.REPLBackendRef})()
   @ REPL /opt/julia-1.11.0/share/julia/stdlib/v1.11/REPL/src/REPL.jl:461

[ Info: Disabling history file for this session
julia> DEPOT_PATH
3-element Vector{String}:
 "/opt/julia"
 "/opt/julia-1.11.0/local/share/julia"
 "/opt/julia-1.11.0/share/julia"

[IanButterworth]

Sorry, I misread the issue. To me this seems like an unintended effect of #51448 rather than an intentional change, and that a leading : was just not accounted/tested for. Seems fixable.

[Moelf]

I read the code and that PR, I think this is intentional, and I still think this is breaking.

        # otherwise, populate the depot path with the entries in JULIA_DEPOT_PATH,
        # expanding empty strings to the bundled depot

---

at least from reading the code, the workaround is JULIA_DEPOT_PATH="~/.julia:/opt/julia"

[IanButterworth]

@maleadt

[KristofferC]

The previous situation was quite bad and it caused you to have to recompile the full set of stdlibs used. I'd suggest fixing up the script over going back to that.

[IanButterworth]

Shouldn't it be

JULIA_DEPOT_PATH = /my/depot
DEPOT_PATH = ["/my/depot"]

JULIA_DEPOT_PATH = /my/depot:
DEPOT_PATH = ["/my/depot", "/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/local/share/julia", "/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia"]

JULIA_DEPOT_PATH = :/my/depot
DEPOT_PATH = ["/home/jiling/.julia", "/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/local/share/julia", "/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia", "/my/depot"]

The last of which is currently broken because it misses the default depot.

JULIA_DEPOT_PATH = :/my/depot
DEPOT_PATH = ["/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/local/share/julia", "/home/jiling/.julia/juliaup/julia-1.10.5+0.x64.linux.gnu/share/julia", "/my/depot"]

I don't think that was intentional?

[Moelf]

I think if the difference between /my/depot: and :/my/depot is more than just changing the order, it would be confusing to explain in the docs but I don't see another way, since apparently the current behavior is highly sought after.

Right now (after the breaking change) the docs explain this by saying "if the whole thing isn't empty, then any empty component will be expanded to the bundled paths".

If instead <empty>:/my/depot expands to [$HOME/.julia, bundled_paths..., /my/depot] but /my/depot:<empty> does not expand to [/my/depot, $HOME/.julia, bundled_paths...], we will have to do more explaining in the docs in a not so intuitive way. But maybe this is our only option?

[IanButterworth]

#56195

[maleadt]

Sorry, didn't have time to look at this yesterday. This looks like unfortunate/unintended breakage indeed, and I guess requires making the empty string expansion even more complicated...