Downloads EASY_HOOK regression in v1.11.0? #56091

GhostOfElectric · 2024-10-10T16:03:34Z

I have this as my startup.jl file:

# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.

using Downloads
Downloads.DOWNLOADER[] = nothing
Downloads.EASY_HOOK[] = (easy, info) -> begin
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXY, "http://madeupproxy.com:8080")
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYUSERPWD, ":")
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYAUTH, Downloads.Curl.CURLAUTH_NTLM)
end

In 1.10.5 everything's fine:

julia> versioninfo()
Julia Version 1.10.5
Commit 6f3fdf7b36 (2024-08-27 14:19 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 4 × Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, cascadelake)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
  JULIA_DEPOT_PATH = E:\Program Files\JuliaDepot
  JULIA_HISTORY = U:\julialog\julia_repl.jl
  JULIA_NUM_THREADS = 4

(@v1.10) pkg> up
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.10\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.10\Manifest.toml`

In 1.11.0, it's not, trying to add anything that has an artifact will fail.

julia> versioninfo()
Julia Version 1.11.0
Commit 501a4f25c2 (2024-10-07 11:40 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 4 × Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, cascadelake)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
  JULIA_DEPOT_PATH = E:\Program Files\JuliaDepot
  JULIA_HISTORY = U:\julialog\julia_repl.jl
  JULIA_NUM_THREADS = 4

(@v1.11) pkg> up
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`

(@v1.11) pkg> add DuckDB
   Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
   Installed DuckDB_jll ─ v1.1.0+0
   Installed DuckDB ───── v1.1.0
     Failure artifact: DuckDB
     Failure artifact: DuckDB
ERROR: Unable to automatically download/install artifact 'DuckDB' from sources listed in 'E:\Program Files\JuliaDepot\packages\DuckDB_jll\IFrj8\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/96856948f83b372315f21905a5e7fec7c380b2ad
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/artifact/96856948f83b372315f21905a5e7fec7c380b2ad
- https://github.com/JuliaBinaryWrappers/DuckDB_jll.jl/releases/download/DuckDB-v1.1.0+0/DuckDB.v1.1.0.x86_64-w64-mingw32-cxx11.tar.gz
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://github.com/JuliaBinaryWrappers/DuckDB_jll.jl/releases/download/DuckDB-v1.1.0+0/DuckDB.v1.1.0.x86_64-w64-mingw32-cxx11.tar.gz

The text was updated successfully, but these errors were encountered:

KristofferC · 2024-10-10T16:33:40Z

Doing something like

Downloads.EASY_HOOK[] = (easy, info) -> begin
    println("RUNNING EASY HOOK!!!")
end

I get

(@v1.11) pkg> add Plots
RUNNING EASY HOOK!!!
    Updating registry at `~/.julia/registries/General.toml`
RUNNING EASY HOOK!!!
   Resolving package versions...
RUNNING EASY HOOK!!!
RUNNING EASY HOOK!!!

so the hook is at least running...

GhostOfElectric · 2024-10-10T16:44:54Z

Changed it to:

# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.

println("Defo read startup file!")
using Downloads
Downloads.DOWNLOADER[] = nothing
Downloads.EASY_HOOK[] = (easy, info) -> begin
    println("RUNNING EASY HOOK!!!")
end

PS E:\Program Files\julia-1.11.0\bin> ./julia
Defo read startup file!
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0 (2024-10-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> add Plots
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
   Resolving package versions...
     Failure artifact: x265
     Failure artifact: x265
ERROR: Unable to automatically download/install artifact 'x265' from sources listed in 'E:\Program Files\JuliaDepot\packages\x265_jll\9auB3\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/da2e0ac84abbe712490f8ee73aca3d512f1eaaab
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/artifact/da2e0ac84abbe712490f8ee73aca3d512f1eaaab
- https://github.com/JuliaBinaryWrappers/x265_jll.jl/releases/download/x265-v3.5.0+0/x265.v3.5.0.x86_64-w64-mingw32.tar.gz
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://github.com/JuliaBinaryWrappers/x265_jll.jl/releases/download/x265-v3.5.0+0/x265.v3.5.0.x86_64-w64-mingw32.tar.gz

giordano · 2024-10-10T21:24:42Z

Diff between Downloads.jl in v1.10.5 vs v1.11.0 is JuliaLang/Downloads.jl@ead289a...89d3c7d, it shouldn't be difficult to bisect (assuming the issue is there and not somewhere else).

KristofferC · 2024-10-10T21:37:34Z

Can you try load Pkg, set the hook through Pkg.Downloads and use the Pkg commands (not the REPL mode) to add the package?

GhostOfElectric · 2024-10-11T08:15:44Z

@KristofferC It works fine if done like that:

julia> using Pkg

julia> Downloads.DOWNLOADER[] = nothing

julia> Downloads.EASY_HOOK[] = (easy, info) -> begin
           Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXY, "http://madeupproxy:8080")
               Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYUSERPWD, ":")
           Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYAUTH, Downloads.Curl.CURLAUTH_NTLM)
           end
#3 (generic function with 1 method)

julia> Pkg.update()
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`

julia> Pkg.add("Plots")
   Resolving package versions...
  Downloaded artifact: x265
  Downloaded artifact: GR
  Downloaded artifact: LERC
  Downloaded artifact: XZ
  Downloaded artifact: fzf

In fact, it works fine, just going:

julia> using Pkg

julia> Pkg.update()
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`

It's just going via the REPL where it gets upset.

IanButterworth · 2024-10-17T15:04:21Z

So this seems to be a require_stdlib issue, but I've not been able to replicate it

With ~/.julia/config/startup.jl or etc/julia/startup.jl containing the above test code, the easy hook is run properly after a pkg repl switch (loading Pkg via require_stdlib)

julia % julia +1.11
Defo read startup file!
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.1 (2024-10-16)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> up
RUNNING EASY HOOK!!!
    Updating registry at `~/.julia/registries/General.toml`
...

Perhaps this is a windows-specific issue.

topolarity · 2024-10-17T15:11:59Z

That's because whether require_stdlib loads a duplicate copy of a package or not depends on your compile cache state

Often a stdlib ends up pre-compiled for an unrelated project/workflow, and then the pre-compiled stdlib starts to load as a duplicate version everywhere

You have to check via Base.loaded_modules_array() whether a duplicate copy was loaded, in which case I expect you'll hit this MWE even on Linux

topolarity · 2024-10-17T15:18:18Z

The premise of require_stdlib is that normal code does not interact with any of the side-loaded stdlibs (they operate in isolation), which this kind of "public global" is incompatible with

require_stdlib makes it illegal for stdlibs to have "public globals" like DOWNLOADER

IanButterworth · 2024-10-17T15:20:11Z

I very much appreciate you being enlightened about this. So removing use of "public globals" is the fix?

topolarity · 2024-10-17T15:20:31Z

One fix would be to move the globals in question into Base, where uniqueness is still a thing

topolarity · 2024-10-17T15:33:29Z

Yeah.. I'm not happy about the restrictions either TBH. I think they're a rough combo of unintuitive, hard to enforce, and lead to rare bugs (that they are rare is worse, not better IMO) but it's the situation we have right now

Moving the globals to Base should fix the issue

FWIW, I don't really know why we added this behavior to require_stdlib in the first place.

It might be worth revisiting?

IanButterworth · 2024-10-17T15:38:00Z

IIUC at least one of the reasons was to ensure that if the user wants to use a newer stdlib version than was bundled, say Pkg, we can still load the version that the repl expects, to avoid breakage.

It has been a bumpy road for sure..

topolarity · 2024-10-17T16:06:34Z

Oof. Moving these into Base is going to be non-trivial..

default_downloader! and DOWNLOADER[] both use types from Curl.jl

You can't update DOWNLOADER[] for two copies of Downloads.jl at once if they have different Downloader types...

KristofferC · 2024-10-17T16:35:56Z

For Pkg, if you update it, I would want the repl to be the updated one. We just have to fix the common load failures from out of date manifests.

IanButterworth · 2024-10-17T17:49:39Z

We should probably fork this discussion to a dedicated issue, or a call, but for completeness.
require_stdlib was added in #53326 which focused on fixing #53365

vtjnash · 2024-10-17T19:47:04Z

That's because whether require_stdlib loads a duplicate copy of a package or not depends on your compile cache state

This is fixed by #55908, which was marked for backporting to avoid this issue

mbauman changed the title ~~Downloads regression in v1.11.0?~~ Downloads EASY_HOOK regression in v1.11.0? Oct 10, 2024

GhostOfElectric mentioned this issue Oct 17, 2024

Added a further kwarg to PlatformEngines download function JuliaLang/Pkg.jl#4050

Open

vtjnash mentioned this issue Oct 17, 2024

REPLExt: no method matching repl_init(::REPL.LineEditREPL) when loading packages in REPL mode. #56216

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Downloads EASY_HOOK regression in v1.11.0? #56091

Downloads EASY_HOOK regression in v1.11.0? #56091

GhostOfElectric commented Oct 10, 2024

KristofferC commented Oct 10, 2024

GhostOfElectric commented Oct 10, 2024 •

edited

Loading

giordano commented Oct 10, 2024

KristofferC commented Oct 10, 2024

GhostOfElectric commented Oct 11, 2024 •

edited

Loading

IanButterworth commented Oct 17, 2024

topolarity commented Oct 17, 2024

topolarity commented Oct 17, 2024

IanButterworth commented Oct 17, 2024

topolarity commented Oct 17, 2024

topolarity commented Oct 17, 2024

IanButterworth commented Oct 17, 2024

topolarity commented Oct 17, 2024

KristofferC commented Oct 17, 2024

IanButterworth commented Oct 17, 2024

vtjnash commented Oct 17, 2024

Downloads EASY_HOOK regression in v1.11.0? #56091

Downloads EASY_HOOK regression in v1.11.0? #56091

Comments

GhostOfElectric commented Oct 10, 2024

KristofferC commented Oct 10, 2024

GhostOfElectric commented Oct 10, 2024 • edited Loading

giordano commented Oct 10, 2024

KristofferC commented Oct 10, 2024

GhostOfElectric commented Oct 11, 2024 • edited Loading

IanButterworth commented Oct 17, 2024

topolarity commented Oct 17, 2024

topolarity commented Oct 17, 2024

IanButterworth commented Oct 17, 2024

topolarity commented Oct 17, 2024

topolarity commented Oct 17, 2024

IanButterworth commented Oct 17, 2024

topolarity commented Oct 17, 2024

KristofferC commented Oct 17, 2024

IanButterworth commented Oct 17, 2024

vtjnash commented Oct 17, 2024

GhostOfElectric commented Oct 10, 2024 •

edited

Loading

GhostOfElectric commented Oct 11, 2024 •

edited

Loading