Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build of Julia 1.8.2 has broken #48698

Closed
kpamnany opened this issue Feb 16, 2023 · 16 comments
Closed

Build of Julia 1.8.2 has broken #48698

kpamnany opened this issue Feb 16, 2023 · 16 comments
Labels
building Build system, or building Julia or its dependencies

Comments

@kpamnany
Copy link
Contributor

$ git clone https://github.com/JuliaLang/julia
$ cd julia
$ git checkout v1.8.2
$ make -j

Results in:

  ERROR: sha512 checksum failure on Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2.tar.gz, should be:
      27e6f153f119638c4ed8e29127db10c1aff4fe5c14217a86a65d2bcb7ffbd3ed
      8e22613ed26fe0b9ffbb525ba00fc673be989d9da50e10fa12fd9a460ceeddcf
  But `sha512sum /home/kpamnany/tmp/julia/stdlib/srccache/Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2.tar.gz | awk '{ print $1; }'` results in:
      c22dc7c5aa41886a4abc683b5d2be04538ff043cc81e3f5ca482fed0dd160cdf
      ebe7c3d3366b324dc92678d69ce069a92a8ce38d396a7be6fc6bf10c1abfa67e
  This can happen due to bad downloads or network proxies, please check your
  network proxy/firewall settings and delete
  /home/kpamnany/tmp/julia/stdlib/srccache/Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2.tar.gz
  to force a redownload when you are ready

So, Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2.tar.gz has changed somehow where it's being downloaded (?) but deps/checksums/Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2.tar.gz/sha512 has not.

@JeffBezanson JeffBezanson added the building Build system, or building Julia or its dependencies label Feb 16, 2023
@giordano
Copy link
Contributor

I wonder if this is somehow related to #48466, since we download an automatically generated archive:

STATISTICS_TAR_URL = https://api.github.com/repos/JuliaStats/Statistics.jl/tarball/$1
But the breakage of the tarballs was reverted for the moment.

@benlorenz
Copy link
Contributor

benlorenz commented Feb 17, 2023

A stable tarball would have helped but in this case I think it is not the gzip algorithm but the fact that this repo changed organization and the tarball seems to reflect this:

$ diffoscope Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2-srccache.tar.gz Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2-github.tar.gz 
--- Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2-srccache.tar.gz
+++ Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2-github.tar.gz
│   --- Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2-srccache.tar
├── +++ Statistics-61a021bcb330e6c52f2435f2abaffc77875ab6f2-github.tar
│ ├── file list
│ │ @@ -1,18 +1,18 @@
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/.ci/
│ │ --rw-rw-r--   0 root         (0) root         (0)      780 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/.ci/test_and_change_uuid.jl
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/.github/
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/.github/workflows/
│ │ --rw-rw-r--   0 root         (0) root         (0)     1941 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/.github/workflows/ci.yml
│ │ --rw-rw-r--   0 root         (0) root         (0)     1300 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/LICENSE.md
│ │ --rw-rw-r--   0 root         (0) root         (0)      324 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/Project.toml
│ │ --rw-rw-r--   0 root         (0) root         (0)     1164 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/README.md
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/docs/
│ │ --rw-rw-r--   0 root         (0) root         (0)       89 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/docs/Project.toml
│ │ --rw-rw-r--   0 root         (0) root         (0)      365 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/docs/make.jl
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/docs/src/
│ │ --rw-rw-r--   0 root         (0) root         (0)      186 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/docs/src/index.md
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/src/
│ │ --rw-rw-r--   0 root         (0) root         (0)    37648 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/src/Statistics.jl
│ │ -drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/test/
│ │ --rw-rw-r--   0 root         (0) root         (0)    38547 2022-02-07 08:43:01.000000 JuliaLang-Statistics.jl-61a021b/test/runtests.jl
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/.ci/
│ │ +-rw-rw-r--   0 root         (0) root         (0)      780 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/.ci/test_and_change_uuid.jl
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/.github/
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/.github/workflows/
│ │ +-rw-rw-r--   0 root         (0) root         (0)     1941 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/.github/workflows/ci.yml
│ │ +-rw-rw-r--   0 root         (0) root         (0)     1300 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/LICENSE.md
│ │ +-rw-rw-r--   0 root         (0) root         (0)      324 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/Project.toml
│ │ +-rw-rw-r--   0 root         (0) root         (0)     1164 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/README.md
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/docs/
│ │ +-rw-rw-r--   0 root         (0) root         (0)       89 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/docs/Project.toml
│ │ +-rw-rw-r--   0 root         (0) root         (0)      365 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/docs/make.jl
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/docs/src/
│ │ +-rw-rw-r--   0 root         (0) root         (0)      186 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/docs/src/index.md
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/src/
│ │ +-rw-rw-r--   0 root         (0) root         (0)    37648 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/src/Statistics.jl
│ │ +drwxrwxr-x   0 root         (0) root         (0)        0 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/test/
│ │ +-rw-rw-r--   0 root         (0) root         (0)    38547 2022-02-07 08:43:01.000000 JuliaStats-Statistics.jl-61a021b/test/runtests.jl
│ │   --- JuliaLang-Statistics.jl-61a021b/.ci/test_and_change_uuid.jl
│ ├── +++ JuliaStats-Statistics.jl-61a021b/.ci/test_and_change_uuid.jl
│ │┄ Files identical despite different names
│ │   --- JuliaLang-Statistics.jl-61a021b/.github/workflows/ci.yml
│ ├── +++ JuliaStats-Statistics.jl-61a021b/.github/workflows/ci.yml
│ │┄ Files identical despite different names
│ │   --- JuliaLang-Statistics.jl-61a021b/LICENSE.md
│ ├── +++ JuliaStats-Statistics.jl-61a021b/LICENSE.md
│ │┄ Files identical despite different names
│ │   --- JuliaLang-Statistics.jl-61a021b/README.md
│ ├── +++ JuliaStats-Statistics.jl-61a021b/README.md
│ │┄ Files identical despite different names
│ │   --- JuliaLang-Statistics.jl-61a021b/src/Statistics.jl
│ ├── +++ JuliaStats-Statistics.jl-61a021b/src/Statistics.jl
│ │┄ Files identical despite different names
│ │   --- JuliaLang-Statistics.jl-61a021b/test/runtests.jl
│ ├── +++ JuliaStats-Statistics.jl-61a021b/test/runtests.jl
│ │┄ Files identical despite different names

The srccache tarball was extracted from julia-1.8.2-full.tar.gz.

@NHDaly
Copy link
Member

NHDaly commented Feb 21, 2023

CC: @KristofferC, this ended up breaking our build unexpectedly. We have a workaround now, but it's a bit hacky. Is there a way to resolve this by putting the Statistics package back where it was expected to be in the 1.8.2 build?

@KristofferC
Copy link
Member

KristofferC commented Feb 21, 2023

Moving back JuliaStats/Statistics.jl to JuliaLang/Statistics.jl? The issue with that is that newer versions have hashes generated based on the new location though, no?

@fredrikekre
Copy link
Member

JuliaLang can fork JuliaStats/Statistics.jl and then archive it. Should also change the build setup such that the GitHub org of packages doesn't end up in the tarball perhaps.

@NHDaly
Copy link
Member

NHDaly commented Feb 22, 2023

Yeah, thanks @fredrikekre, i think that's what i had in mind. that sounds great. thanks!

@inkydragon
Copy link
Member

inkydragon commented Feb 22, 2023

JuliaLang can fork JuliaStats/Statistics.jl and then archive it.

We could archive, upload and release an older version in JuliaStats/Statistics.jl.
Specify the prefix manually when archiving. (--prefix=)
Then update the link and hash.

Note: According to #48466 only manually uploaded release packages has a stable hash.


An unstable Workaround https://github.com/JuliaStats/Statistics.jl/archive/$1.tar.gz

Details
curl -L -o with-repo-name.tar.gz  https://api.github.com/repos/JuliaStats/Statistics.jl/tarball/61a021bcb330e6c52f2435f2abaffc77875ab6f2
curl -L -o no-repo-name.tar.gz  https://github.com/JuliaStats/Statistics.jl/archive/61a021bcb330e6c52f2435f2abaffc77875ab6f2.tar.gz
diff <(tar -tvf with-repo-name.tar.gz | sort) <(tar -tvf no-repo-name.tar.gz | sort)

The latter API generates a folder name that does not contain the name of the github organization.

output:

1,18c1,18
< -rw-rw-r-- root/root        89 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/docs/Project.toml
< -rw-rw-r-- root/root       186 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/docs/src/index.md
< -rw-rw-r-- root/root       324 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/Project.toml
< -rw-rw-r-- root/root       365 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/docs/make.jl
< -rw-rw-r-- root/root       780 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/.ci/test_and_change_uuid.jl
< -rw-rw-r-- root/root      1164 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/README.md
< -rw-rw-r-- root/root      1300 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/LICENSE.md
< -rw-rw-r-- root/root      1941 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/.github/workflows/ci.yml
< -rw-rw-r-- root/root     37648 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/src/Statistics.jl
< -rw-rw-r-- root/root     38547 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/test/runtests.jl
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/.ci/
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/.github/
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/.github/workflows/
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/docs/
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/docs/src/
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/src/
< drwxrwxr-x root/root         0 2022-02-07 16:43 JuliaStats-Statistics.jl-61a021b/test/
---
> -rw-rw-r-- root/root        89 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/docs/Project.toml
> -rw-rw-r-- root/root       186 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/docs/src/index.md
> -rw-rw-r-- root/root       324 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/Project.toml
> -rw-rw-r-- root/root       365 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/docs/make.jl
> -rw-rw-r-- root/root       780 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/.ci/test_and_change_uuid.jl
> -rw-rw-r-- root/root      1164 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/README.md
> -rw-rw-r-- root/root      1300 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/LICENSE.md
> -rw-rw-r-- root/root      1941 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/.github/workflows/ci.yml
> -rw-rw-r-- root/root     37648 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/src/Statistics.jl
> -rw-rw-r-- root/root     38547 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/test/runtests.jl
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/.ci/
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/.github/
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/.github/workflows/
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/docs/
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/docs/src/
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/src/
> drwxrwxr-x root/root         0 2022-02-07 16:43 Statistics.jl-61a021bcb330e6c52f2435f2abaffc77875ab6f2/test/

@inkydragon
Copy link
Member

One quick fix is to update the hash directly.

Which versions do we need to fix?

  • v1.6.7
  • ...

@vchuravy
Copy link
Member

I just forked the repo back to JuliaSyntax.jl this should recreate the archives with the correct sha's.

@giordano
Copy link
Contributor

JuliaSyntax.jl?

@vchuravy
Copy link
Member

Sorry... thinking about something else while typing... JuliaLang/Statistics.jl https://github.com/JuliaLang/Statistics.jl

@vchuravy
Copy link
Member

@inkydragon tested this in #48787 so it seems that worked?

@NHDaly
Copy link
Member

NHDaly commented Feb 28, 2023

🙏 thank you @vchuravy! 🫶

@fredrikekre
Copy link
Member

Cache still seems to be poisoned:

$ curl -fsSL https://cache.julialang.org/https://api.github.com/repos/JuliaLang/Statistics.jl/tarball/61a021bcb330e6c52f2435f2abaffc77875ab6f2 | sha512sum
c22dc7c5aa41886a4abc683b5d2be04538ff043cc81e3f5ca482fed0dd160cdfebe7c3d3366b324dc92678d69ce069a92a8ce38d396a7be6fc6bf10c1abfa67e  -

$ curl -fsSL https://api.github.com/repos/JuliaLang/Statistics.jl/tarball/61a021bcb330e6c52f2435f2abaffc77875ab6f2 | sha512sum
27e6f153f119638c4ed8e29127db10c1aff4fe5c14217a86a65d2bcb7ffbd3ed8e22613ed26fe0b9ffbb525ba00fc673be989d9da50e10fa12fd9a460ceeddcf  -

while building Julia 1.8.5.

@fredrikekre fredrikekre reopened this Mar 1, 2023
@vtjnash
Copy link
Member

vtjnash commented Mar 1, 2023

Is that the only one? Elliot may need to tell https://cache.julialang.org to reload that file from the source manually (the ETag is probably unchanged, since it is from the same commit hash).

@fredrikekre
Copy link
Member

@staticfloat fixed the cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
building Build system, or building Julia or its dependencies
Projects
None yet
Development

No branches or pull requests

10 participants