Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: YN0018 in Windows, but not macOS/Linux: "remote archive doesn't match the expected checksum" #5795

Closed
1 task
trusktr opened this issue Oct 17, 2023 · 19 comments
Labels
bug Something isn't working stale Issues that didn't get attention waiting for feedback Will autoclose in a while unless more data are provided

Comments

@trusktr
Copy link

trusktr commented Oct 17, 2023

Self-service

  • I'd be willing to implement a fix

Describe the bug

I just started using Yarn for the first time in a project (Berry, not Classic) and I'm loving it! But I'm now having this issue,

in GitHub Actions only, not locally.

To reproduce

Here's the job with the error:

https://github.com/lume/lume/actions/runs/6538252764/job/17753779120

The relevant output there:

➤ YN0000: ┌ Fetch step
Fetch step
➤ YN0018: docsify-cli@https://github.com/trusktr/docsify-cli.git#commit=6f78abeee273c7641e964fb5f9397b6b72e6753b: The remote archive doesn't match the expected checksum
➤ YN0058: three@https://github.com/trusktr/three.js.git#commit=d2fbde04e079890747e318adb2b3b88f8c556d38: Packing the package failed (exit code 1, logs can be found here: C:\Users\RUNNER~1\AppData\Local\Temp\xfs-8ba5194c\pack.log)
➤ YN0000: └ Completed in 9m 1s
➤ YN0000: Failed with errors in 9m 2s

The package docsify-cli is being installed from git, from github.com/trusktr/docsify-cli. Does this have anything to do with it perhaps?

I wonder if it is CRLFs...

Is this a known issue with packages installed from git?

Environment

System:
    OS: macOS 13.4
    CPU: (8) arm64 Apple M2
  Binaries:
    Node: 20.6.1 - /private/var/folders/7y/wy4mhdj114g5xj1ktvh6hdz80000gn/T/xfs-9bb50836/node
    Yarn: 3.6.4 - /private/var/folders/7y/wy4mhdj114g5xj1ktvh6hdz80000gn/T/xfs-9bb50836/yarn
    npm: 10.2.0 - ~/.npm-packages/bin/npm
    pnpm: 7.24.3 - ~/.npm-packages/bin/pnpm

Additional context

EDIT:

I tried putting .gitattributes with eol=lf in the trusktr/docsify-cli repo (its the add-cors branch), recreated my yarn.lock, and also added git config core.eol lf && git config core.autocrlf input to the beginning of my GitHub Actions before it runs the yarn install, but no luck.

@trusktr trusktr added the bug Something isn't working label Oct 17, 2023
@trusktr trusktr changed the title [Bug]: YN0018 in GitHub Actions, not local: "remote archive doesn't match the expected checksum" [Bug]: YN0018 in GitHub Actions (not local): "remote archive doesn't match the expected checksum" Oct 17, 2023
@trusktr
Copy link
Author

trusktr commented Oct 17, 2023

I'm not sure what is going on yet, but I think I have to use checksumBehavior: "ignore" for now. I assume that packages from npm and github are over HTTPS, so not that much of a security risk (let me know if it is). If something local can modify packages, it can also modify the lock file sums.

@clemyan
Copy link
Member

clemyan commented Oct 18, 2023

When installing a git dependency, the repo is cloned to the local filesystem then zipped to produce the package. So there are room for differences across systems (e.g. regarding file permissions: #5136)

I checked the zip archive I generated on a Windows machine with zipinfo and this is what I get

Archive:  docsify-cli-https-e76244022d-8.zip
Zip file size: 13746 bytes, number of entries: 27
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/lib/
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/lib/template/
-rw-r--r--  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/lib/template/.nojekyll
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/bin/
-rw-r--r--  6.3 unx     2959 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/bin/docsify
-rw-r--r--  6.3 unx     1067 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/LICENSE
-rw-r--r--  6.3 unx      612 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/lib/template/index.html
-rw-r--r--  6.3 unx      574 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/lib/template/index.local.html
-rw-r--r--  6.3 unx      132 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/lib/index.js
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/lib/util/
-rw-r--r--  6.3 unx      597 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/lib/util/index.js
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/tools/
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/tools/locales/
-rw-r--r--  6.3 unx      766 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/tools/locales/index.js
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/lib/commands/
-rw-r--r--  6.3 unx     1821 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/lib/commands/init.js
-rw-r--r--  6.3 unx     1190 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/lib/commands/serve.js
-rw-r--r--  6.3 unx     2569 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/lib/commands/start.js
-rw-r--r--  6.3 unx     1360 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/tools/locales/de.json
-rw-r--r--  6.3 unx     1228 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/tools/locales/en.json
-rw-r--r--  6.3 unx     1798 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/package.json
-rw-r--r--  6.3 unx     1049 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/tools/locales/zh.json
-rw-r--r--  6.3 unx     7308 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/CHANGELOG.md
-rw-r--r--  6.3 unx       34 b- stor 84-Jun-22 21:50 node_modules/docsify-cli/lib/template/README.md
-rw-r--r--  6.3 unx     2552 b- defX 84-Jun-22 21:50 node_modules/docsify-cli/README.md
27 files, 27616 bytes uncompressed, 9574 bytes compressed:  65.3%

Since you don't get the error locally I assume the zip on your system matches the checksum? Can you check whether your zip archive in your yarn cache matches the above (especially the perms on docsify-cli/bin/docsify)?

@trusktr trusktr changed the title [Bug]: YN0018 in GitHub Actions (not local): "remote archive doesn't match the expected checksum" [Bug]: YN0018 in Windows, but not macOS/Linux: "remote archive doesn't match the expected checksum" Oct 18, 2023
@trusktr
Copy link
Author

trusktr commented Oct 18, 2023

Ah, it was late. I should have clarified after I realized:

By "not locally" I meant "not locally on macOS" (that's what I was using at the time, shown in the Environment section), but then I was pushing to the repo and it was failing in CI for Windows.

But I in fact verified later after I got onto my Windows machine that the checksum error was in fact happening locally too.

I updated the title to be more accurate.

@arcanis
Copy link
Member

arcanis commented Oct 18, 2023

Probably something like: #2774 (comment)

If not, the ideal would be for you to exfiltrate the problematic archive and post it here (along with with the expected cache entry) so we can compare its byte content.

@trusktr
Copy link
Author

trusktr commented Oct 18, 2023

Here's the checksum: 5677d2b4d002b825463141b9980ddd74b2b0ca61d0e5fefb363b6c444436ae79f21b3dd6ef66542d42f8e634771b6cdb7c6eedd07b41e404f0bebf081e130c08

I'm not sure which archive for three.js is the one, I see this from yarn info after the failed install in the workspace that uses three:

PS D:\src\lume+lume\apps\first-person-shooter> yarn info three
└─ three@https://github.com/trusktr/three.js.git#commit=d2fbde04e079890747e318adb2b3b88f8c556d38
   └─ Version: 0.139.1

but I don't see 0.139.1 in the cache. Here are all the three zips I see in .yarn/cache/:

three-npm-0.157.0-1b19c1ef00-444797461c.zip
three-npm-0.139.2-62a04e8b92-9e016065b3.zip
three-npm-0.112.1-c2f478971f-01e4c84b62.zip
three-npm-0.111.0-aa9a45040c-a7d63c008b.zip

I'm guessing none of those are it. Any idea where to get the zip after the failed install? @clemyan perhaps you can upload yours?


Would it be ok if the checksumBehavior option accepted a new value like pkg-only that checks the sums of downloaded archives only, but otherwise won't check the sum for things like packages from git? It would be better than ignoring everything.

@clemyan
Copy link
Member

clemyan commented Oct 18, 2023

@trusktr My understanding is that you have a (local) MacOS machine that can install without checksum errors. Meanwhile me on my Windows machine is seeing checksum errors. If that is the case, we have two differing zip archives.

One of the errors is on docsify-cli, so I dug up the zip archive for it. I posted the zipinfo above

docsify-cli-https-e76244022d-8.zip

You can find the global cache in $(yarn config globalFolder)/cache. Assuming you are not seeing checksum errors on your MacOS machine, can you check your docsify-cli-https-e76244022d-8.zip under there? It should have the SHA512 hash 08952b6482601ea8bdac349251bc304022ebf07f4353b149452f755844d712b9b6794e0e7f5e6273d54f5782fcf46c4ce993e3967c3eb544e9725cc984bdabc8

@trusktr
Copy link
Author

trusktr commented Oct 21, 2023

Oh wow, I'm sorry, I thought I was seeing this error for the three package (was I? Now I'm not sure.) I'll circle back to this.

@arcanis arcanis added the waiting for feedback Will autoclose in a while unless more data are provided label Oct 27, 2023
@Asuza
Copy link

Asuza commented Nov 1, 2023

I'm having a very similar issue. When running yarn install locally on Windows, I have no problems whatsoever. However, when the CI pipeline runs (this machine is also running Windows), every single package ends up with a The remote archive doesn't match the expected checksum error. I don't want to ignore checksums, so I'm not sure how to proceed at this point.

This is happening with Yarn 4.0.1.

@Asuza
Copy link

Asuza commented Nov 1, 2023

After some fiddling around, I finally understood that my git repository was not set up to know that .zip files should be considered binary, so git was the problem. It was likely performing some kind of line ending conversion on the .zip files.

I had to add the following to my .gitattributes file:

*.zip binary

Then, had to clear my yarn cache and reinstall so that the archives were back in the correct format and would not be changed by git.

@yarnbot
Copy link
Collaborator

yarnbot commented Dec 1, 2023

Hi! 👋

It seems like this issue as been marked as probably resolved, or missing important information blocking its progression. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it.

@yarnbot yarnbot added the stale Issues that didn't get attention label Dec 1, 2023
@trusktr
Copy link
Author

trusktr commented Dec 2, 2023

I don't think this issue is resolved, but I've moved on by not installing some packages from git and now installing them from npm.

@yarnbot
Copy link
Collaborator

yarnbot commented Jan 1, 2024

Hi! 👋

It seems like this issue as been marked as probably resolved, or missing important information blocking its progression. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it.

@yarnbot yarnbot added stale Issues that didn't get attention and removed stale Issues that didn't get attention labels Jan 1, 2024
@bdunderscore
Copy link

bdunderscore commented Jan 7, 2024

This is still an issue, see e.g. https://github.com/bdunderscore/modular-avatar/actions/runs/7436253662/job/20232535460 where the yarn install --immutable succeeds on Windows and WSL, but fails in Github actions.

In this case I'm not checking in any archives, so this is purely an issue with how yarn retrieves archives from NPM.

bdunderscore added a commit to bdunderscore/modular-avatar that referenced this issue Jan 7, 2024
yarn --immutable seems to be unreliable due to yarnpkg/berry#5795
bdunderscore added a commit to bdunderscore/modular-avatar that referenced this issue Jan 7, 2024
yarn --immutable seems to be unreliable due to yarnpkg/berry#5795
@yarnbot
Copy link
Collaborator

yarnbot commented Feb 6, 2024

Hi! 👋

It seems like this issue as been marked as probably resolved, or missing important information blocking its progression. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it.

@yarnbot yarnbot added the stale Issues that didn't get attention label Feb 6, 2024
@yarnbot yarnbot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 11, 2024
@ospfranco
Copy link

I just encountered this. For me, it is not an option to not use git dependencies as I have a forked project and I need to pull it via https. I checked to see if I have any binaries but that's not the case. Disabling checksums is a workaround but sounds insecure. Do you have any other suggestions as to what might be going on?

HigherOrderLogic added a commit to the-forked-via/app that referenced this issue May 20, 2024
HigherOrderLogic added a commit to the-forked-via/app that referenced this issue May 20, 2024
@odinho
Copy link

odinho commented Jun 3, 2024

This was something of a WTF. It was only a problem on Github Actions, I was unable to reproduce otherwise. I solved it by upgrading yarn + the lock file, and a lot of things dependent on it:

  • Upgraded to yarn berry in all the deps coming from other git repos
  • Also upgraded yarn in my main repo
  • Had to also add "exports": "./dist/index.js" to the package.json of dependency repo (else build would fail in main repo for some reason (I didn't change any node versions))
  • Had to switch the prepare script to prepack (for compiling the typescript)
  • Then I had to hunt down every yarn.lock file in main repo (git ls-files | grep yarn.lock) and run yarn again to update it

@n0099
Copy link
Contributor

n0099 commented Jun 16, 2024

#5795 (comment)

In this case I'm not checking in any archives, so this is purely an issue with how yarn retrieves archives from NPM.

I'm encountering this too after opt-out pnp: n0099/open-tbm@82904a0

 YN0018:  @fortawesome/vue-fontawesome@npm:3.0.8: The remote archive doesn't match the expected checksum
 YN0018:  v-viewer@npm:3.0.13: The remote archive doesn't match the expected checksum
 YN0018:  vue-types@npm:3.0.2: The remote archive doesn't match the expected checksum
 YN0018:  vue-router@npm:4.3.3: The remote archive doesn't match the expected checksum
 YN0018:  ant-design-vue@npm:4.2.3: The remote archive doesn't match the expected checksum

Linux:

Archive:  /home/n0099/.yarn/berry/cache/vue-router-npm-4.3.3-af8e67016f-10c0.zip
Zip file size: 812925 bytes, number of entries: 23
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/vue-router/
-rw-r--r--  6.3 unx     1100 b- stor 84-Jun-22 21:50 node_modules/vue-router/LICENSE
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/
-rw-r--r--  6.3 unx   148647 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.cjs
-rw-r--r--  6.3 unx   112962 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.prod.cjs
-rw-r--r--  6.3 unx      179 b- stor 84-Jun-22 21:50 node_modules/vue-router/index.js
-rw-r--r--  6.3 unx       50 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.cjs.js
-rw-r--r--  6.3 unx       55 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.cjs.prod.js
-rw-r--r--  6.3 unx   148093 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.esm-browser.js
-rw-r--r--  6.3 unx       38 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.esm-bundler.js
-rw-r--r--  6.3 unx   161626 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.global.js
-rw-r--r--  6.3 unx    24990 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.global.prod.js
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/vue-router/vetur/
-rw-r--r--  6.3 unx     1847 b- stor 84-Jun-22 21:50 node_modules/vue-router/vetur/attributes.json
-rw-r--r--  6.3 unx     5218 b- stor 84-Jun-22 21:50 node_modules/vue-router/package.json
-rw-r--r--  6.3 unx      636 b- stor 84-Jun-22 21:50 node_modules/vue-router/vetur/tags.json
-rw-r--r--  6.3 unx     4361 b- stor 84-Jun-22 21:50 node_modules/vue-router/README.md
-rw-r--r--  6.3 unx   150716 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.mjs
-rw-r--r--  6.3 unx       76 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.node.mjs
-rw-r--r--  6.3 unx       95 b- stor 84-Jun-22 21:50 node_modules/vue-router/vue-router-auto-routes.d.ts
-rw-r--r--  6.3 unx      101 b- stor 84-Jun-22 21:50 node_modules/vue-router/vue-router-auto.d.ts
-rw-r--r--  6.3 unx    48483 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.d.ts
23 files, 809273 bytes uncompressed, 809273 bytes compressed:  0.0%

Windows:

Archive:  /c/Users/n0099/AppData/Local/Yarn/Berry/cache/vue-router-npm-4.3.3-af8e67016f-10c0.zip
Zip file size: 776245 bytes, number of entries: 23
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/vue-router/
-rw-r--r--  6.3 unx     1100 b- stor 84-Jun-22 21:50 node_modules/vue-router/LICENSE
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/
-rw-r--r--  6.3 unx   148647 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.cjs
-rw-r--r--  6.3 unx   112962 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.prod.cjs
-rw-r--r--  6.3 unx      179 b- stor 84-Jun-22 21:50 node_modules/vue-router/index.js
-rw-r--r--  6.3 unx       50 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.cjs.js
-rw-r--r--  6.3 unx       55 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.cjs.prod.js
-rw-r--r--  6.3 unx   148093 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.esm-browser.js
-rw-r--r--  6.3 unx       38 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.esm-bundler.js
-rw-r--r--  6.3 unx   161626 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.global.js
-rw-r--r--  6.3 unx    24990 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.global.prod.js
drwxr-xr-x  6.3 unx        0 b- stor 84-Jun-22 21:50 node_modules/vue-router/vetur/
-rw-r--r--  6.3 unx     1847 b- stor 84-Jun-22 21:50 node_modules/vue-router/vetur/attributes.json
-rw-r--r--  6.3 unx     5218 b- stor 84-Jun-22 21:50 node_modules/vue-router/package.json
-rw-r--r--  6.3 unx      636 b- stor 84-Jun-22 21:50 node_modules/vue-router/vetur/tags.json
-rw-r--r--  6.3 unx     4361 b- stor 84-Jun-22 21:50 node_modules/vue-router/README.md
-rw-r--r--  6.3 unx   150716 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.mjs
-rw-r--r--  6.3 unx       76 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.node.mjs
-rw-r--r--  6.3 unx       95 b- stor 84-Jun-22 21:50 node_modules/vue-router/vue-router-auto-routes.d.ts
-rw-r--r--  6.3 unx      101 b- stor 84-Jun-22 21:50 node_modules/vue-router/vue-router-auto.d.ts
-rw-r--r--  6.3 unx    48601 b- defX 24-Jun-15 16:04 node_modules/vue-router/dist/vue-router.d.ts
23 files, 809391 bytes uncompressed, 772593 bytes compressed:  4.5%
-Archive:  /home/n0099/.yarn/berry/cache/vue-router-npm-4.3.3-af8e67016f-10c0.zip
-Zip file size: 812925 bytes, number of entries: 23
+Archive:  /c/Users/n0099/AppData/Local/Yarn/Berry/cache/vue-router-npm-4.3.3-af8e67016f-10c0.zip
+Zip file size: 776245 bytes, number of entries: 23
--rw-r--r--  6.3 unx    48483 b- stor 84-Jun-22 21:50 node_modules/vue-router/dist/vue-router.d.ts
-23 files, 809273 bytes uncompressed, 809273 bytes compressed:  0.0%
+-rw-r--r--  6.3 unx    48601 b- defX 24-Jun-15 16:04 node_modules/vue-router/dist/vue-router.d.ts
+23 files, 809391 bytes uncompressed, 772593 bytes compressed:  4.5%

#2774 (comment)
#5957 (comment)
#6105 (comment)
Run following

yarn cache clean --all
rm yarn.lock
yarn

on windows side will sync to the linux one and fix this.

@nkallen
Copy link

nkallen commented Jul 31, 2024

I also ran into this issue, also trying to build on Windows in github actions, but the original dependency was added to package.json on a mac:

package.json

   "my-dependency": "github:..."

@Nantris
Copy link

Nantris commented Nov 8, 2024

If people have to use ignore checksum, it would be nice if we could do this on a selective basis - because otherwise using Yarn Berry is more insecure than using Yarn 1.x...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working stale Issues that didn't get attention waiting for feedback Will autoclose in a while unless more data are provided
Projects
None yet
Development

No branches or pull requests