Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yarn install fails with ENOENT: no such file or directory occasionally #2629

Closed
NiGhTTraX opened this issue Feb 4, 2017 · 175 comments · Fixed by linkerd/linkerd2#3826
Closed
Assignees
Labels
cat-bug fixed-in-modern This issue has been fixed / implemented in Yarn 2+.

Comments

@NiGhTTraX
Copy link

Running yarn install as part of a build step for a Docker image based on node:7 fails on Travis CI with ENOTEMPTY, EEXISTS errors. It always seems to error on the webdriverio package.

yarn install v0.19.1
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/webdriverio/-/webdriverio-4.6.2.tgz: ENOENT: no such file or directory, open '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol/timeouts.js'".

When Travis runs yarn install as part of the install phase it works just fine. The error only happens when building a Docker image.

Repo which reproduces this issue.

node:7
OS: Docker + Travis CI
yarn: 0.19.1
package.json
yarn.lock

I've tried installing yarn both with npm install -g and with apt and both methods cause the failure on Travis.

Weirdly enough, the image builds successfully on my local machine which runs Ubuntu 16.04.1 LTS with Docker version 1.13.0, build 49bf474.

@Daniel15
Copy link
Member

Daniel15 commented Feb 5, 2017

Interesting, so it only fails on Travis, but works when testing locally? That's very weird given Docker is supposed to ensure the environment is consistent.

@NiGhTTraX
Copy link
Author

@Daniel15 I know right...

I've downgraded node to version 6 and it still fails on Travis. I've added the --verbose flag to yarn install and all I got was

verbose Performing "GET" request to "https://registry.yarnpkg.com/spawn-wrap/-/spawn-wrap-1.3.4.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/yargs/-/yargs-6.6.0.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-4.2.1.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/fibers/-/fibers-1.0.15.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/selenium-standalone/-/selenium-standalone-5.11.2.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/tcp-port-used/-/tcp-port-used-0.1.2.tgz".
verbose Performing "GET" request to "https://registry.yarnpkg.com/babel-runtime/-/babel-runtime-5.8.38.tgz".
verbose Error: ENOTEMPTY: directory not empty, rmdir '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol'
    at Error (native)
error An unexpected error occurred: "ENOTEMPTY: directory not empty, rmdir '/usr/local/share/.cache/yarn/npm-webdriverio-4.6.2-dd095ee618896a21c8f1b9d4278736d85a64ca0f/lib/protocol'".

I am open to ideas on how to debug this.

@twooster
Copy link

twooster commented Feb 7, 2017

Downgrading to yarn 0.18.1 seemed to fix this for me. Seems like 0.19 might have a regression; see #1834

@victornoel
Copy link
Contributor

victornoel commented Feb 23, 2017

I also have this problem with yarn 0.23.3, it's happening not when building an image but simply when running some CI.
The error is the following:

$ time yarn --frozen-lockfile
yarn install v0.20.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "ENOTEMPTY: directory not empty, rmdir '/builds/linagora/petals-cockpit/yarncache/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/src'".
info If you think this is a bug, please open a bug report with the information provided in "/builds/linagora/petals-cockpit/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

real	0m9.812s
user	0m7.596s
sys	0m0.932s

I think there may be some strange way of doing remove of files…

Important point: the cache was empty!

@victornoel
Copy link
Contributor

And on my machine, if I try to repro, I get this:

yarn install v0.20.3
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-beta.8.tgz: EEXIST: file already exists, mkdir '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/src/metadata'".
info If you think this is a bug, please open a bug report with the information provided in "/home/vnoel/Linagora/Petals/dev/git/petals-cockpit-new/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

@victornoel
Copy link
Contributor

And with yarn 0.21.2:

yarn install v0.21.2
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-beta.8.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-beta.8-8d9c8a64e7c26ff7208404e716deea94bb509cd7/bundles/core.umd.js'".
info If you think this is a bug, please open a bug report with the information provided in "/home/vnoel/Linagora/Petals/dev/git/petals-cockpit-new/frontend/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

That's terrible!

@victornoel
Copy link
Contributor

And I concur with @twooster about 0.18.1 working ok!

@victornoel
Copy link
Contributor

@Daniel15 it doesn't work locally either. Actually, it NEVER work when the cache is empty for me!

@bestander
Copy link
Member

@victornoel the recent error could be #2714

@victornoel
Copy link
Contributor

@bestander I did try 0.19.1 at the time and it didn't work…

I retried, and now the bug:

  • does not appear with an empty cache, but does appear in the following case (I really hope it is reproducible…):
    • rm -rf the yarn cache
    • clone https://gitlab.com/linagora/petals-cockpit.git
    • checkout 5f31ccb4b2357201baa50539b30702cffceb6992
    • run yarn in the frontend directory
    • checkout master
    • run yarn again in the frontend directory
    • I get: error An unexpected error occurred: "http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, utime '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core/testing.js'". (I am using my own registry, but the same happens without it)
  • does appear with yarn 0.21.2, 0.19.1 but not with 0.18.2

So I don't think it is the same, let's hope you can reproduce it at least…

@victornoel
Copy link
Contributor

(actually, I just tried again and reproduced the bug with an empty cache and yarn 0.21.2 while it wasn't the case before, maybe there is another file somewhere else that is the source of this problem, and that is not in the cache?)

@victornoel
Copy link
Contributor

victornoel commented Feb 27, 2017

@bestander I will still test yarn as soon as #2744 is available as a nightly build :)

@bestander
Copy link
Member

Ping me if I can help.
The best action is to send a PR with a broken (and skipped) e2e test.

@victornoel
Copy link
Contributor

@bestander well, nope, I still get errors such as:


➜  frontend git:(master) ✗ yarn
yarn install v0.22.0-20170227.1509
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/facade/lang.d.ts'".

or:

➜  frontend git:(master) ✗ yarn
yarn install v0.22.0-20170227.1509
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/typescript/-/typescript-2.2.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-typescript-2.2.1-4862b662b988a4c8ff691cc7969622d24db76ae9/lib/typescriptServices.js'".

I will see if I can make an e2e test.

@victornoel
Copy link
Contributor

@bestander anyway I can get a complete stacktrace of the error?

I only see this in the yarn-error.log:

Trace: 
  Error: http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.es5.js'
      at Error (native)

That's a bit useless :)

@victornoel
Copy link
Contributor

the detailled error is:

{ Error: http://docker0.gso.lan:8081/repository/npm/@angular/core/-/core-4.0.0-rc.1.tgz: ENOENT: no such file or directory, lstat '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js'
    at Error (native)
  errno: -2,
  code: 'ENOENT',
  syscall: 'lstat',
  path: '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js',
  fstream_type: 'File',
  fstream_path: '/home/vnoel/.cache/yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/@angular/core.js',
  fstream_class: 'FileWriter',
  fstream_stack: 
   [ '/home/vnoel/Linagora/Petals/dev/git/yarn/node_modules/fstream/lib/writer.js:285:28',
     '/home/vnoel/Linagora/Petals/dev/git/yarn/node_modules/graceful-fs/polyfills.js:284:29',
     'FSReqWrap.oncomplete (fs.js:123:15)' ] }

Not sure exactly what to do with this… it's happening at package-fetcher.js, line 56 exactly,  I have trouble finding the source though…

@victornoel
Copy link
Contributor

It seems stupid, but I feel like it fails only when my networked npm mirror (a sonatype nexus in my company) has mirrored the @angular/core artefact. If it hasn't, things go well and then it fails on another artefact that is already mirrored (typescript in this case).

If I remove the artefact from the nexus mirror by hand, it works!

So… it is a bit like yarn can't keep up if things arrive too fast ^^ because when I use the normal npm registry without using my mirror, things usually goes well (we have a slow internet connection).
And it would explain why it often fails on CI systems because they usually have very fast internet connections…

That's a bit of stretch to conclude that, but it could help find the origin of the problem.
WDYT @bestander?

For the record, I think the error is emanating from tar.Extract step in the fetching pipeline, but I'm not totally sure ^^

@bestander
Copy link
Member

Thanks for researching more, @victornoel, you might be on something here.

@bestander
Copy link
Member

bestander commented Mar 1, 2017

I can repro scenario from #2629 (comment).
Looking into it.

I get

error An unexpected error occurred: "https://registry.yarnpkg.com/typescript/-/typescript-2.2.1.tgz: ENOENT: no such file or directory, lstat '/Users/bestander/Library/Caches/Yarn/npm-typescript-2.2.1-4862b662b988a4c8ff691cc7969622d24db76ae9/lib/typescriptServices.js'".

However if I just try yarn install again and again it will eventually finish the installation.
Looks like exploding the .tgz file ends with error.

Update:

  • the .tgz seems fine, I can unzip manually the one that fails during fetch phase
  • I wonder if tar package is throwing this error for some reason, could it be concurrency?

@bestander
Copy link
Member

Some help investigating why untaring those few dependencies (in my case typescript and angular-core) causes error is welcome.
Concurrency? Bug in https://github.com/npm/node-tar?

@bestander
Copy link
Member

@victornoel, can you reproduce the bug with yarn install --network-concurrency 1?

@victornoel
Copy link
Contributor

@bestander with --network-concurrency 1 the bug does not appear (while without it, it appears every time).
But what is the default value of this parameter? Whichever value I choose for it (1, 2, 4, 8), it works, while if I don't put it at all, it fails…

@bestander
Copy link
Member

The default is 15, I can repro the issue with concurrency 15 with a clean checkout of https://gitlab.com/linagora/petals-cockpit.git#075bac4c54fee466568c000c7ffe8025f593e212.

@victornoel
Copy link
Contributor

Excellent news! One step further toward a solution AND a workaround :)

@bestander
Copy link
Member

bestander commented Mar 2, 2017

Some results.

TL;DR I am out of ideas how to properly fix it for good, this needs some deeper Node.js knowledge.

  1. Network can be eliminated from possible issues.
    I've setup offline mirror for the .tgz files in yarn.lock and can reproduce the issue with packages installed from disk.

The issue is in unzip/untar stream in tarball-fetcher code.

  1. I tried a different library that extracts tar - https://github.com/mafintosh/tar-fs vs current https://github.com/npm/node-tar/. They both fail the same way.
    Going a bit deeper - exceptions are happening in node when doing multiple mkdirp operations
Error: ENOENT: no such file or directory, chmod '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts'
  errno: -2,
  code: 'ENOENT',
  syscall: 'chmod',
  path: '/Users/bestander/Library/Caches/Yarn/npm-@angular/core-4.0.0-rc.1-7f87b7696b407476e45d6d3c1880a50d5afbb6e3/typings/src/di/injector.d.ts' }

I think core-4.0.0 and typescript-2.2.1 fail because they have quite a few files and deep folder structures, and they fail to install while making many concurrent mkdir/copy operations.

Every time there is a different syscall that fails: chmod, rmdir, mkdir, lstat, utime.

And it is not something obvious in the libraries' code.

  1. Fails the same on Node 4, 6 and 7.

  2. I could not reproduce the error with concurrency set to 8, so I'll send a PR to reduce default network concurrency.


  1. I was wondering how concurrency affects installation speed.

5.1. Using offline-mirror (no download), on my MBPro 13", clean cache and using node-tar to unzip files.
Concurrency 12 - fails
Concurrency 8 - 18 seconds
Concurrency 4 - 18 seconds
Concurrency 2 - 21 seconds

5.2. Using offline-mirror (no download), on my MBPro 13", clean cache and using tar-fs to unzip files.
Concurrency 12 - 15 seconds
Concurrency 8 - 15 seconds
Concurrency 4 - 17 seconds
Concurrency 2 - 18 seconds

5.3. Downloading packages from internet, on my MBPro 13", clean cache and using tar-fs to unzip files.
Concurrency 12 - failed once
Concurrency 8 - 21 seconds
Concurrency 4 - 23 seconds
Concurrency 2 - 34 seconds

Looks like setting concurrency to 8 is safe enough, also it makes sense to switch the tar library.
I'll follow up with a PR.

@bnbarak
Copy link

bnbarak commented Jun 5, 2020

May I ask if there is a solution in sight? We keep deleting 20 node_modules and a yarn.lock to fix it.

@clero
Copy link

clero commented Jun 5, 2020

May I ask if there is a solution in sight? We keep deleting 20 node_modules and a yarn.lock to fix it.

I personally have switched to lerna regarding workspace handling.

@k3n
Copy link

k3n commented Jun 5, 2020

Fairly sure that current Lerna simply passes-thru to Yarn.

I was able to work-around my issues by adding a nohoist config for Ember packages -- my current env is using an older version of Ember, which is incompatible with workspaces.

    "nohoist": [
      "**/ember-package/*ember*",
      "**/ember-package/*ember*/**",
      "**/ember-package/loader.js"
    ]

@cinderblock
Copy link

cinderblock commented Jun 6, 2020

I think we've got a minimal repro here now: #7212 (comment)

@co-codr
Copy link

co-codr commented Jun 16, 2020

Deleting yarn.lock then yarn install worked for me as well

@cadavre
Copy link

cadavre commented Jul 29, 2020

Any news here? Our CI process just went down with all pipelines failing because of dependency installation failure from yarn. That's ridiculous.

Setting --network-concurrency fixes nothing and jobs run on clean machines (no node_modules, no yarn .cache).

@merceyz
Copy link
Member

merceyz commented Aug 1, 2020

@cadavre Not making any guarantees but this might not be a problem in v2, you can try it with

yarn set version 2 && yarn config set nodeLinker node-modules

https://yarnpkg.com/getting-started/install#per-project-install
https://yarnpkg.com/configuration/yarnrc#nodeLinker

@garyo
Copy link

garyo commented Sep 1, 2020

This just started happening to me too, after upgrading some of my vue & firebase dependencies. Now 100% repeatable in my CI and dev machines. Adding --network-concurrency 1 does not reliably fix it. I'm not out of disk space or inodes. I'm on WSL1. Yarn 1.22.4.

@cadavre
Copy link

cadavre commented Sep 1, 2020

I fixed it by temporarily changing cache directory, that I delete just afterwards.

For me it's Docker build:

RUN yarn install --check-files --cache-folder .ycache && rm -rf .ycache

@merceyz
Copy link
Member

merceyz commented Jan 2, 2021

Closing as fixed in v2 where the cache has been greatly improved to avoid these sorts of issues

https://yarnpkg.com/getting-started/migration

@merceyz merceyz closed this as completed Jan 2, 2021
@merceyz merceyz added the fixed-in-modern This issue has been fixed / implemented in Yarn 2+. label Jan 2, 2021
ori-moisis added a commit to ori-moisis/salto that referenced this issue Jan 27, 2021
ori-moisis added a commit to ori-moisis/salto that referenced this issue Jan 27, 2021
ori-moisis added a commit to ori-moisis/salto that referenced this issue Jan 27, 2021
ori-moisis added a commit to ori-moisis/salto that referenced this issue Jan 31, 2021
ori-moisis added a commit to salto-io/salto that referenced this issue Jan 31, 2021
* Update jest version
* Retry install in circle CI to work around yarn bug - yarnpkg/yarn#2629
somewhatabstract added a commit to Khan/render-gateway that referenced this issue Mar 26, 2021
This is to see if we can workaround the issue where yarn can spawn more versions of itself, causing it to corrupt it's own tgz expansion

See yarnpkg/yarn#6312 and yarnpkg/yarn#2629
somewhatabstract added a commit to Khan/render-gateway that referenced this issue Mar 26, 2021
The prepare script added by husky causes yarn to be re-entrant in its installs, which can lead to the same tarball being unpacked more than once at the same time, corrupting itself

See yarnpkg/yarn#6312 and yarnpkg/yarn#2629
somewhatabstract added a commit to Khan/render-gateway that referenced this issue Mar 26, 2021
## Summary:
The prepare script added by husky causes yarn to be re-entrant in its installs, which can lead to the same tarball being unpacked more than once at the same time, corrupting itself

See yarnpkg/yarn#6312 and yarnpkg/yarn#2629


Issue: none

## Test plan:
`yarn test`

When we deploy to gcloud, it should deploy.

Author: somewhatabstract

Reviewers: jeresig

Required Reviewers: 

Approved by: jeresig

Checks: ✅ codecov/project, ✅ Test (macOS-latest, 12.x), ✅ Test (macOS-latest, 10.x), ✅ Test (ubuntu-latest, 10.x), ✅ Coverage (ubuntu-latest, 12.x), ✅ Publish to branch, ⏭  Auto-approve dependabot PRs, ✅ Lint and flow check (ubuntu-latest, 12.x)

Pull request URL: #456
@re-ali

This comment has been minimized.

@cinderblock

This comment has been minimized.

@yarnpkg yarnpkg locked as resolved and limited conversation to collaborators May 29, 2021
@yarnpkg yarnpkg deleted a comment from Hintalo Aug 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cat-bug fixed-in-modern This issue has been fixed / implemented in Yarn 2+.
Projects
None yet
Development

Successfully merging a pull request may close this issue.