[BUG] ^7.20.3 no longer resolves local package first on install (workspaces)

[rijkvanzanten]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

In a workspaces based environment, for example:

.
+-- package.json
`-- packages
   +-- a
   |   `-- package.json
   `-- b
       `-- package.json

Running npm install b --workspace a will no longer install the locally linked package correctly. Instead, it'll use the version hosted on npm, or throw a 404 if you use a custom (unique) package name.

Expected Behavior

In <7.20.x, NPM would resolve the local package, and correctly alter package.json and package-lock.json.

Steps To Reproduce

With node v16.6.1 and npm 7.20.3 installed:

1. Make a new directory

mkdir ./npm-install-debug

2. Add a package.json

cd ./npm-install-debug
npm init -y

3. Setup 2 workspace packages

npm init -w ./packages/a
npm init -w ./packages/b

(I used @rijk/a and @rijk/b as names, to avoid confusion with the existing packages a and b.)

5. Run npm install in the root, to give it a chance to npm link all the packages (not sure if needed)

npm install

6. Try installing @rijk/b as a dependency of @rijk/a

npm install @rijk/b -w @rijk/a

5. See error...

npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@rijk%2fb - Not found
npm ERR! 404
npm ERR! 404  '@rijk/b@*' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/rijkvanzanten/.npm/_logs/2021-08-10T20_07_25_648Z-debug.log

---

If you install 7.20.2 or below (npm i -g npm@7.20.2), the above flow still works as expected.

Environment

• OS: macOS 11.4, also seen in GitHub Actions
• Node: v16.6.1
• npm: 7.20.3 and up

[gribnoysup]

Running into the same issue without even trying to install specific packages inside the workspaces, just running npm install from the project root where some workspaces are not yet published / missing from the registry. If that's any help for triaging, here is a very verbose install log of a failed install

[ParadoxInfinite]

I could reproduce the result by using the steps in the issue.
After a bit of shallow debugging, I found that the error E404 is found even in v7.20.2, it is not thrown however. In the latest build v7.20.6, it is. So, not sure if this is intended or a bug.

Now, I did a bit more digging around and found there are a couple of interesting keys which differ between v7.20.6 and v7.20.2

In the parent node:

ArboristVirtualNode {
  name: 'a',
  packageName: '@paradox/a',
  version: '1.0.0',
  location: '',
  path: '/Users/paradoxinfinite/Documents/testing/packages/a',
  extraneous: true,
  dev: true,
  optional: true,
  peer: true,
  isProjectRoot: true,  // this field is not present in v7.20.2
  edgesOut: Map(1) { '@paradox/b' => { prod @paradox/b@^1.0.0 MISSING } },
  sourceReference: ArboristNode {
    name: 'a',
    packageName: '@paradox/a',
    version: '1.0.0',
    location: 'packages/a',
    path: '/Users/paradoxinfinite/Documents/testing/packages/a',
    isWorkspace: true,  // this field is not present in v7.20.2
    edgesOut: Map(1) {
      '@paradox/b' => { prod @paradox/b@^1.0.0 -> node_modules/@paradox/b }
    }
  }
}

I couldn't get any further than this, so if someone who knows the https://github.com/npm/arborist codebase better could help or point me out on where to look, that'd be great!

Finally, by the looks of it, this issue would be more relevant to https://github.com/npm/arborist, you could try raising an issue there!

[willnode]

I had a problem with Gatsby running with Workspaces.

npm i gatsby-cli -g
npm init -y
gatsby new a
gatsby new b

Then adding workspaces in package.json

Before dedupe

D:\m\a>npm list methods
gatsby-starter-default@0.1.0 D:\m\a
`-- gatsby@3.11.1
  `-- express@4.17.1
    `-- methods@1.1.2

After dedupe

D:\m\a>npm list methods
a@0.1.0 D:\m\a
`-- (empty)
D:\m\b>npm list methods
b@0.1.0 D:\m\b
`-- (empty)

Despite

D:\m>npm list methods
m@1.0.0 D:\m
`-- a@0.1.0 -> .\a
  `-- gatsby@3.11.1
    `-- express@4.17.1
      `-- methods@1.1.2

This broke my gatsby builds.

Unfortunately downgrading to 7.20.2 didn't work for me :( could be a separate issue.

My node is v14.17.3

[simllll]

Anything we can do to get this fixed? Super annoying bug for us, I need to downgrade npm every few days due to this issue.

[rijkvanzanten]

We temporarily updated our CI/CD pipeline to a locked outdated version of NPM to mitigate this:

      - name: Use Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '16.x'

      # See https://github.com/npm/cli/issues/3637
      - run: npm i -g npm@7.20.2

https://github.com/directus/directus/blob/607e32ea36c115a9d6cf820a26475ebbf9e6e9ed/.github/workflows/release.yml#L36-L42

For what it's worth, OSS - Sprint 36 says it's due September 07, so lets hope for the best 🤞🏻

[simllll]

Is there any status update to this issue? Will it be fixed? Is it not reproducable on your side? Or will this be the new intended behaviour? Thanks

all are actually in the same monorepo:

npm WARN
npm WARN Could not resolve dependency:
npm WARN @hokify/shared-components@"^1.2.9" from @hokify/pwa-core@1.2.10
npm WARN packages/pwa-core
npm WARN @hokify/pwa-core@1.2.6
npm WARN node_modules/@hokify/pwa-core
npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@hokify%2fuser-app - Not found
npm ERR! 404
npm ERR! 404 '@hokify/user-app@^1.2.10' is not in this registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)

[rijkvanzanten]

I'm wondering if the regression that caused this might be related to #3881

[simllll]

It seems it works without a package lock, as soon as there is one though, a second npm install fails .. this is such an annoying bug, please let me know what I can contribute to get this fixed.

[simllll]

Another thing that I discovered:
it seems if package-lock is "not" identical to the actual packaeg version the lookup fails.
Our monorepo setup increases the version of e.g. @hokify/user-app.. but let's the package-lock untouched.
Another install now fails, as it seems that the version in package-lock is looked for, the local version is higher though..and therefore it probabaly tries to look it up on npm..where it also doesn't find the package.
I guess a rule in this case shuold be: ignore the version number for local packages?

[simllll]

I found one workaround that seems to do it for now: we are using "file:../path/to/package" now instead for references to private packages. This is probably not a solution for everyone, but maybe it helps someone.

[rijkvanzanten]

For what it's worth, the same problem persists on 8+ (just tested on 8.1.0)

[eddyw]

omg, we wasted a lot of time yesterday on this (recently upgraded to npm 8)
If it's useful to anyone, what worked for us is to add the dependency manually to the package.json (of a workspace) file, then run npm i at the root.

[qinyang912]

'npx npm@7.20.2 install <local_package>' can temporarily solve this problem 😅

[SugarDarius]

Hello !

Tested on Windows with:
npm@8.1.2
node@16.3.1

Unfortunately the issue still persists :/

[jimfranke]

This issue still persists and is a major problem IMO when working with workspaces. Is this being looked in to?
Windows, npm@8.1.2, node 16.3.1

[bigtimebuddy]

Can we re-open this issue? This error still reproduces in 8.4.0.

To reproduce:

git clone https://github.com/bigtimebuddy/npm-bug-package-lock-only
cd npm-bug-package-lock-only
npm i npm@8.4.0 -g
npm test

Should throw the following error:

npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@tools%2fa - Not found
npm ERR! 404 
npm ERR! 404  '@tools/a@2.0.0' is not in this registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

Reference #3964

[nlf]

ok, i think this pull request will fix the problem for real. #4371

it passes the test in your reproduction repo at least @bigtimebuddy, thank you for that! it was really helpful for tracking this down.

[nlf]

closing. this fix will be out in our next release later this week

[manolakis]

Can we re-open this issue? The problem is happening again with node 18.14.0 and npm 9.3.1.

[ljharb]

@manolakis have you tried npm 9.5?

[manolakis]

ok, I did a try in modern versions and seems to be fixed in version 9.4.2+

[DesignByOnyx]

I wanted to report that npm@8.19.4 (released 6 days ago, March 1, 2023) fixes this issue. I was previously on 8.19.3 and the issue went away after upgrading. A couple notes:

• Thank you to the npm team for back porting the fix to version 8.x.
• If you cannot upgrade, you can update your package.json files directly. The format is "<package_name>": "*" where the version number is "*". If you do this, npm will recognize it as a local dependency. I tested this prior to upgrading.

[SergkeiM]

@manolakis have you tried npm 9.5?

node: v18.14.2
npm: 9.5.0

Doesn't work...

npm --no-git-tag-version --save -w packages/core version 2.0.0
@org/core
v2.0.0
npm ERR! code E404
npm ERR! 404 Not Found - GET https://npm.pkg.github.com/@org%2fcore - npm package "core" does not exist under owner "org"
npm ERR! 404 
npm ERR! 404  '@org/core@^1.6.0' is not in this registry.
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/sergkeim/.npm/_logs/2023-07-04T11_38_53_258Z-debug-0.log

[szszoke]

I am having this issue with npm 9.8.0 as well.

[davidvpe]

same here 9.5.1

[LoganArnett]

This is happening again on v10.2.0

[lukekarrys]

@LoganArnett can you open a new issue with your reproduction for this?

[gavenkoa]

node/16.14.0 with

# npm --version
8.3.1

has a problem. Upgrade to node/16.20.2 with

# npm --version
8.19.4

fixes the issue.

[DesignByOnyx]

FWIW - Node 16.x has reached the end of it's life and should not be used any more. Also npm can be upgraded independently of node by doing npm i -g npm@latest.

[philly-vanilly]

I am on latest node lts v20.10.0, 10.2.3 and this is still an issue. I suspect it is one with npm, not node though

[StanleyDharan]

No longer an issue on node v20.11.0 npm v10.2.4

[Anonyfox]

actually still a problem with node v22.4.1 and npm v10.8.2