Dependencies referenced from a GitHub repository can't be installed using npm 8

[fluiddot]

Description

Related to #36041 (comment).

Installing dependencies via npm install using npm 8 fails with the following error:

npm ERR! Tracker "idealTree:inflate:" already exists

This makes the upgrade of the package-lock.json file to format version 2 fail. However, I managed to fix it by doing the following steps:

1. [Using npm 6] Remove the dependencies referenced from a GitHub repository, this can be done by applying the following patch.

Click here to display patch

diff --git a/packages/react-native-editor/package.json b/packages/react-native-editor/package.json
index 74f1841dad22b6ccf4bba8da696718624d05c8b7..a31da4e8020b752942328ebeae586e196fc52e17 100644
--- a/packages/react-native-editor/package.json
+++ b/packages/react-native-editor/package.json
@@ -32,7 +32,6 @@
 		"@babel/runtime": "^7.16.0",
 		"@react-native-clipboard/clipboard": "1.9.0",
 		"@react-native-community/blur": "3.6.0",
-		"@react-native-community/slider": "https://raw.githubusercontent.com/wordpress-mobile/react-native-slider/v3.0.2-wp-2/react-native-community-slider-3.0.2-wp-2.tgz",
 		"@react-native-masked-view/masked-view": "0.2.6",
 		"@react-navigation/core": "5.12.0",
 		"@react-navigation/native": "5.7.0",
@@ -54,25 +53,16 @@
 		"fast-average-color": "^4.3.0",
 		"gettext-parser": "^1.3.1",
 		"jed": "^1.1.1",
-		"jsdom-jscore-rn": "git+https://github.com/iamcco/jsdom-jscore-rn.git#a562f3d57c27c13e5bfc8cf82d496e69a3ba2800",
 		"node-fetch": "^2.6.0",
 		"react-native": "0.66.2",
-		"react-native-gesture-handler": "https://raw.githubusercontent.com/wordpress-mobile/react-native-gesture-handler/2.2.0-wp-4/react-native-gesture-handler-2.2.0-wp-4.tgz",
 		"react-native-get-random-values": "1.4.0",
-		"react-native-hr": "git+https://github.com/Riglerr/react-native-hr.git#2d01a5cf77212d100e8b99e0310cce5234f977b3",
-		"react-native-hsv-color-picker": "https://raw.githubusercontent.com/wordpress-mobile/react-native-hsv-color-picker/v1.0.1-wp-2/react-native-hsv-color-picker-1.0.1-wp-2.tgz",
-		"react-native-keyboard-aware-scroll-view": "https://raw.githubusercontent.com/wordpress-mobile/react-native-keyboard-aware-scroll-view/v0.8.8-wp-1/react-native-keyboard-aware-scroll-view-0.8.8-wp-1.tgz",
-		"react-native-linear-gradient": "https://raw.githubusercontent.com/wordpress-mobile/react-native-linear-gradient/v2.5.6-wp-2/react-native-linear-gradient-2.5.6-wp-2.tgz",
 		"react-native-modal": "^11.10.0",
-		"react-native-prompt-android": "https://raw.githubusercontent.com/wordpress-mobile/react-native-prompt-android/v1.0.0-wp-2/react-native-prompt-android-1.0.0-wp-2.tgz",
-		"react-native-reanimated": "https://raw.githubusercontent.com/wordpress-mobile/react-native-reanimated/2.4.1-wp-1/react-native-reanimated-2.4.1-wp-1.tgz",
 		"react-native-safe-area": "^0.5.0",
 		"react-native-safe-area-context": "3.2.0",
 		"react-native-sass-transformer": "^1.1.1",
 		"react-native-screens": "2.9.0",
 		"react-native-svg": "9.13.6",
 		"react-native-url-polyfill": "^1.1.2",
-		"react-native-video": "https://raw.githubusercontent.com/wordpress-mobile/react-native-video/5.2.0-wp-2/react-native-video-5.2.0-wp-2.tgz",
 		"react-native-webview": "11.6.2"
 	},
 	"publishConfig": {

2. Run npm install.
3. Change to npm 8 by applying the following patch:

Click here to display patch

diff --git a/.nvmrc b/.nvmrc
index 8351c19397..b6a7d89c68 100644
--- a/.nvmrc
+++ b/.nvmrc
@@ -1 +1 @@
-14
+16
diff --git a/package.json b/package.json
index e312274da1..40781e2952 100755
--- a/package.json
+++ b/package.json
@@ -15,8 +15,8 @@
 		"url": "https://github.com/WordPress/gutenberg/issues"
 	},
 	"engines": {
-		"node": ">=10.0.0",
-		"npm": ">=6.9.0 <7"
+		"node": ">=12",
+		"npm": ">=8"
 	},
 	"config": {
 		"IS_GUTENBERG_PLUGIN": true

4. Run nvm use.
5. Run npm install.
6. Revert the package removal done in step 1.
7. Run npm install again.

However, although this fixes the dependencies installation, the CI jobs are failing with the following error:

Extracted from this CI job:

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@github.com/iamcco/jsdom-jscore-rn.git
npm ERR! 
npm ERR! git@github.com: Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 12[8](https://github.com/WordPress/gutenberg/runs/6115653184?check_suite_focus=true#step:4:8)

Looks like the CI instances can't check out the GitHub repositories due to permissions issues. I identified that this is only happening for the following dependencies that use the git+https protocol:

• gutenberg/packages/react-native-editor/package.json

  Line 57 in e7fbb88

  "jsdom-jscore-rn": "git+https://github.com/iamcco/jsdom-jscore-rn.git#a562f3d57c27c13e5bfc8cf82d496e69a3ba2800",

• gutenberg/packages/react-native-editor/package.json

  Line 62 in e7fbb88

  "react-native-hr": "git+https://github.com/Riglerr/react-native-hr.git#2d01a5cf77212d100e8b99e0310cce5234f977b3",

For testing purposes, I removed these packages to see if the rest of the forked repositories dependencies would work (reference). But I spotted a different error related to linting and TypeScript. I'm not sure if this error is caused by any of React Native dependencies or other reasons not related to RN 🤔 .

Step-by-step reproduction instructions

1. Apply the following patch:

diff --git a/.nvmrc b/.nvmrc
index 8351c19397..b6a7d89c68 100644
--- a/.nvmrc
+++ b/.nvmrc
@@ -1 +1 @@
-14
+16
diff --git a/package.json b/package.json
index e312274da1..40781e2952 100755
--- a/package.json
+++ b/package.json
@@ -15,8 +15,8 @@
 		"url": "https://github.com/WordPress/gutenberg/issues"
 	},
 	"engines": {
-		"node": ">=10.0.0",
-		"npm": ">=6.9.0 <7"
+		"node": ">=12",
+		"npm": ">=8"
 	},
 	"config": {
 		"IS_GUTENBERG_PLUGIN": true

2. Run npm install.
3. Observe that the installation fails with the error:

npm ERR! Tracker "idealTree:inflate:" already exists

Expected behaviour

All dependencies should be installed with npm 8.

Actual behaviour

Some dependencies can't be installed with npm 8.

Screenshots or screen recording (optional)

N/A

WordPress information

• WordPress version: N/A
• Gutenberg version: N/A
• Are all plugins except Gutenberg deactivated? N/A
• Are you using a default theme (e.g. Twenty Twenty-One)? N/A

Device information

• Device: N/A
• Operating system: N/A
• WordPress app version: N/A

[fluiddot]

It would be great to focus on this sooner than later because, by the end of May 2022, Node.js 12 will be deprecated. This means that only version 14 will be the one that still supports npm 6, which will be deprecated in May 2023.

[geriux]

So I took some time to investigate this as I wanted to try Appium 2 and it requires npm 8.

It looks like the npm ERR! Tracker "idealTree:inflate:" already exists issue was related to this bug, which was addressed and merged.

I think by the time this effort was initiated, it didn't have the updated version of @npmcli/arborist in 8.1.1.

I just tried the 8.19.3 version and it updated the package-lock.json successfully without changing the GitHub/Repo URLs 🎉

We do need to change the URLs that point to the GitHub repo directly for jsdom-jscore-rn and react-native-hr, for the first one, there's actually a release with the needed commit so we can update it to 0.1.8, for the other one I created a fork and a tarball to solve this issue while we work on getting rid of that package.

There's an issue with the generation of the React Native bundle that will be solved with the 0.69.4 upgrade.

I created a draft PR with the changes here, there are two tests that failed Compressed Size and Performance Tests due to engine Not compatible with your version of node/npm: gutenberg@14.6.0-rc.1 I'm sure there's a change needed somewhere that I'm not aware of 😅 .

Hopefully, the React Native Upgrade will land soon so we can finally update Node and npm.

[fluiddot]

Thanks for the update and the information @geriux 🙇 !

I just tried the 8.19.3 version and it updated the package-lock.json successfully without changing the GitHub/Repo URLs 🎉

Great news that with the new NPM version this is no longer an issue.

We do need to change the URLs that point to the GitHub repo directly for jsdom-jscore-rn and react-native-hr, for the first one, there's actually a release with the needed commit so we can update it to 0.1.8, for the other one I created a fork and a tarball to solve this issue while we work on getting rid of that package.

To be honest, I think we could remove the react-native-hr depedency by creating our custom implementation of the <hr> component, as it seems to be relatively easy to implement.

There's an issue with the generation of the React Native bundle that will be solved with the 0.69.4 #43485.

🎊

I created a draft PR with the changes #45941, there are two tests that failed Compressed Size and Performance Tests due to engine Not compatible with your version of node/npm: gutenberg@14.6.0-rc.1 I'm sure there's a change needed somewhere that I'm not aware of 😅 .

Interesting, probably we need to change something else. I checked the original PR where we tried to upgrade the NPM version, but couldn't find clues related to this issue.

[geriux]

To be honest, I think we could remove the react-native-hr depedency by creating our custom implementation of the <hr> component, as it seems to be relatively easy to implement.

I agree, we should do that when we can.

For now, I've created a PR to update those two packages to unblock upgrading to NPM 8, now that the React Native Upgrade is merged. I've assigned it to you as a reviewer 😃

[fluiddot]

Using changes from #46422, I tried installing the dependencies using npm 8.19.3 (node version 14.20.1) and I confirm the error described in this issue is no longer happening. However, installing dependencies with npm 8 still fails although due to a different error:

npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR!
npm ERR! While resolving: @storybook/addon-knobs@6.2.9
npm ERR! Found: react@18.2.0
npm ERR! node_modules/react
npm ERR!   dev react@"18.2.0" from the root project
npm ERR!   peer react@">=16.3.0" from @emotion/core@10.1.1
npm ERR!   node_modules/@emotion/core
npm ERR!     peer @emotion/core@"^10.0.28" from @emotion/styled-base@10.0.31
npm ERR!     node_modules/@emotion/styled-base
npm ERR!       @emotion/styled-base@"^10.0.27" from @emotion/styled@10.0.27
npm ERR!       node_modules/@storybook/addon-knobs/node_modules/@emotion/styled
npm ERR!         @emotion/styled@"^10.0.27" from @storybook/theming@6.2.9
npm ERR!         node_modules/@storybook/addon-knobs/node_modules/@storybook/theming
npm ERR!     peer @emotion/core@"^10.0.27" from @emotion/styled@10.0.27
npm ERR!     node_modules/@storybook/addon-knobs/node_modules/@emotion/styled
npm ERR!       @emotion/styled@"^10.0.27" from @storybook/theming@6.2.9
npm ERR!       node_modules/@storybook/addon-knobs/node_modules/@storybook/theming
npm ERR!         @storybook/theming@"6.2.9" from @storybook/addon-knobs@6.2.9
npm ERR!         node_modules/@storybook/addon-knobs
npm ERR!         3 more (@storybook/addons, @storybook/api, @storybook/components)
npm ERR!     3 more (@storybook/theming, emotion-theming, react-select)
npm ERR!   91 more (@emotion/primitives-core, @emotion/react, ...)
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peerOptional react@"^16.8.0 || ^17.0.0" from @storybook/addon-knobs@6.2.9
npm ERR! node_modules/@storybook/addon-knobs
npm ERR!   dev @storybook/addon-knobs@"6.2.9" from the root project
npm ERR!
npm ERR! Conflicting peer dependency: react@17.0.2
npm ERR! node_modules/react
npm ERR!   peerOptional react@"^16.8.0 || ^17.0.0" from @storybook/addon-knobs@6.2.9
npm ERR!   node_modules/@storybook/addon-knobs
npm ERR!     dev @storybook/addon-knobs@"6.2.9" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

I'm going to close the issue and open a new one with this information.

UPDATE: Follow-up of the error in #46443.

[geriux]

However, installing dependencies with npm 8 still fails although due to a different error:

Yeah that didn't happen a few weeks ago when I tried 😅 , I think it's related to the React 18 upgrade. Thanks for creating a follow-up ticket for that!