External dependencies missing in projectGraph on Windows #19520

JonWallsten · 2023-10-10T05:52:05Z

Current Behavior

I'm currently investigating why module federation is broken in nx >= 16.8.1, #19121.
The root cause is that externalModules are missing in the projectGraph, so host/remote cannot share dependencies.

After a lot of digging it seems like the reason is that the WorkspaceContext is missing externalModules.
In nx@16.2.2 the external modules are added to the project graph by calling buildNpmPackageNodes.
In nx@16.8.1 something called getNxWorkspaceFilesFromContext is used instead. It's using a WorkspaceContext to retrieve the externalModules, but it's always empty.

Since WorkspaceContext is created by a binary I'm not sure if I can investigate this further.
But since this is only an issue on Windows and this is a native module I guess that it's not that far fetched that it could be the source of the issue.

projectGraph {
  nodes: {
    remote: { name: 'remote', type: 'app', data: [Object] },
    host: { name: 'host', type: 'app', data: [Object] }
  },
  externalNodes: {},
  dependencies: { remote: [], host: [] },
  version: '6.0'
}

Expected Behavior

WorkspaceContext should add npm dependencies as externalModules so it can be added to the projectGraph

GitHub Repo

https://github.com/JonWallsten/nx-repro/tree/dynamic

Steps to Reproduce

npm i
npm run build

Nx Report

Node   : 18.14.1
   OS     : win32-x64
   npm    : 9.3.1

   nx                 : 16.10.0
   @nx/js             : 16.10.0
   @nx/jest           : 16.10.0
   @nx/linter         : 16.10.0
   @nx/workspace      : 16.10.0
   @nx/angular        : 16.10.0
   @nx/cypress        : 16.10.0
   @nx/devkit         : 16.10.0
   @nx/eslint-plugin  : 16.10.0
   @nrwl/tao          : 16.10.0
   @nx/web            : 16.10.0
   @nx/webpack        : 16.10.0
   typescript         : 5.1.6

Failure Logs

No response

Package Manager Version

No response

Operating System

macOS
Linux
Windows
Other (Please specify)

Additional Information

No response

The text was updated successfully, but these errors were encountered:

JonWallsten · 2023-10-25T15:56:32Z

@AgentEnder Is there anything I can do to help with this?

JonWallsten · 2023-11-30T12:40:49Z

Just updated to nx@17.x.x and this is still an issue. So we're still stuck on 16.2.2. Is there any plan to make this work on Windows? I'm already offered to help, but I need support to get further.
Can anyone else assist me in this?
@FrozenPandaz / @juristr / @Cammisuli / @Coly010

JonWallsten · 2023-12-01T06:55:25Z

Finally figured out how to use --inspect-brk on the child process that creates the graph. So now I at least have a chance to figure this out:

I'm trying to find out why the externalNodes is always empty. I think that's the key. Any help appreciated.

JonWallsten · 2023-12-01T09:04:51Z

Okey, I finally found the issue! 🥳
I tried to find all the places where externalNodes are affected somehow. And found this getNodes function in npm-parser.ts (we're using npm).
It takes data: NpmLockFile as input. But I'm not using a lock-file.
So added package-lock=true to the .npmrc and built again, but it still didn't work.
I remember seeing .gitignore somewhere when I debugged.
So I removed package-lock.json from .gitignore and finally it worked.

But since I now have to push the package-lock.json to our repo, because it's no longer in the .gitignore file I can't get the build to work in Gitlab, because the lock file was prepared on windows.

I get this error:
Error: Cannot find module '@nx/nx-linux-x64-gnu'

So I guess we have two issue here:

It should work without a package-lock.json
package-lock.json should not be ignored when building the graph
But if 1 is fixed, 2 is not necessary to fix.

@AgentEnder

Cammisuli · 2023-12-01T11:54:30Z

Make sure the lock file is regenerated. Follow the instructions here:
https://nx.dev/recipes/troubleshooting/troubleshoot-nx-install-issues#how-to-fix

Basically the lock file should include all the optional platform deps, so that the single lock file can be used on multiple systems.

Otherwise, if you really want to keep the lock file in the gitignore, you could try adding it to the .nxignore file with !package-lock.json

JonWallsten · 2023-12-01T12:43:08Z

Make sure the lock file is regenerated. Follow the instructions here: https://nx.dev/recipes/troubleshooting/troubleshoot-nx-install-issues#how-to-fix

Basically the lock file should include all the optional platform deps, so that the single lock file can be used on multiple systems.

Otherwise, if you really want to keep the lock file in the gitignore, you could try adding it to the .nxignore file with !package-lock.json

We're using npm ci --cache .npm --prefer-offline to install.
And i can see this in the package-lock.json:

But I still get the same error. I've tried to clear the runner's cache as well.

Cammisuli · 2023-12-01T18:35:08Z

on the CI, can you try running find node_modules -name "*.node" | grep -E "(@nx|@nrwl)" and making sure that the linux binary shows up there?

For example, on my computer running the command on the nx repo, i get this result:

node_modules/.pnpm/@nrwl+nx-darwin-arm64@15.8.0/node_modules/@nrwl/nx-darwin-arm64/nx.darwin-arm64.node
node_modules/.pnpm/@nx+nx-darwin-arm64@17.1.1/node_modules/@nx/nx-darwin-arm64/nx.darwin-arm64.node

JonWallsten · 2023-12-04T07:08:50Z

@Cammisuli: Not sure what going on, I just won't work with the package-lock.json. I've tried all that I can think of. Removing the package-lock file is the only thing I can get to work. But then the issue is that I don't have a package-lock file when building. So I end up with the same problem as locally and the loop is closed!

npm install || true
npm WARN cleanup Failed to remove some directories [
npm WARN cleanup   [
npm WARN cleanup     '/builds/....../node_modules/esbuild',
npm WARN cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/builds/....../node_modules/esbuild/bin'] {
npm WARN cleanup       errno: -39,
npm WARN cleanup       code: 'ENOTEMPTY',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: '/builds/....../node_modules/esbuild/bin'
npm WARN cleanup     }
npm WARN cleanup   ],
npm WARN cleanup   [
npm WARN cleanup     '/builds/....../node_modules/@swc',
npm WARN cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/builds/....../node_modules/@swc'] {
npm WARN cleanup       errno: -39,
npm WARN cleanup       code: 'ENOTEMPTY',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: '/builds/....../node_modules/@swc'
npm WARN cleanup     }
npm WARN cleanup   ],
npm WARN cleanup   [
npm WARN cleanup     '/builds/....../node_modules',
npm WARN cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/builds/....../node_modules/@swc'] {
npm WARN cleanup       errno: -39,
npm WARN cleanup       code: 'ENOTEMPTY',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: '/builds/....../node_modules/@swc'
npm WARN cleanup     }
npm WARN cleanup   ]
npm WARN cleanup ]
npm ERR! code 1
npm ERR! path /builds/....../node_modules/nx
npm ERR! command failed
npm ERR! command sh -c node ./bin/post-install
npm ERR! /builds/....../node_modules/nx/src/native/index.js:244
npm ERR!     throw loadError
npm ERR!     ^
npm ERR! 
npm ERR! Error: Cannot find module '@nx/nx-linux-x64-gnu'
npm ERR! Require stack:
npm ERR! - /builds/....../node_modules/nx/src/native/index.js
npm ERR! - /builds/....../node_modules/nx/src/hasher/task-hasher.js
npm ERR! - /builds/....../node_modules/nx/src/tasks-runner/run-command.js
npm ERR! - /builds/....../node_modules/nx/src/nx-cloud/utilities/get-cloud-options.js
npm ERR! - /builds/....../node_modules/nx/bin/post-install.js
npm ERR!     at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
npm ERR!     at Module._load (node:internal/modules/cjs/loader:922:27)
npm ERR!     at Module.require (node:internal/modules/cjs/loader:11[43](https://gitlab.scania.com/....../-/jobs/32749823#L43):19)
npm ERR!     at require (node:internal/modules/cjs/helpers:119:18)
npm ERR!     at Object.<anonymous> (/builds/....../node_modules/nx/src/native/index.js:184:31)
npm ERR!     at Module._compile (node:internal/modules/cjs/loader:12[56](https://gitlab.scania.com/....../-/jobs/32749823#L56):14)
npm ERR!     at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
npm ERR!     at Module.load (node:internal/modules/cjs/loader:1119:32)
npm ERR!     at Module._load (node:internal/modules/cjs/loader:9[60](https://gitlab.scania.com/....../-/jobs/32749823#L60):12)
npm ERR!     at Module.require (node:internal/modules/cjs/loader:1143:19) {
npm ERR!   code: 'MODULE_NOT_FOUND',
npm ERR!   requireStack: [
npm ERR!     '/builds/....../node_modules/nx/src/native/index.js',
npm ERR!     '/builds/....../node_modules/nx/src/hasher/task-hasher.js',
npm ERR!     '/builds/....../node_modules/nx/src/tasks-runner/run-command.js',
npm ERR!     '/builds/....../node_modules/nx/src/nx-cloud/utilities/get-cloud-options.js',
npm ERR!     '/builds/....../node_modules/nx/bin/post-install.js'
npm ERR!   ]
npm ERR! }
npm ERR! 
npm ERR! Node.js v18.18.2
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-12-04T07_03_29_[79](https://gitlab.xxxxxxxxx.com/....../-/jobs/32749823#L79)6Z-debug-0.log
$ find node_modules -name "*.node" | grep -E "(@nx|@nrwl)"
Cleaning up project directory and file based variables

Installing it manually works:

$ npm install @nx/nx-linux-x64-gnu || true
added 1930 packages, and audited 1931 packages in 15s
270 packages are looking for funding
  run `npm fund` for details
5 moderate severity vulnerabilities
To address issues that do not require attention, run:
  npm audit fix
To address all issues (including breaking changes), run:
  npm audit fix --force
Run `npm audit` for details.
$ find node_modules -name "*.node" | grep -E "(@nx|@nrwl)"
node_modules/@nx/nx-linux-x64-gnu/nx.linux-x64-gnu.node
Cleaning up project directory and file based variables
00:00
Job succeeded

It only worked until I tried to build:

>  NX   Cannot find module @rollup/rollup-linux-x64-gnu. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.

This ended up working (but is of course a bad solution):

 - npm i @rollup/rollup-linux-x64-gnu @nx/nx-linux-x64-gnu --cache .npm --prefer-offline

We need to use the package-lock.json file since we use Gitlabs cache feature for the npm modules. So we can't rely on creating the package-lock.json at install, since the command is nog run when cache is used.

Cammisuli · 2023-12-04T11:45:12Z

Are the optional dependencies listed in the package-lock.json file? The fact that you can't find rollup's native binary either is suspicious of the package-lock. All the optional dependencies should be listed in the package-lock. For example this is how the nx ones look:

    "node_modules/@nx/nx-darwin-arm64": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-darwin-arm64/-/nx-darwin-arm64-17.2.2.tgz",
      "integrity": "sha512-8dPPmX8KQT/HtizaDmz/+TIAytAwuC0Mvx/UqhiFNL5l0o5HnK+AfNEe2GWk9kfZjg22nmMfBIBDIxl6odNVBQ==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "darwin"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-darwin-x64": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-darwin-x64/-/nx-darwin-x64-17.2.2.tgz",
      "integrity": "sha512-hJaqyF+wAdFznXv2WQ+qTkCX5+Dr4cCtA9M67/Rln70bUkiQ0upQAacI0BUrHLZOKAS6OKW2V37V1xgCLk+NHw==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "darwin"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-freebsd-x64": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-freebsd-x64/-/nx-freebsd-x64-17.2.2.tgz",
      "integrity": "sha512-CxbtqfL33MHqaEXKNjgrZCf9Sot13KmZvkbET/DOQGJVX6yD1kKV8yTN9KfoeUwf1UorCCyTy5NdyFKq0nkIOQ==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "freebsd"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-arm-gnueabihf": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-arm-gnueabihf/-/nx-linux-arm-gnueabihf-17.2.2.tgz",
      "integrity": "sha512-r5kAqC3c6kIXUfDK27zQ3LsZO/tsDzmiNilxc00DmUIzLPBLt3qlXbctD5jxE80xVkgNSCEZtSFIgTpYdymnAQ==",
      "cpu": [
        "arm"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-arm64-gnu": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-arm64-gnu/-/nx-linux-arm64-gnu-17.2.2.tgz",
      "integrity": "sha512-EowQSciBfUg82jwHruFhCnY6/raWoo8FsGPaTNaKPcXtckYCi1hj9XUgHpeG06Z5lgPvO/1pFIUSSkIc3vefQA==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-arm64-musl": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-arm64-musl/-/nx-linux-arm64-musl-17.2.2.tgz",
      "integrity": "sha512-MQgXf5u12EyCINLiuntjOE5sR2Vp4CjHMrXt8aT8ZRURUc6PBkubDjslVhVJ7Xmot38i5Fh/ngrdXzFrQETAUw==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-x64-gnu": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-x64-gnu/-/nx-linux-x64-gnu-17.2.2.tgz",
      "integrity": "sha512-qPB42G0xLlwbpC40r8RxgS+xsKoURpkBmcwhXSo+5iBnfOvW2qk4ox+0h+l+mTT8RB30H/7XNEEXergu2Px1Zw==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-x64-musl": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-x64-musl/-/nx-linux-x64-musl-17.2.2.tgz",
      "integrity": "sha512-AnD7rtxyU6LIF0ewCO0gmvrQuJPSPZPXAzrUn95YP+nh357bfweOPV6uyHYn/Nqax1f3FPFe7Ervo48TAzunug==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-win32-arm64-msvc": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-win32-arm64-msvc/-/nx-win32-arm64-msvc-17.2.2.tgz",
      "integrity": "sha512-xNvdlfGnTBx6FygCR+AenduVWATh4HtaNibFM7ELdsTM4posYFYk7jpNXf5dFtSANJ+Is2vWAEqjWx1Th0/ZjQ==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "win32"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-win32-x64-msvc": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-win32-x64-msvc/-/nx-win32-x64-msvc-17.2.2.tgz",
      "integrity": "sha512-2pULDtVSGJUYiFKFJTASaOVpSF9AkjOvVgkRTV92Cx2ykT/9wwg7EfY/WZmLxfAkYFjiQyD9s4Wq06yp/rgd4w==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "win32"
      ],
      "engines": {
        "node": ">= 10"
      }
    },

Other package's optional native binaries should be around the same area too.

Did you try deleting node_modules + the package-lock and doing npm i again? Did you try updating your npm version?

sagarbasavaraj · 2024-02-28T15:36:44Z

I am having the same problem with npm but it worked with yarn package manager as we have yarn.lock file.

gabynevada · 2024-03-19T18:25:49Z

I switched our package manager to bun and having the same issue as it uses the bun/lockb file.

matheo · 2024-03-26T18:36:25Z

I'm using pnpm and I added pnpm-lock.yaml to .nxignore to prevent nx format issues but then my dep-graph was messed up without external dependencies 🤔

JonWallsten added the type: bug label Oct 10, 2023

AgentEnder added the scope: core core nx functionality label Oct 12, 2023

AgentEnder self-assigned this Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External dependencies missing in projectGraph on Windows #19520

External dependencies missing in projectGraph on Windows #19520

JonWallsten commented Oct 10, 2023 •

edited

Loading

JonWallsten commented Oct 25, 2023

JonWallsten commented Nov 30, 2023 •

edited

Loading

JonWallsten commented Dec 1, 2023

JonWallsten commented Dec 1, 2023 •

edited

Loading

Cammisuli commented Dec 1, 2023 •

edited

Loading

JonWallsten commented Dec 1, 2023

Cammisuli commented Dec 1, 2023

JonWallsten commented Dec 4, 2023 •

edited

Loading

Cammisuli commented Dec 4, 2023

sagarbasavaraj commented Feb 28, 2024

gabynevada commented Mar 19, 2024

matheo commented Mar 26, 2024

External dependencies missing in projectGraph on Windows #19520

External dependencies missing in projectGraph on Windows #19520

Comments

JonWallsten commented Oct 10, 2023 • edited Loading

Current Behavior

Expected Behavior

GitHub Repo

Steps to Reproduce

Nx Report

Failure Logs

Package Manager Version

Operating System

Additional Information

JonWallsten commented Oct 25, 2023

JonWallsten commented Nov 30, 2023 • edited Loading

JonWallsten commented Dec 1, 2023

JonWallsten commented Dec 1, 2023 • edited Loading

Cammisuli commented Dec 1, 2023 • edited Loading

JonWallsten commented Dec 1, 2023

Cammisuli commented Dec 1, 2023

JonWallsten commented Dec 4, 2023 • edited Loading

Cammisuli commented Dec 4, 2023

sagarbasavaraj commented Feb 28, 2024

gabynevada commented Mar 19, 2024

matheo commented Mar 26, 2024

JonWallsten commented Oct 10, 2023 •

edited

Loading

JonWallsten commented Nov 30, 2023 •

edited

Loading

JonWallsten commented Dec 1, 2023 •

edited

Loading

Cammisuli commented Dec 1, 2023 •

edited

Loading

JonWallsten commented Dec 4, 2023 •

edited

Loading