[core] Deduplicate packages

[merceyz]

• I have followed (at least) the PR section of the contributing guide.

• Added yarn deduplicate which automatically deduplicates packages using yarn-deduplicate then runs yarn install

• Made the CI fail if there are duplicated packages

Output:

$ yarn upgrade @babel/core && node scripts/postinstall.js
yarn upgrade v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents@1.2.9: The platform "win32" is incompatible with this module.
info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > @material-ui/pickers@3.1.1" has unmet peer dependency "@date-io/core@^1.3.6".
warning " > jss-rtl@0.2.3" has unmet peer dependency "jss@^9.0.0".
[4/4] Rebuilding all packages...
success Saved lockfile.
success Saved 9 new dependencies.
info Direct dependencies
└─ @babel/core@7.5.4
info All dependencies
├─ @babel/core@7.5.4
├─ @babel/helpers@7.5.4
├─ @babel/highlight@7.5.0
├─ ansi-styles@3.2.1
├─ chalk@2.4.2
├─ color-convert@1.9.3
├─ color-name@1.1.3
├─ js-tokens@4.0.0
└─ supports-color@5.5.0
Done in 16.74s.
22 duplicated package(s) found
  Package "@babel/core" wants ^7.1.6 and could get 7.5.4, but got 7.4.4
  Package "@babel/core" wants ^7.4.4 and could get 7.5.4, but got 7.4.4
  Package "@babel/generator" wants ^7.1.2 and could get 7.5.0, but got 7.4.4
  Package "@babel/generator" wants ^7.4.0 and could get 7.5.0, but got 7.4.4
  Package "@babel/helpers" wants ^7.1.2 and could get 7.5.4, but got 7.4.4
  Package "@babel/parser" wants ^7.0.0 and could get 7.5.0, but got 7.4.4
  Package "@babel/parser" wants ^7.1.2 and could get 7.5.0, but got 7.4.4
  Package "@babel/parser" wants ^7.1.6 and could get 7.5.0, but got 7.4.4
  Package "@babel/parser" wants ^7.4.3 and could get 7.5.0, but got 7.4.4
  Package "@babel/traverse" wants ^7.0.0 and could get 7.5.0, but got 7.4.4
  Package "@babel/traverse" wants ^7.1.0 and could get 7.5.0, but got 7.4.4
  Package "@babel/traverse" wants ^7.4.3 and could get 7.5.0, but got 7.4.4
  Package "@babel/types" wants ^7.1.2 and could get 7.5.0, but got 7.4.4
  Package "@babel/types" wants ^7.2.0 and could get 7.5.0, but got 7.4.4
  Package "@babel/types" wants ^7.3.0 and could get 7.5.0, but got 7.4.4
  Package "@babel/types" wants ^7.4.0 and could get 7.5.0, but got 7.4.4
  Package "ms" wants ^2.0.0 and could get 2.1.2, but got 2.1.1
  Package "resolve" wants ^1.10.0 and could get 1.11.1, but got 1.10.1
  Package "resolve" wants ^1.10.1 and could get 1.11.1, but got 1.10.1
  Package "resolve" wants ^1.4.0 and could get 1.11.1, but got 1.10.1
  Package "resolve" wants ^1.5.0 and could get 1.11.1, but got 1.10.1
  Package "resolve" wants ^1.8.1 and could get 1.11.1, but got 1.10.1
Deduplicating package(s)
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents@1.2.9: The platform "win32" is incompatible with this module.
info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > @material-ui/pickers@3.1.1" has unmet peer dependency "@date-io/core@^1.3.6".
warning " > jss-rtl@0.2.3" has unmet peer dependency "jss@^9.0.0".
[4/4] Building fresh packages...

$ node scripts/postinstall.js
Done in 12.57s.

[mui-pr-bot]

No bundle size changes comparing 42f78c6...e51c925

Generated by 🚫 dangerJS against e51c925

[eps1lon]

Can't lerna sync those?

Updating dependencies should probably be handled by a bot. Though it should only run weekly given the amount of packages in the workspace.

[merceyz]

Can't lerna sync those?

As far as I can tell, no.

Updating dependencies should probably be handled by a bot. Though it should only run weekly given the amount of packages in the workspace.

Probably, but that isn't the goal of this PR.
I didn't update(yarn upgrade) anything

[merceyz]

@eps1lon I've added it into the CI task and moved it away from postinstall.

To deduplicate packages one can now use yarn deduplicate.

Example of a CI failure where there was duplicated packages: https://circleci.com/gh/mui-org/material-ui/106720?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

updated

[eps1lon]

Isn't it simpler to run yarn yarn-deduplicate in CI and check if changes occurred? Basically just move it between install and "Should not have any git not staged".

I don't see anything meaningful being done in scripts/deduplicate

[merceyz]

Only benefit it provides locally is that it runs yarn install for you while on the CI it doesn't write to the disk and provides a more useful error message.

[oliviertassinari]

Both approaches sound OK, we had this duplication problems surface twice in the past on dependencies upgrade PR, great to see it fixed systematically. @eps1lon, your call.

[eps1lon]

Makes more sense to have all steps in a single script file. Nice job!