This repository has been archived by the owner on Apr 1, 2022. It is now read-only.
Add support for Yarn v2 projects #244
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
|
|
||
| * Overview | ||
| Yarn is a buildtool primarily used for building and managing javascript projects. It's functionally a superset of the =npm= cli. | ||
|
|
||
| Yarn uses the same package manifest file as =npm= -- =package.json= -- but uses a novel lockfile format to pin dependencies, saved as =yarn.lock=. | ||
|
|
||
| For dependency analysis, we focus exclusively on the lockfile. | ||
|
|
||
| * Lockfile (=yarn.lock=) | ||
| As part of the update from Yarn v1 to Yarn v2, some major changes were made to the lockfile. Most notably: | ||
|
|
||
| + The lockfile [[https://dev.to/arcanis/introducing-yarn-2-4eh1#new-lockfile-format][is now real yaml]]. Yarn v1 used an almost-but-not-quite pseudo-yaml format. | ||
| + While Yarn v1's lockfile contained information only about dependencies of user projects, Yarn v2's lockfile is much more information-rich. It contains information about first-party user projects ("workspaces"), the version ranges specified in =package.json= for dependencies ("descriptors"), and the resolved version for each dependency ("locators") | ||
|
|
||
| ** Concepts | ||
| *** Workspaces | ||
| Workspaces are first-party package directories (directories that contain =package.json=). Workspaces are always available locally on disk, and are specified by a relative reference to a directory (e.g., =.= or =./foo/bar= or =../baz=) | ||
|
|
||
| A yarn project can have several workspaces, where workspaces may (but *are not required to*) depend on each other in a DAG. This is similar to "multi-module projects" in other buildtools like maven or gomodules. | ||
|
|
||
| Every yarn project will contain at least one workspace. | ||
|
|
||
| *** Locators | ||
| /Somewhat/ similar to fossa locators, a yarn locator is an unambiguous reference to a specific version of a package and where to find it. | ||
|
|
||
| Locators have three components: | ||
| + Package scope (optional) -- like =@babel= -- a scope on npm. | ||
| + Package name -- like =underscore=. | ||
| + Package reference -- which can vary in shape depending on where the package is coming from. For example, this could be a pointer to specific package version on the npm registry, a pointer to a git repo at a specific commit, or a link to a tarball. | ||
|
|
||
| Package scope and name in a locator, for the purposes of dependency resolution, *are unused*. Only the package reference matters. | ||
|
|
||
| Yarn supports a handful of reference types by default, and plugins can be added to support new reference types. See the =Resolvers= section below. | ||
|
|
||
| *** Descriptors | ||
| Descriptors are similar to locators, but may point to a /range/ of package versions. For the purposes of dependency analysis, we don't care much about the shape and content of descriptors. | ||
|
|
||
| Descriptors have three components: | ||
| + Package scope (optional) -- like =@babel= -- a scope on npm | ||
| + Package name -- like =underscore= | ||
| + Package range -- which can vary in shape depending on where the package is coming from. For example, this could be a semver for a package on the npm registry, a pointer to a git repo on a branch, or a link to a tarball | ||
|
|
||
| All locators are valid descriptors; not all descriptors are valid locators. | ||
|
|
||
| *** Resolvers | ||
| Plugins are the yarn v2 mechanism used to add support for, among other things, new types of locators. | ||
|
|
||
| A plugin can export zero or more "Resolvers", each of which can add support for new types of locator. Yarn itself implements support for "built-in" locator types (npm dependencies, git dependencies, etc) as resolvers in bundled plugins. | ||
|
|
||
| For dependency analysis, we support locators produced by [[https://github.com/yarnpkg/berry/blob/8afcaa2a954e196d6cd997f8ba506f776df83b1f/packages/yarnpkg-cli/package.json#L68-L82][all of the built-in plugins]]. | ||
| ** Format | ||
| #+BEGIN_SRC yaml | ||
| # This file is generated by running "yarn install" inside your project. | ||
| # Manual changes might be lost - proceed with caution! | ||
|
|
||
| __metadata: | ||
| version: 4 | ||
| cacheKey: 7 | ||
|
|
||
| "bar@workspace:bar": | ||
| version: 0.0.0-use.local | ||
| resolution: "bar@workspace:bar" | ||
| dependencies: | ||
| underscore: 1.13.1 | ||
| languageName: unknown | ||
| linkType: soft | ||
|
|
||
| "foo@workspace:foo": | ||
| version: 0.0.0-use.local | ||
| resolution: "foo@workspace:foo" | ||
| dependencies: | ||
| underscore: ^1.13.0 | ||
| languageName: unknown | ||
| linkType: soft | ||
|
|
||
| "quux@workspace:quux": | ||
| version: 0.0.0-use.local | ||
| resolution: "quux@workspace:quux" | ||
| dependencies: | ||
| underscore: "jashkenas/underscore#tag=1.13.1" | ||
| languageName: unknown | ||
| linkType: soft | ||
|
|
||
| "toplevel@workspace:.": | ||
| version: 0.0.0-use.local | ||
| resolution: "toplevel@workspace:." | ||
| languageName: unknown | ||
| linkType: soft | ||
|
|
||
| "underscore@jashkenas/underscore#tag=1.13.1": | ||
| version: 1.13.1 | ||
| resolution: "underscore@https://github.com/jashkenas/underscore.git#commit=cbb48b79fc1205aa04feb03dbc055cdd28a12652" | ||
| checksum: 560609fdb4ba2c30e79db95ea37269982d1a2788d49b78f0de4f391da711bc2495d5fbddd6d24e7716fccf69959e445916af83eb5de1ad137b215777e2d32e4d | ||
| languageName: node | ||
| linkType: hard | ||
|
|
||
| "underscore@npm:1.13.1, underscore@npm:^1.13.0": | ||
| version: 1.13.1 | ||
| resolution: "underscore@npm:1.13.1" | ||
| checksum: 19527b2db3d34f783c3f2db9716a2c1221fef2958866925545697c46f430f59d1b384b8105cc7e7c809bdf0dc9075f2bfff90b8fb270b9d3a6c58347de2dd79d | ||
| languageName: node | ||
| linkType: hard | ||
|
|
||
| #+END_SRC | ||
|
|
||
| Ignoring the =__metadata= field, the yarn lockfile is a mapping from =a comma-separated list of descriptors= to a =package description=. | ||
|
|
||
| *** Package description fields | ||
|
|
||
| Of a package's fields, we only care about =resolution= and =dependencies= | ||
|
|
||
| **** =resolution= | ||
| The locator used for this package | ||
|
|
||
| **** =dependencies= | ||
| An optional field containing =package: descriptor-range= mappings for each dependency of the package. *This includes dev dependencies* if they were included when running =yarn install=. | ||
|
|
||
| This field is copied identically from a package's =dependencies= and =devDependencies= fields in =package.json= | ||
|
|
||
| Full dependency descriptors can be reconstructed by joining key-value pairs on =@=: =underscore: ^1.13.0= is =underscore@^1.13.0=. Each dependency's descriptor is a key for a package at the top level of the yarn lockfile | ||
|
|
||
| #+BEGIN_QUOTE | ||
| *NOTE*: a fun note about dependency descriptors | ||
|
|
||
| A keen eye may notice that in the lockfile above, some descriptor keys contain =npm:= at the top-level. For example, there's =underscore@npm:1.13.1= -- but that descriptor isn't used anywhere as a dependency. The closest is =underscore@1.13.1=, a dependency of the =bar= workspace. | ||
|
|
||
| In an interesting design decision, yarn makes the default resolver for packages configurable. When a user provides a raw version (e.g., =1.13.1=) or semver (=^1.13.1=) for a dependency in =package.json=, a "default protocol" string is prepended to the descriptor range. This option [[https://next.yarnpkg.com/configuration/yarnrc#defaultProtocol][is configured]] as =defaultProtocol=, which defaults to =npm:=. | ||
|
|
||
| As a workaround, when using a descriptor =name@range= to look up a package in the lockfile, we must also try =name@npm:range= | ||
| #+END_QUOTE | ||
|
|
||
| *** Lockfile sources | ||
| The above lockfile was generated from the following files | ||
|
|
||
| =package.json= | ||
| #+BEGIN_SRC json | ||
| { | ||
| "name": "toplevel", | ||
| "private": true, | ||
| "workspaces": [ | ||
| "foo", | ||
| "bar", | ||
| "quux" | ||
| ] | ||
| } | ||
| #+END_SRC | ||
|
|
||
| =foo/package.json= | ||
| #+BEGIN_SRC json | ||
| { | ||
| "name": "foo", | ||
| "version": "1.0.0", | ||
| "dependencies": { | ||
| "underscore": "^1.13.0" | ||
| } | ||
| } | ||
| #+END_SRC | ||
|
|
||
| =bar/package.json= | ||
| #+BEGIN_SRC json | ||
| { | ||
| "name": "bar", | ||
| "version": "1.0.0", | ||
| "dependencies": { | ||
| "underscore": "1.13.1" | ||
| } | ||
| } | ||
| #+END_SRC | ||
|
|
||
| =quux/package.json= | ||
|
|
||
| Note that =name/repo= is implicitly treated as a github repo reference | ||
| #+BEGIN_SRC json | ||
| { | ||
| "name": "quux", | ||
| "version": "1.0.0", | ||
| "dependencies": { | ||
| "underscore": "jashkenas/underscore#tag=1.13.1" | ||
| } | ||
| } | ||
| #+END_SRC | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,19 @@ | ||||||||||||||||||||
| module Algebra.Graph.AdjacencyMap.Extra ( | ||||||||||||||||||||
| gtraverse, | ||||||||||||||||||||
| ) where | ||||||||||||||||||||
|
|
||||||||||||||||||||
| import Algebra.Graph.AdjacencyMap qualified as AM | ||||||||||||||||||||
| import Data.Set qualified as S | ||||||||||||||||||||
|
|
||||||||||||||||||||
| -- | It's 'traverse', but for graphs | ||||||||||||||||||||
| -- | ||||||||||||||||||||
| -- It's also unlawful. 'f' might be called several times for each node in the graph | ||||||||||||||||||||
| gtraverse :: | ||||||||||||||||||||
| (Applicative f, Ord b) => | ||||||||||||||||||||
| (a -> f b) -> | ||||||||||||||||||||
| AM.AdjacencyMap a -> | ||||||||||||||||||||
| f (AM.AdjacencyMap b) | ||||||||||||||||||||
| gtraverse f = fmap mkAdjacencyMap . traverse (\(a, xs) -> (,) <$> f a <*> traverse f xs) . AM.adjacencyList | ||||||||||||||||||||
| where | ||||||||||||||||||||
| mkAdjacencyMap :: Ord c => [(c, [c])] -> AM.AdjacencyMap c | ||||||||||||||||||||
| mkAdjacencyMap = AM.fromAdjacencySets . fmap (fmap S.fromList) | ||||||||||||||||||||
|
Comment on lines
+16
to
+19
There was a problem hiding this comment. [optional, readability]: I hate the
Suggested change
|
||||||||||||||||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is currently formatted as an emacs org-mode document, because it's substantially easier for me to edit that way. I'll make sure to convert to markdown before merging