Add support for Yarn v2 projects #244

cnr · 2021-06-01T04:40:56Z

Overview

This PR adds support for the v2 yarn lockfile format. When analyzing yarn projects, we first attempt to analyze yarn.lock as a v1 lockfile, then fall back to analyzing as a v2 lockfile.

Developer documentation for the yarn v2 lockfile is rendered here

Acceptance criteria

fossa analyze supports projects containing a yarn v2 lockfile, including all standard dependency types contained therein (see associated devdocs and Strategy.Yarn.V2.Resolvers module for more info)
fossa analyze continues to support projects containing a yarn v1 lockfile

Testing plan

Automated unit tests, including an end-to-end test on the lockfile in the devdocs
Manual sanity check tests on various yarn projects:
1. Analyze a yarn v1 project in --output mode: fossa analyze --output
2. Set the yarn version to v2: yarn set version berry
3. Update the lockfile: yarn install
4. Analyze the same project in --output mode: fossa analyze --output
5. The dependency output from steps 1 and 4 should be identical (but with the addition of dependencies marked as "direct". Thanks yarn v2!)

Risks

None

References

Closes https://github.com/fossas/team-analysis/issues/526

Checklist

Remaining:

Convert devdocs to markdown

… found

cnr · 2021-06-01T04:46:12Z

devdocs/buildtools/yarnv2.org

@@ -0,0 +1,186 @@
+


This file is currently formatted as an emacs org-mode document, because it's substantially easier for me to edit that way. I'll make sure to convert to markdown before merging

cnr · 2021-06-01T04:47:24Z

src/Control/Effect/Diagnostics.hs

    ]
+    ++ if null failureWarnings


This change is unrelated to adding yarn support. It excludes the "Relevant warnings include" addendum to error messages if there are no relevant warnings

cnr · 2021-06-01T04:50:31Z

src/Strategy/Yarn.hs

    }

 getDeps :: (Has ReadFS sig m, Has Diagnostics sig m) => YarnProject -> m (G.Graphing Dependency)
-getDeps = context "Yarn" . context "Static analysis" . YarnLock.analyze' . yarnLock
+getDeps project = context "Yarn" $ getDepsV1 project <||> getDepsV2 project


this function can also be written as:

getDeps :: (Has ReadFS sig m, Has Diagnostics sig m) => YarnProject -> m (G.Graphing Dependency) getDeps = (<||>) <$> getDepsV1 <*> getDepsV2

.. which is really cute but unmaintainable

cnr · 2021-06-01T04:51:38Z

src/Strategy/Yarn/V2/Lockfile.hs

+-- Fortunately, aeson provides a mechanism for us to represent this:
+-- 'FromJSONKey', which is used in the 'FromJSON' instance for 'Map'. It allows
+-- us to arbitrarily decode any type as a Map key (assuming an 'Ord' instance,
+-- of course).


It's incredible that aeson has this. It made parsing this format SO much easier

cnr · 2021-06-01T04:52:57Z

src/Strategy/Yarn/V2/Lockfile.hs

+-- From yarn's structUtils.tryParseDescriptor/tryParselocator (in strict mode):
+--
+-- @
+--     string.match(/^(?:@([^/]+?)\/)?([^/]+?)(?:@(.+))$/)


I never want to see another regex in my life.

Special shoutout to whatever the heck this is:

range.match(/^([^#:]*:)?((?:(?!::)[^#])*)(?:#((?:(?!::).)*))?(?:::(.*))?$/);

elldritch · 2021-06-02T16:33:14Z

I really like the explanations you provided in the developer docs. Can you link to the parts of the Yarn documentation (or other source material e.g. blog posts, GH source permalinks) that you're referencing for some of the content? For example, you talk about the semantics behind specific fields in the lockfile - is there a doc somewhere in Yarn that describes those semantics?

Having these references makes it easier for new developers to see where to look when we inevitably need to make bugfixes or keep up with spec changes (so we have somewhere to go to answer "are we actually doing the semantically correct thing?").

cnr · 2021-06-02T16:54:08Z

is there a doc somewhere in Yarn that describes those semantics?

unfortunately, their source code is the documentation for most of this. I'll add more source code references

skilly-lily

LGTM, few nits and one new test case for a Text function.

skilly-lily · 2021-06-02T18:57:47Z

test/Yarn/V2/testdata/yarn.lock

+# This file is generated by running "yarn install" inside your project.
+# Manual changes might be lost - proceed with caution!


[nit]: We should probably remove these comments. They're a little misleading for test purposes. We can use other comments if we're interested in checking the parser.

skilly-lily · 2021-06-03T22:02:23Z

spectrometer.cabal

    StrictData
    TupleSections
    TypeApplications
    TypeOperators
    TypeSynonymInstances
+    ImportQualifiedPost
+    StandaloneKindSignatures


[nit]: did cabal-fmt put these out-of-order?

yeah, I'm not sure why it's doing that 😕

skilly-lily · 2021-06-03T22:11:42Z

src/Algebra/Graph/AdjacencyMap/Extra.hs

+gtraverse f = fmap mkAdjacencyMap . traverse (\(a, xs) -> (,) <$> f a <*> traverse f xs) . AM.adjacencyList
+  where
+    mkAdjacencyMap :: Ord c => [(c, [c])] -> AM.AdjacencyMap c
+    mkAdjacencyMap = AM.fromAdjacencySets . fmap (fmap S.fromList)


[optional, readability]: I hate the (,) constructor so much, it's really annoying to read. I usually use tuplify, since names are easier to read in prefix position than operators-in-parenthesis.

Suggested change

gtraverse f = fmap mkAdjacencyMap . traverse (\(a, xs) -> (,) <$> f a <*> traverse f xs) . AM.adjacencyList

where

mkAdjacencyMap :: Ord c => [(c, [c])] -> AM.AdjacencyMap c

mkAdjacencyMap = AM.fromAdjacencySets . fmap (fmap S.fromList)

gtraverse f = fmap mkAdjacencyMap . traverse (\(a, xs) -> tuplify <$> f a <*> traverse f xs) . AM.adjacencyList

where

tuplify a b = (a, b)

mkAdjacencyMap :: Ord c => [(c, [c])] -> AM.AdjacencyMap c

mkAdjacencyMap = AM.fromAdjacencySets . fmap (fmap S.fromList)

skilly-lily · 2021-06-03T22:13:45Z

src/Data/Text/Extra.hs

@@ -46,3 +48,6 @@ underBS f = decodeUtf8 . f . encodeUtf8

 showT :: Show a => a -> Text
 showT = T.pack . show
+
+dropPrefix :: Text -> Text -> Text
+dropPrefix pre txt = fromMaybe txt (T.stripPrefix pre txt)


Can we add a single test case in test/Extra/TextSpec.hs?

This fixes an issue where a dependency with a version like `bar` in: ``` dependencies: foo: 1.0.0 bar: 1 ``` .. would cause a parse error, because it's not canonically Text

cnr added 20 commits May 25, 2021 10:39

WIP - add parsers for locator and descriptor

1cdef17

WIP - define json/yaml parsers

2c8311f

Use the best parser

5abb9cd

Add LockfileV2 and Resolvers modules

12a3b26

Add resolvers and partial analysis logic

342b503

Use real diagnostics

2bc3d06

Implement graph-building logic

41ac122

Add tar resolver

653e59e

use better message for unsupported locator

3bcac7b

Add remaining unhandled resolvers

3d75f50

format cabal file

96f0118

Add devdocs for yarn v2

23031f7

Add a couple of links to yarnv2 devdocs

2ba18c6

Add docs to LockfileV2

aa4070d

Add docs to Resolvers

eac91a2

Mention hack for npm: protocol lookup

4c928cf

Reorganize yarn strategy module layout

72dc874

Strategy.Npm: ignore directories and subdirectories when yarn.lock is…

b329091

… found

Yarn: try V1 lockfile then V2 lockfile

bff5f1b

Factor out duplication in unsupported resolvers

915dcf3

cnr requested a review from skilly-lily June 1, 2021 04:40

cnr commented Jun 1, 2021

View reviewed changes

cnr added 4 commits June 1, 2021 08:38

Copy editing

41c61f3

Add tests for resolvers

4e17b1a

Kill redundant resolver names

316f5ee

Add end-to-end test for the example lockfile

ef4c432

Update changelog

8a78159

skilly-lily approved these changes Jun 3, 2021

View reviewed changes

cnr added 7 commits June 4, 2021 09:45

Add more links to yarnv2 devdocs

9195ffc

Kill scary comment at the top of test yarn.lock

0b8c7fd

Add tests for Data.Text.Extra.dropPrefix

e3e7303

Allow dependency versions as numbers

f858a6b

This fixes an issue where a dependency with a version like `bar` in: ``` dependencies: foo: 1.0.0 bar: 1 ``` .. would cause a parse error, because it's not canonically Text

Strip selectors from npm locators

052d18d

Add workaround for semver range coalescing

878fc2e

Merge remote-tracking branch 'origin/master' into yarn-v2

5a0340e

cnr merged commit c4a6d0a into master Jun 4, 2021

cnr deleted the yarn-v2 branch June 4, 2021 20:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Yarn v2 projects #244

Add support for Yarn v2 projects #244

cnr commented Jun 1, 2021 •

edited

Loading

cnr Jun 1, 2021

cnr Jun 1, 2021 •

edited

Loading

cnr Jun 1, 2021

cnr Jun 1, 2021

cnr Jun 1, 2021 •

edited

Loading

elldritch commented Jun 2, 2021

cnr commented Jun 2, 2021 •

edited

Loading

skilly-lily left a comment

skilly-lily Jun 2, 2021

skilly-lily Jun 3, 2021

cnr Jun 4, 2021

skilly-lily Jun 3, 2021

skilly-lily Jun 3, 2021

		# This file is generated by running "yarn install" inside your project.
		# Manual changes might be lost - proceed with caution!

Add support for Yarn v2 projects #244

Add support for Yarn v2 projects #244

Conversation

cnr commented Jun 1, 2021 • edited Loading

Overview

Acceptance criteria

Testing plan

Risks

References

Checklist

Choose a reason for hiding this comment

cnr Jun 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cnr Jun 1, 2021 • edited Loading

Choose a reason for hiding this comment

elldritch commented Jun 2, 2021

cnr commented Jun 2, 2021 • edited Loading

skilly-lily left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cnr commented Jun 1, 2021 •

edited

Loading

cnr Jun 1, 2021 •

edited

Loading

cnr Jun 1, 2021 •

edited

Loading

cnr commented Jun 2, 2021 •

edited

Loading