support for manually specified dependencies

.github/pull_request_template.md

@@ -26,8 +26,8 @@ _Example:_ Closes https://github.com/example/repo/issues/123.
 
 ## Checklist
 
-- [ ] I added tests for this PR's change.
-- [ ] I linted (`hlint`) and formatted (`ormolu`) any files I touched in this PR.
-- [ ] If this PR introduced a user-visible change, I added documentation into `docs/`.
-- [ ] If this PR marked a release, I updated `CHANGELOG.md`.
-- [ ] I linked this PR to any referenced GitHub issues.
+- [] I added tests for this PR's change.
+- [] I linted (`hlint`) and formatted (`ormolu`) any files I touched in this PR.
+- [] If this PR introduced a user-visible change, I added documentation into `docs/`.
+- [] I updated `CHANGELOG.md`. If the PR did not mark a release, update the `#Unreleased section at the top`.
+- [] I linked this PR to any referenced GitHub issues.

Changelog.md

@@ -1,4 +1,5 @@
 # Unreleased
+- Support for manually specified dependencies through `fossa-deps.yaml` ([#236](https://github.com/fossas/spectrometer/pull/236))
 
 # v2.5.15
 

docs/userguide.md

@@ -9,6 +9,7 @@
         - [Printing results without uploading to FOSSA](#printing-results-without-uploading-to-fossa)
         - [Running in a specific directory](#running-in-a-specific-directory)
         - [Scanning archive contents](#scanning-archive-contents)
+        - [Manually specifying dependencies](#manually-specifying-dependencies)
     - [`fossa test`](#fossa-test)
         - [Specifying a timeout](#specifying-a-timeout)
         - [Print issues as json](#print-issues-as-json)
@@ -169,6 +170,40 @@ We support the following archive formats:
 - `.jar`
 - `.rpm`
 
+### Manually specifying dependencies
+
+FOSSA offers a way to manually upload dependencies provided we support the dependency type. Manually specifying dependencies is very helpful in the event your package manager is unsupported or you are using a custom and nonstandard dependency management solution.
+
+The FOSSA CLI will automatically read `fossa-deps.yml` in the root directory when `fossa analyze` is run and parse dependencies from it.
+
+> Tip: Use a script to generate this file before running `fossa analyze` to keep your results updated.
+
+```yaml
+dependencies:
+- type: gem
+  package: iron
+- type: pip
+  package: Django
+  version: 2.1.7
+```
+The `package` and `type` fields are required and specify the name of the dependency and where to find it. The `version` field is optional and specifies the preferred version of dependency.
+
+Supported dependency types:
+- `cargo` - Rust dependencies that a typically found at [crates.io](https://crates.io/).
+- `carthage` - Dependencies as specified by the [Carthage](https://github.com/Carthage/Carthage) package manager.
+- `composer` - Dependencies specified by the PHP package manager [Composer](https://getcomposer.org/), which are located on [Packagist](https://packagist.org/).
+- `gem` - Dependencies which can be found at [RubyGems.org](https://rubygems.org/).
+- `git` - Github projects (which appear as dependencies in many package managers). Specified as the full github repository `https://github.com/fossas/spectrometer`.
+- `go` - Golang specific dependency. Many golang dependencies are located on Github, but there are some which look like the following `go.mongodb.org/mongo-driver` that have custom golang URLs.
+- `hackage` - Haskell dependencies found at [Hackage](https://hackage.haskell.org/).
+- `hex` - Erlang and Elixir dependencies that are found at [Hex.pm](https://hex.pm/).
+- `maven` - Maven dependencies that can be found at many different sources. Specified as `package: javax.xml.bind:jaxb-api` where the convention is `groupId:artifactId`.
+- `npm` - Javascript dependencies found at [npmjs.com](https://www.npmjs.com/).
+- `nuget` - .NET dependencies found at [NuGet.org](https://www.nuget.org/).
+- `python` - Python dependencies found at [Pypi.org](https://pypi.org/).
+- `cocoapods` - Swift and Objective-C dependencies found at [Cocoapods.org](https://cocoapods.org/).
+- `url` - The URL type allows you to specify only the download location of a compressed file in the `package` field and the FOSSA backend will attempt to download and scan it. Example for a Maven dependency `https://repo1.maven.org/maven2/aero/m-click/mcpdf/0.2.3/mcpdf-0.2.3-jar-with-dependencies.jar`. The `version` field will be ignored for `url` type dependencies.
+
 ## `fossa test`
 
 The test command checks whether the most-recent scan of your FOSSA project raised license-policy or vulnerability issues. This command is usually run immediately after `fossa analyze`

spectrometer.cabal

@@ -247,6 +247,7 @@ library
     Strategy.Ruby.GemfileLock
     Strategy.Scala
     Strategy.Yarn
+    Strategy.UserSpecified.YamlDependencies
     Text.URI.Builder
     Types
     VCS.Git
@@ -322,6 +323,7 @@ test-suite unit-tests
     RPM.SpecFileSpec
     Ruby.BundleShowSpec
     Ruby.GemfileLockSpec
+    UserSpecified.YamlDependenciesSpec
 
   build-tool-depends: hspec-discover:hspec-discover ^>=2.7.1
   build-depends:

src/App/Fossa/Analyze.hs

@@ -76,6 +76,7 @@ import qualified Strategy.Python.Setuptools as Setuptools
 import qualified Strategy.RPM as RPM
 import qualified Strategy.Rebar3 as Rebar3
 import qualified Strategy.Scala as Scala
+import qualified Strategy.UserSpecified.YamlDependencies as UserYaml
 import qualified Strategy.Yarn as Yarn
 import System.Exit (die, exitFailure)
 import Types
@@ -164,7 +165,8 @@ discoverFuncs =
     ProjectAssetsJson.discover,
     ProjectJson.discover,
     Glide.discover,
-    Pipenv.discover
+    Pipenv.discover,
+    UserYaml.discover
   ]
 
 runDependencyAnalysis ::

src/DepTypes.hs

@@ -61,6 +61,7 @@ data DepType =
   | GoType -- ^ Go dependency
   | CargoType -- ^ Rust Cargo Dependency
   | RPMType -- ^ RPM dependency
+  | URLType -- ^ URL dependency
   | HackageType -- ^ Hackage Registry
   -- TODO: does this break the "location" abstraction?
   | CarthageType -- ^ A Carthage dependency -- effectively a "git" dependency. Name is repo path and version is tag/branch/hash

src/Srclib/Converter.hs

@@ -105,5 +105,6 @@ depTypeToFetcher = \case
   CarthageType -> "cart"
   CargoType -> "cargo"
   RPMType -> "rpm"
+  URLType -> "url"
   ComposerType -> "comp"
   HackageType -> "hackage"

src/Strategy/UserSpecified/YamlDependencies.hs

@@ -0,0 +1,120 @@
+{-# LANGUAGE RecordWildCards #-}
+{-# LANGUAGE TemplateHaskell #-}
+{-# LANGUAGE TypeApplications #-}
+
+module Strategy.UserSpecified.YamlDependencies
+  ( analyze,
+    discover,
+    buildGraph,
+    UserDependencies (..),
+  )
+where
+
+import Control.Effect.Diagnostics
+import Data.Aeson
+import Data.Aeson.Types (Parser)
+import qualified Data.Map.Strict as M
+import Data.Text (Text, unpack)
+import DepTypes
+import Effect.ReadFS
+import Graphing (Graphing)
+import qualified Graphing
+import Path
+import Types
+
+discover :: (Has ReadFS sig m, Has ReadFS rsig run, Has Diagnostics rsig run) => Path Abs Dir -> m [DiscoveredProject run]
+discover dir = map mkProject <$> findProjects dir
+
+-- Only search for fossa-deps.yaml in the root directory. We can extend this to subdirectories in the future.
+findProjects :: (Has ReadFS sig m) => Path Abs Dir -> m [UserDependenciesYamlProject]
+findProjects dir = do
+  let file = dir </> $(mkRelFile "fossa-deps.yml")
+  exists <- doesFileExist file
+  if exists
+    then pure [UserDependenciesYamlProject file dir]
+    else pure []
+
+data UserDependenciesYamlProject = UserDependenciesYamlProject
+  { dependenciesFile :: Path Abs File,
+    dependenciesDir :: Path Abs Dir
+  }
+  deriving (Eq, Ord, Show)
+
+mkProject :: (Has ReadFS sig n, Has Diagnostics sig n) => UserDependenciesYamlProject -> DiscoveredProject n
+mkProject project =
+  DiscoveredProject
+    { projectType = "user-specified-yaml",
+      projectBuildTargets = mempty,
+      projectDependencyGraph = const $ getDeps project,
+      projectPath = dependenciesDir project,
+      projectLicenses = pure []
+    }
+
+getDeps :: (Has ReadFS sig m, Has Diagnostics sig m) => UserDependenciesYamlProject -> m (Graphing Dependency)
+getDeps = analyze . dependenciesFile
+
+analyze :: (Has ReadFS sig m, Has Diagnostics sig m) => Path Abs File -> m (Graphing Dependency)
+analyze file = buildGraph <$> readContentsYaml @UserDependencies file
+
+buildGraph :: UserDependencies -> Graphing Dependency
+buildGraph lockfile = Graphing.fromList (map toDependency direct)
+  where
+    direct = dependencies lockfile
+    toDependency UserDependency {..} =
+      Dependency
+        { dependencyType = depType,
+          dependencyName = depPackage,
+          dependencyVersion = CEq <$> depVersion,
+          dependencyLocations = [],
+          dependencyEnvironments = [],
+          dependencyTags = M.empty
+        }
+
+data UserDependencies = UserDependencies
+  { dependencies :: [UserDependency]
+  }
+  deriving (Eq, Ord, Show)
+
+data UserDependency = UserDependency
+  { depPackage :: Text,
+    depType :: DepType,
+    depVersion :: Maybe Text
+  }
+  deriving (Eq, Ord, Show)
+
+instance FromJSON UserDependencies where
+  parseJSON = withObject "Dependencies" $ \obj ->
+    UserDependencies <$> obj .:? "dependencies" .!= []
+
+depTypeParser :: Text -> Parser DepType
+depTypeParser text = case depTypeFromText text of
+  Just t -> pure t
+  Nothing -> fail $ "dep type: " <> unpack text <> " not supported"
+
+instance FromJSON UserDependency where
+  parseJSON = withObject "UserDependency" $ \obj ->
+    UserDependency <$> obj .: "package"
+      <*> (obj .: "type" >>= depTypeParser)
+      <*> obj .:? "version"
+
+-- Parse supported dependency types into their respective type or return Nothing.
+depTypeFromText :: Text -> Maybe DepType
+depTypeFromText text = case text of
+  "cargo" -> Just CargoType
+  "carthage" -> Just CarthageType
+  "composer" -> Just ComposerType
+  "gem" -> Just GemType
+  "git" -> Just GitType
+  "go" -> Just GoType
+  "hackage" -> Just HackageType
+  "hex" -> Just HexType
+  "maven" -> Just MavenType
+  "npm" -> Just NodeJSType
+  "nuget" -> Just NuGetType
+  "python" -> Just PipType
+  "cocoapods" -> Just PodType
+  "url" -> Just URLType
+  _ -> Nothing -- unsupported dep, need to respond with an error and skip this dependency
+  -- rpm is an unsupported type. This is because we currently have 2 RPM fetchers
+  -- and we should wait for a need to determine which one to use for manually
+  -- specified dependencies.

test/UserSpecified/YamlDependenciesSpec.hs

@@ -0,0 +1,53 @@
+{-# LANGUAGE TypeApplications #-}
+
+module UserSpecified.YamlDependenciesSpec
+  ( spec,
+  )
+where
+
+import Control.Algebra
+import qualified Data.ByteString as BS
+import qualified Data.Map.Strict as M
+import Data.Yaml
+import DepTypes
+import Effect.Grapher
+import Graphing (Graphing)
+import Strategy.UserSpecified.YamlDependencies
+import Test.Hspec
+
+expected :: Graphing Dependency
+expected = run . evalGrapher $ do
+  direct $
+    Dependency
+      { dependencyType = GemType,
+        dependencyName = "one",
+        dependencyVersion = Nothing,
+        dependencyLocations = [],
+        dependencyEnvironments = [],
+        dependencyTags = M.empty
+      }
+  direct $
+    Dependency
+      { dependencyType = URLType,
+        dependencyName = "two",
+        dependencyVersion = Just (CEq "1.0.0"),
+        dependencyLocations = [],
+        dependencyEnvironments = [],
+        dependencyTags = M.empty
+      }
+
+spec :: Spec
+spec = do
+  testFile <- runIO (BS.readFile "test/UserSpecified/testdata/valid-deps.yaml")
+  unsupportedTypeFile <- runIO (BS.readFile "test/UserSpecified/testdata/invalid-deps.yaml")
+
+  describe "yaml user specified dependencies" $ do
+    it "works end to end" $
+      case decodeEither' testFile of
+        Right res -> buildGraph res `shouldBe` expected
+        Left err -> expectationFailure $ "failed to parse: " <> show err
+
+    it "fails with unsupported deps" $
+      case decodeEither' @UserDependencies unsupportedTypeFile of
+        Right res -> expectationFailure $ "Expected a failure to parse due to unsupported dependency, but got: " <> show res
+        Left _ -> pure ()

test/UserSpecified/testdata/invalid-deps.yaml

@@ -0,0 +1,3 @@
+dependencies:
+- package: one
+  type: unsupported

test/UserSpecified/testdata/valid-deps.yaml

@@ -0,0 +1,6 @@
+dependencies:
+- package: one
+  type: gem
+- type: url
+  package: two
+  version: 1.0.0