Dependencies excluded in Artifacts

[darkfrog26]

I'm using the following code to resolve and build a list of files for JARs to load my application:

      import coursier._
      val start = Resolution(Set(Dependency(Module(vd.group, vd.name), vd.version.toString())))
      val fetch = Fetch.from(repositories, Cache.fetch())
      val resolution = start.process.run(fetch).unsafePerformSync
      resolution.errorCache.foreach {
        case (key, value) => scribe.error(s"Error Cache: $key = $value")
      }
      val errors = resolution.metadataErrors
      val conflicts = resolution.conflicts

      if (errors.nonEmpty) {
        logger.warn(s"Errors for ($vd): $errors")
      }
      if (conflicts.nonEmpty) {
        throw new RuntimeException(s"Conflicts for ($vd): $conflicts")
      }
      val localArtifacts = Task.gatherUnordered(
        resolution.artifacts.map(Cache.file(_, logger = Some(monitor.cacheLogger)).run)
      ).unsafePerformSync
      val fileErrors = localArtifacts.map(_.toEither).collect {
        case Left(err) => err
      }
      if (fileErrors.nonEmpty) {
        throw new RuntimeException(s"File Errors for ($vd): $fileErrors")
      }
      localArtifacts.map(_.toEither).collect {
        case Right(f) => f
      }

However, when I attempt to load http://search.maven.org/#artifactdetails%7Cio.youi%7Cyoui-server-example_2.12%7C0.3.4%7Cjar it includes youi-app dependencies (resolution.dependencies) as shown in the Maven dependencies, but when I get the artifacts (resolution.artifacts) it is not included. Further, when I try to actually load the application it can't find dependencies within that JAR. Something is causing Dependencies to be excluded from Artifacts and it's causing serious problems when I try to use this.

[alexarchambault]

This seems to work for me... Getting those jars:

List(
  "args4j/args4j/2.33/args4j-2.33.jar",
  "com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar",
  "com/google/code/gson/gson/2.7/gson-2.7.jar",
  "com/google/errorprone/error_prone_annotations/2.0.18/error_prone_annotations-2.0.18.jar",
  "com/google/guava/guava/20.0/guava-20.0.jar",
  "com/google/javascript/closure-compiler-externs/v20170423/closure-compiler-externs-v20170423.jar",
  "com/google/javascript/closure-compiler/v20170423/closure-compiler-v20170423.jar",
  "com/google/jsinterop/jsinterop-annotations/1.0.0/jsinterop-annotations-1.0.0.jar",
  "com/google/protobuf/protobuf-java/3.0.2/protobuf-java-3.0.2.jar",
  "com/lihaoyi/derive_2.12/0.4.4/derive_2.12-0.4.4.jar",
  "com/lihaoyi/sourcecode_2.12/0.1.3/sourcecode_2.12-0.1.3.jar",
  "com/lihaoyi/upickle_2.12/0.4.4/upickle_2.12-0.4.4.jar",
  "com/outr/hasher_2.12/1.2.1/hasher_2.12-1.2.1.jar",
  "com/outr/reactify_2.12/1.5.3/reactify_2.12-1.5.3.jar",
  "com/outr/scribe_2.12/1.4.2/scribe_2.12-1.4.2.jar",
  "com/typesafe/akka/akka-actor_2.12/2.5.1/akka-actor_2.12-2.5.1.jar",
  "com/typesafe/config/1.3.1/config-1.3.1.jar",
  "io/undertow/undertow-core/1.4.14.Final/undertow-core-1.4.14.Final.jar",
  "io/youi/youi-app_2.12/0.3.4/youi-app_2.12-0.3.4.jar",
  "io/youi/youi-communication_2.12/0.3.4/youi-communication_2.12-0.3.4.jar",
  "io/youi/youi-core_2.12/0.3.4/youi-core_2.12-0.3.4.jar",
  "io/youi/youi-optimizer_2.12/0.3.4/youi-optimizer_2.12-0.3.4.jar",
  "io/youi/youi-server-example_2.12/0.3.4/youi-server-example_2.12-0.3.4.jar",
  "io/youi/youi-server-undertow_2.12/0.3.4/youi-server-undertow_2.12-0.3.4.jar",
  "io/youi/youi-server_2.12/0.3.4/youi-server_2.12-0.3.4.jar",
  "io/youi/youi-stream_2.12/0.3.4/youi-stream_2.12-0.3.4.jar",
  "io/youi/youi-template_2.12/0.3.4/youi-template_2.12-0.3.4.jar",
  "javax/annotation/jsr250-api/1.0/jsr250-api-1.0.jar",
  "net/sf/qualitycheck/quality-check/1.3/quality-check-1.3.jar",
  "net/sf/uadetector/uadetector-core/0.9.22/uadetector-core-0.9.22.jar",
  "net/sf/uadetector/uadetector-resources/2014.10/uadetector-resources-2014.10.jar",
  "org/jboss/logging/jboss-logging/3.2.1.Final/jboss-logging-3.2.1.Final.jar",
  "org/jboss/xnio/xnio-api/3.3.6.Final/xnio-api-3.3.6.Final.jar",
  "org/powerscala/powerscala-io_2.12/2.0.5/powerscala-io_2.12-2.0.5.jar",
  "org/scala-lang/modules/scala-java8-compat_2.12/0.8.0/scala-java8-compat_2.12-0.8.0.jar",
  "org/scala-lang/modules/scala-xml_2.12/1.0.6/scala-xml_2.12-1.0.6.jar",
  "org/scala-lang/scala-library/2.12.2/scala-library-2.12.2.jar",
  "org/scala-lang/scala-reflect/2.12.2/scala-reflect-2.12.2.jar",
  "org/scalactic/scalactic_2.12/3.0.3/scalactic_2.12-3.0.3.jar",
  "org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar",
  "org/spire-math/jawn-parser_2.12/0.10.3/jawn-parser_2.12-0.10.3.jar"
)

(full code here)

Note that you should filter the artifacts, keeping only those ending in .jar now, as poms are returned too (or use dependencyArtifacts, that keeps the types, and filter artifacts with the .`type` field, keeping only types "jar" and "bundle").

[darkfrog26]

@alexarchambault your code does work, which led me to dive deeper into the distinctions. It would seem when I have Ivy2 Local and Cache in my repositories list is when it is dropping the JARs for some reason. Is this by design? If so, why? Finally, is there any way to keep the ivy local and cache while still making this work right?

[darkfrog26]

@alexarchambault still waiting on a response here...it's keeping me from moving forward on my project.

[alexarchambault]

I'd recommend not using ~/.ivy2/cache as a repo... It's been a long time I haven't looked at it, it may not be fine with some of the latest developments in coursier. ~/.ivy2/local is fine otoh.

[darkfrog26]

I can get by without using cache, local is the only one I actually need, but this does seem like a worthwhile bug to fix since it's extremely redundant to potentially have to download multiple copies of JARs.

[darkfrog26]

@alexarchambault sorry for the delay in coming back around to this. After removing cache things proceeded further, but it's still missing a dependency when I try to run my specific example. It can't find the XNIO provider. I believe this is caused by http://search.maven.org/#artifactdetails%7Cio.undertow%7Cundertow-core%7C1.4.16.Final%7Cjar referencing:

        <dependency>
            <groupId>org.jboss.xnio</groupId>
            <artifactId>xnio-nio</artifactId>
            <scope>runtime</scope>
        </dependency>

Does it exclude runtime scoped artifacts?

[alexarchambault]

You can ask for a given configuration / scope, by setting the configuration field of Dependency. Setting it to "runtime" or "default(runtime)" should work for you. Via the API, it defaults to "compile", which may not be the right choice after all, but that's its value for now...

[darkfrog26]

Wow, that did the trick. I've been looking for that for weeks now. A better naming that "configuration" might be helpful as I came across that multiple times but dismissed it as irrelevant since it just takes a String and is called "configuration".