Support SBT IntegrationTest configuration

[avdv]

Hi.

In my project, I have enabled the IntegrationTest config:

// build.sbt

Defaults.itSettings

I am using addSbtPlugin("ch.epfl.scala" % "sbt-scalafix" % "0.5.0-RC1").

When running scalafix RemoveUnusedImports or even it:scalafix RemoveUnusedImports there are a lot of warnings about the scala files of the IntegrationTest configuration, a la "No semanticdb associated with" them.

[info] Running scalafix -r RemoveUnusedImports --classpath /home/claudio/src/data/target/scala-2.12/classes --sourceroot /home/claudio/src/data /home/claudio/src/data/app /home/claudio/src/data/app
...
error: No semanticdb associated with /home/claudio/src/data/app/persistence/StencilMapperMongo.scala. Is --sourceroot correct?

Of course, the classpath is not correctly set, as the classes are written to targetDirectory / scalaBinaryVersion / "it-classes".

I used

scalafix in IntegrationTest := ScalafixPlugin.scalafixTaskImpl(IntegrationTest).evaluated

to workaround that problem and it is half-working now - some files are properly processed. Some are not.

Is it possible to set up the scalafix task appropriately for each configuration (projectConfigurations: Seq[Configuration])?

\edit: Note, I also get some exceptions during the compilation stage:

[warn] failed to generate semanticdb for /home/claudio/src/data/test/it/helpers/TestApplication.scala:
[warn] org.scalameta.UnreachableError: this code path should've been unreachable
[warn] where tpe = helpers.TestApplication with org.specs2.SpecificationLike
[warn] 	at org.scalameta.UnreachableError$.raise(package.scala:40)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps$SyntheticCodePrinter.printType(PrinterOps.scala:406)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps$SyntheticCodePrinter.processTreePrinting(PrinterOps.scala:801)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps$SyntheticCodePrinter.printTree(PrinterOps.scala:415)
[warn] 	at scala.tools.nsc.ast.Printers$TreePrinter.$anonfun$print$1(Printers.scala:33)
[warn] 	at scala.tools.nsc.ast.Printers$TreePrinter.$anonfun$print$1$adapted(Printers.scala:17)
[warn] 	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
[warn] 	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
[warn] 	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
[warn] 	at scala.tools.nsc.ast.Printers$TreePrinter.print(Printers.scala:17)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps$SyntheticCodePrinter.super$print(PrinterOps.scala:55)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps$SyntheticCodePrinter.$anonfun$print$1(PrinterOps.scala:60)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps$SyntheticCodePrinter.$anonfun$print$1$adapted(PrinterOps.scala:52)
[warn] 	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
[warn] 	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
[warn] 	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps$SyntheticCodePrinter.print(PrinterOps.scala:52)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps.showSynthetic(PrinterOps.scala:22)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps.showSynthetic$(PrinterOps.scala:19)
[warn] 	at scala.meta.internal.SemanticdbPlugin.showSynthetic(ScalahostPlugin.scala:9)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps.showSynthetic(PrinterOps.scala:17)
[warn] 	at scala.meta.internal.semanticdb.PrinterOps.showSynthetic$(PrinterOps.scala:14)
[warn] 	at scala.meta.internal.SemanticdbPlugin.showSynthetic(ScalahostPlugin.scala:9)
[warn] 	at scala.meta.internal.semanticdb.DenotationOps$XtensionGSymbolMDenotation.info(DenotationOps.scala:94)
[warn] 	at scala.meta.internal.semanticdb.DenotationOps$XtensionGSymbolMDenotation.toDenotation(DenotationOps.scala:109)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.success$1(DocumentOps.scala:169)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.tryMstart$1(DocumentOps.scala:196)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.tryFindMtree(DocumentOps.scala:253)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.traverse(DocumentOps.scala:389)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.traverse(DocumentOps.scala:149)
[warn] 	at scala.reflect.api.Trees$Traverser.traverseSelfType(Trees.scala:2481)
[warn] 	at scala.reflect.internal.Trees.traverseComponents$1(Trees.scala:1242)
[warn] 	at scala.reflect.internal.Trees.itraverse(Trees.scala:1341)
[warn] 	at scala.reflect.internal.Trees.itraverse$(Trees.scala:1211)
[warn] 	at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:16)
[warn] 	at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:16)
[warn] 	at scala.reflect.api.Trees$Traverser.traverse(Trees.scala:2475)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.traverse(DocumentOps.scala:398)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.traverse(DocumentOps.scala:149)
[warn] 	at scala.reflect.internal.Trees.$anonfun$itraverse$1(Trees.scala:1218)
[warn] 	at scala.reflect.api.Trees$Traverser.atOwner(Trees.scala:2507)
[warn] 	at scala.reflect.internal.Trees.traverseMemberDef$1(Trees.scala:1214)
[warn] 	at scala.reflect.internal.Trees.itraverse(Trees.scala:1339)
[warn] 	at scala.reflect.internal.Trees.itraverse$(Trees.scala:1211)
[warn] 	at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:16)
[warn] 	at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:16)
[warn] 	at scala.reflect.api.Trees$Traverser.traverse(Trees.scala:2475)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.traverse(DocumentOps.scala:398)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.traverse(DocumentOps.scala:149)
[warn] 	at scala.reflect.api.Trees$Traverser.$anonfun$traverseStats$2(Trees.scala:2498)
[warn] 	at scala.reflect.api.Trees$Traverser.atOwner(Trees.scala:2507)
[warn] 	at scala.reflect.api.Trees$Traverser.$anonfun$traverseStats$1(Trees.scala:2498)
[warn] 	at scala.reflect.api.Trees$Traverser.traverseStats(Trees.scala:2497)
[warn] 	at scala.reflect.internal.Trees.itraverse(Trees.scala:1337)
[warn] 	at scala.reflect.internal.Trees.itraverse$(Trees.scala:1211)
[warn] 	at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:16)
[warn] 	at scala.reflect.internal.SymbolTable.itraverse(SymbolTable.scala:16)
[warn] 	at scala.reflect.api.Trees$Traverser.traverse(Trees.scala:2475)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument$traverser$4$.traverse(DocumentOps.scala:398)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument.$anonfun$toDocument$1(DocumentOps.scala:401)
[warn] 	at scala.meta.internal.ReflectionToolkit$CompilationUnitCache.$anonfun$getOrElse$5(ReflectionToolkit.scala:91)
[warn] 	at scala.Option.getOrElse(Option.scala:121)
[warn] 	at scala.meta.internal.ReflectionToolkit$CompilationUnitCache.getOrElse(ReflectionToolkit.scala:90)
[warn] 	at scala.meta.internal.semanticdb.DocumentOps$XtensionCompilationUnitDocument.toDocument(DocumentOps.scala:14)
[warn] 	at scala.meta.internal.SemanticdbPipeline$ComputeSemanticdbComponent$ComputeSemanticdbPhase.apply(ScalahostPipeline.scala:55)
[warn] 	at scala.tools.nsc.Global$GlobalPhase.$anonfun$applyPhase$1(Global.scala:426)
[warn] 	at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:419)
[warn] 	at scala.tools.nsc.Global$GlobalPhase.$anonfun$run$1(Global.scala:390)
[warn] 	at scala.tools.nsc.Global$GlobalPhase.$anonfun$run$1$adapted(Global.scala:390)
[warn] 	at scala.collection.Iterator.foreach(Iterator.scala:929)
[warn] 	at scala.collection.Iterator.foreach$(Iterator.scala:929)
[warn] 	at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
[warn] 	at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:390)
[warn] 	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1431)
[warn] 	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1416)
[warn] 	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1412)
[warn] 	at scala.tools.nsc.Global$Run.compile(Global.scala:1515)
[warn] 	at xsbt.CachedCompiler0.run(CompilerInterface.scala:116)
[warn] 	at xsbt.CachedCompiler0.run(CompilerInterface.scala:95)
[warn] 	at xsbt.CompilerInterface.run(CompilerInterface.scala:26)
[warn] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[warn] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[warn] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[warn] 	at java.lang.reflect.Method.invoke(Method.java:498)
[warn] 	at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:107)
[warn] 	at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:53)
[warn] 	at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:47)
[warn] 	at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply$mcV$sp(MixedAnalyzingCompiler.scala:50)
[warn] 	at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply(MixedAnalyzingCompiler.scala:50)
[warn] 	at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply(MixedAnalyzingCompiler.scala:50)
[warn] 	at sbt.compiler.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:74)
[warn] 	at sbt.compiler.MixedAnalyzingCompiler.compileScala$1(MixedAnalyzingCompiler.scala:49)
[warn] 	at sbt.compiler.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:64)
[warn] 	at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
[warn] 	at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
[warn] 	at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:66)
[warn] 	at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:64)
[warn] 	at sbt.inc.IncrementalCommon.cycle(IncrementalCommon.scala:32)
[warn] 	at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:72)
[warn] 	at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:71)
[warn] 	at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:99)
[warn] 	at sbt.inc.Incremental$.compile(Incremental.scala:71)
[warn] 	at sbt.inc.IncrementalCompile$.apply(Compile.scala:54)
[warn] 	at sbt.compiler.IC$.compileInternal(IncrementalCompiler.scala:160)
[warn] 	at sbt.compiler.IC$.incrementalCompile(IncrementalCompiler.scala:138)
[warn] 	at sbt.Compiler$.compile(Compiler.scala:155)
[warn] 	at sbt.Compiler$.compile(Compiler.scala:141)
[warn] 	at sbt.Defaults$.sbt$Defaults$$compileIncrementalTaskImpl(Defaults.scala:913)
[warn] 	at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:904)
[warn] 	at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:902)
[warn] 	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
[warn] 	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
[warn] 	at sbt.std.Transform$$anon$4.work(System.scala:63)
[warn] 	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
[warn] 	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
[warn] 	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
[warn] 	at sbt.Execute.work(Execute.scala:237)
[warn] 	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
[warn] 	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
[warn] 	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
[warn] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
[warn] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[warn] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[warn] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[warn] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[warn] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[warn] 	at java.lang.Thread.run(Thread.java:748)

[avdv]

Using set scalafixSemanticdbVersion := "2.0.0-RC3" works better. No exceptions during compliation anymore... 🎉

Only a few warnings:

Stale semanticdb for test/it/helpers/TestApplication.scala, skipping rule. Please recompile.
data] Running RemoveUnusedImports (75.00 %, 252 / 336)

error: Stale semanticdb for test/it/helpers/TestApplication.scala, skipping rule. Please recompile.
[data] Running RemoveUnusedImports (82.14 %, 276 / 336)

But that is probably because the file was modified (import removed) and it is processed more than once... (the warning is shown twice).

[olafurpg]

Thank you for reporting! What scalafix version are you using?

[olafurpg]

Aaah, great. Yes, scalafix 0.5.0-RC1 was released last night and it depends on RC3.

[olafurpg]

For integration tests,

scalafix in IntegrationTest := ScalafixPlugin.scalafixTaskImpl(IntegrationTest).evaluated
scalafix := scalafixTaskImpl(Compile, Test, IntegrationTest).evaluated

is a good workaround for time being. I haven't figured out how to abstract over this yet because of "illegal dynamic reference" errors. Let's keep this issue open, I definitely want to support this feature.

As for the "stale semanticdb" warnings, they could be spurious, see #308. Scalafix has check to make sure it won't override edits to the source file that happen after compilation. In this case, it seems like it's racing with itself.

[avdv]

Aaah, great. Yes, scalafix 0.5.0-RC1 was released last night and it depends on RC3.

OK, would you need to bump the version in the Dependencies.scala file then?

For integration tests,

scalafix in IntegrationTest := ScalafixPlugin.scalafixTaskImpl(IntegrationTest).evaluated
scalafix := scalafixTaskImpl(Compile, Test, IntegrationTest).evaluated
is a good workaround for time being. I haven't figured out how to abstract over this yet because of "illegal dynamic reference" errors. Let's keep this issue open, I definitely want to support this feature.

Nice, thank you!

As for the "stale semanticdb" warnings, they could be spurious, see #308. Scalafix has check to make sure it won't override edits to the source file that happen after compilation. In this case, it seems like it's racing with itself.

After the first scalafix run, I did recompile all the class files and on the second scalafix run those warnings did not show up. This may be related to the fact that in my project the integration test .scala files are in a sub-folder of the test/ source directory, e.g.:

sources in Test := {
      val itDirs = sourceDirectories.in(IntegrationTest).value

      sources.in(Test).value filter { f ⇒ !itDirs.exists(f.relativeTo(_).isDefined) }
    }

Could this be some problem?

[olafurpg]

OK, would you need to bump the version in the Dependencies.scala file then?

It seems I forgot to bump up the version. Good catch! I will make sure it's upgraded before 0.5.0.

Could this be some problem?

Yes. Scalafix uses unmanagedSourceDirectories to decide which files to fix for each configuration. This is the idiomatic way to do it, to my best knowledge, but I guess we could provide way to override this behavior.

[avdv]

Yes. Scalafix uses unmanagedSourceDirectories to decide which files to fix for each configuration. This is the idiomatic way to do it, to my best knowledge, but I guess we could provide way to override this behavior.

Would it be better to use unmanagedSources perhaps? Ie. to only consider the files the scala compiler "sees" instead of trying to infer them from the directories?

[olafurpg]

I agree, unmanagedSources would be a better key to use. I opened #340 to track this change.

[bjaglin]

This feature is now documented, see https://scalacenter.github.io/scalafix/docs/users/installation.html#integration-tests