Error compiling code that is source compatible, but binary incompatible, across library boundaries #4596

bhmiller · 2017-05-16T00:14:41Z

Repro

pants.ini:

[GLOBAL]
pants_version: 1.3.0rc1

BUILD:

scala_library(
    name='MyLibrary',
    sources=['MyClass.scala'],
)

scala_library(
    name='MyApp',
    sources=['MyApp.scala'],
    dependencies=[':MyLibrary'],
)

jvm_binary(
    name='MyApp-bin',
    main='bar.MyApp',
    dependencies=[':MyApp'],
)

MyClass.scala:

package bar

object MyClass {
  def print(str1: String, str2: Option[String] = None) = {
    println(s"$str1 $str2")
  }
}

MyApp.scala:

package bar

object MyApp extends App {
  MyClass.print("hello")
}

Run ./pants run :MyApp-bin:

...
                   [1/2] Compiling 1 zinc source in 1 target (//:MyLibrary).
19:46:32 00:11       [compile]
...
19:47:35 01:14         [zinc]
                       [info] Compiling 1 Scala source to /<dir>/.pants.d/compile/zinc/c95338cea9cc/.MyLibrary/current/classes...
                       [info] Compile success at May 15, 2017 7:47:40 PM [4.662s]
                        
                   [2/2] Compiling 1 zinc source in 1 target (//:MyApp).
19:47:40 01:19       [compile]
                     
19:47:40 01:19         [zinc]
                       [info] Compiling 1 Scala source to /<dir>/.pants.d/compile/zinc/c95338cea9cc/.MyApp/current/classes...
                       [info] Compile success at May 15, 2017 7:47:41 PM [1.013s]
                       
19:47:41 01:20     [jvm-dep-check]
19:47:41 01:20   [run]
19:47:41 01:20     [py]
19:47:41 01:20     [jvm]
19:47:41 01:20       [run]
hello None

               Waiting for background workers to finish.
19:47:41 01:20   [complete]
               SUCCESS

Now edit MyClass.scala to add an optional 3rd argument to the method:

package bar

object MyClass {
  def print(str1: String, str2: Option[String] = None, str3: Option[String] = Some("foo")) = {
    println(s"$str1 $str2 $str3")
  }
}

And re-run ./pants run :MyApp-bin:

...
                   [1/2] Compiling 1 zinc source in 1 target (//:MyLibrary).
19:48:19 00:01       [compile]
                     
19:48:19 00:01         [zinc]
                       [info] Compiling 1 Scala source to /<dir>/.pants.d/compile/zinc/c95338cea9cc/.MyLibrary/current/classes...
                       [info] Compile success at May 15, 2017 7:48:19 PM [0.723s]
                        
                   [2/2] Compiling 1 zinc source in 1 target (//:MyApp).
19:48:19 00:01       [compile]
                     
19:48:19 00:01         [zinc]
                       [info] Compile success at May 15, 2017 7:48:19 PM [0.013s]
                       
19:48:19 00:01     [jvm-dep-check]
19:48:19 00:01   [run]
19:48:19 00:01     [py]
19:48:19 00:01     [jvm]
19:48:19 00:01       [run]
Exception in thread "main" java.lang.NoSuchMethodError: bar.MyClass$.print(Ljava/lang/String;Lscala/Option;)V
	at bar.MyApp$.delayedEndpoint$bar$MyApp$1(MyApp.scala:4)
	at bar.MyApp$delayedInit$body.apply(MyApp.scala:3)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at bar.MyApp$.main(MyApp.scala:3)
	at bar.MyApp.main(MyApp.scala)

FAILURE: java bar.MyApp ... exited non-zero (1)


19:48:20 00:02   [complete]

Interestingly, in futzing around with this repro, when I got a shared cache server involved, after the 2nd run's NoSuchMethodError, running ./pants clean-all, and then running again, the cache server had both compilations and then I could freely flip back and forth between versions of MyClass.scala and have run work fine in both cases without a clean-all. Context here is this error put all my build agents out of commission and so thus I was exploring to what extent the shared cache server helped or hindered here.

The text was updated successfully, but these errors were encountered:

stuhood · 2017-05-16T00:40:37Z

In your second run, you'll notice that the compile of //:MyApp did not compile any sources (the "Compiling X Scala sources to DIR" message is not reported by zinc). So this is almost certainly an incremental compile bug in zinc. We're hoping to update zinc during the 1.4.0.dev series (see #4477), which might address this issue and others.

Pants does not write incremental compiles to the buildcache by default (--no-compile-zinc-incremental-caching is set by default), for precisely this reason. So unless you enabled that option, the results of a incremental compile should never end up in any cache.

But you should always either 1) run ./pants clean-all between CI builds, or 2) set --no-compile-zinc-incremental (note: different option). If you disable incremental compile entirely, you rule out these issues.

zinc incremental compiles are a deal with the devil which is really only safe to make on developer machines. Whether we should invert the default and disable it by default is an open question.

bhmiller · 2017-05-16T00:51:51Z

Hm, we've got [compile.zinc] incremental: False in our CI builds. Will play around to see if I can get a repro with that on.

bhmiller · 2017-05-16T01:09:16Z

No difference. pants.ini:

[GLOBAL]
pants_version: 1.3.0rc1

[compile.zinc]
incremental: False

First run:

...
21:05:01 00:02   [compile]
21:05:01 00:02     [compile-jvm-prep-command]
21:05:01 00:02       [jvm_prep_command]
21:05:01 00:02     [compile-prep-command]
21:05:01 00:02     [compile]
21:05:01 00:02     [zinc]
21:05:01 00:02       [cache]  
                   No cached artifacts for 2 targets.
                   Invalidated 2 targets.
21:05:01 00:02       [isolation-zinc-pool-bootstrap] 
                   [1/2] Compiling 1 zinc source in 1 target (//:MyLibrary).
21:05:01 00:02       [compile]
                     
21:05:01 00:02         [cache] 
                     No cached artifacts for 1 target.
                     Invalidated 1 target.
21:05:01 00:02         [cache] 
21:05:01 00:02         [bootstrap-compiler-interface]
21:05:01 00:02         [cache] 
21:05:01 00:02         [bootstrap-jar-tool]
21:05:02 00:03         [cache] 
21:05:02 00:03         [bootstrap-jarjar]
21:05:03 00:04         [shade-compiler-interface]
21:05:04 00:05         [cache] 
21:05:04 00:05         [bootstrap-compiler-bridge]
21:05:04 00:05         [cache] 
                     No cached artifacts for 1 target.
                     Invalidated 1 target.
21:05:04 00:05         [cache] 
21:05:04 00:05         [bootstrap-zinc]
21:05:08 00:09         [shade-zinc]
21:05:56 00:57         [cache] 
21:05:56 00:57         [bootstrap-scalac_2_12]
21:05:57 00:58         [zinc]
                       [info] Compiling 1 Scala source to /<dir>/.pants.d/compile/zinc/c95338cea9cc/.MyLibrary/current/classes...
                       [info] Compile success at May 15, 2017 9:06:02 PM [4.817s]
                        
                   [2/2] Compiling 1 zinc source in 1 target (//:MyApp).
21:06:02 01:03       [compile]
                     
21:06:02 01:03         [zinc]
                       [info] Compiling 1 Scala source to /<dir>/.pants.d/compile/zinc/c95338cea9cc/.MyApp/current/classes...
                       [info] Compile success at May 15, 2017 9:06:03 PM [1.042s]
                       
21:06:03 01:04     [jvm-dep-check]
21:06:03 01:04   [pyprep]
21:06:03 01:04     [interpreter]
21:06:13 01:14     [requirements]
21:06:13 01:14     [sources]
21:06:14 01:15   [resources]
21:06:14 01:15     [prepare]
21:06:14 01:15     [services]
21:06:14 01:15   [run]
21:06:14 01:15     [py]
21:06:14 01:15     [jvm]
21:06:14 01:15       [run]
hello None

               Waiting for background workers to finish.
21:06:14 01:15   [complete]
               SUCCESS

Update MyClass and then run:

21:06:46 00:01   [compile]
21:06:46 00:01     [compile-jvm-prep-command]
21:06:46 00:01       [jvm_prep_command]
21:06:46 00:01     [compile-prep-command]
21:06:46 00:01     [compile]
21:06:46 00:01     [zinc]
21:06:46 00:01       [cache]  
                   No cached artifacts for 2 targets.
                   Invalidated 2 targets.
21:06:46 00:01       [isolation-zinc-pool-bootstrap] 
                   [1/2] Compiling 1 zinc source in 1 target (//:MyLibrary).
21:06:46 00:01       [compile]
                     
21:06:46 00:01         [zinc]
                       [info] Compiling 1 Scala source to /<dir>/.pants.d/compile/zinc/c95338cea9cc/.MyLibrary/current/classes...
                       [info] Compile success at May 15, 2017 9:06:46 PM [0.786s]
                        
                   [2/2] Compiling 1 zinc source in 1 target (//:MyApp).
21:06:46 00:01       [compile]
                     
21:06:46 00:01         [zinc]
                       [info] Compiling 1 Scala source to /<dir>/.pants.d/compile/zinc/c95338cea9cc/.MyApp/current/classes...
                       [info] Compile success at May 15, 2017 9:06:47 PM [0.701s]
                       
21:06:47 00:02     [jvm-dep-check]
21:06:47 00:02   [run]
21:06:47 00:02     [py]
21:06:47 00:02     [jvm]
21:06:47 00:02       [run]
Exception in thread "main" java.lang.NoSuchMethodError: bar.MyClass$.print(Ljava/lang/String;Lscala/Option;)V
	at bar.MyApp$.delayedEndpoint$bar$MyApp$1(MyApp.scala:4)
	at bar.MyApp$delayedInit$body.apply(MyApp.scala:3)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at bar.MyApp$.main(MyApp.scala:3)
	at bar.MyApp.main(MyApp.scala)

FAILURE: java bar.MyApp ... exited non-zero (1)


               Waiting for background workers to finish.
21:06:48 00:03   [complete]

So, am seeing the Compiling X Scala sources to DIR call you mentioned was previously missing, but still seeing the error.

bhmiller · 2017-05-16T01:15:07Z

Interestingly, with incremental compilation off, ./pants clean-all no longer fixes the problem. I have to manually delete the .pants.d and .cache directories to get run to work again.

stuhood · 2017-05-16T01:26:09Z

Yikes.

baroquebobcat · 2017-05-18T18:51:28Z

I'm going to pull this into an integration test to see if I can check it against other pants versions.

baroquebobcat · 2017-05-18T19:18:42Z

got a reproduction working. Or, to be more precise, not working.

baroquebobcat · 2017-05-18T19:29:30Z

testing on pants 1.2.1

baroquebobcat · 2017-05-18T19:57:54Z

Testing on pants 1.2.1. It looks like 1.2.1 doesn't exhibit the behavior.

baroquebobcat · 2017-05-18T19:59:44Z

now trying 1.3.0.dev4

baroquebobcat · 2017-05-18T21:46:30Z

I've narrowed it down to a change between 1.3.0.dev16 and .dev19

baroquebobcat · 2017-05-18T23:12:44Z

Between .dev18 and .dev19

baroquebobcat · 2017-05-18T23:14:22Z

It's this one 94775d9

### Problem With the 1.0.0-X5 version of sbt, there's a case where having use-classpath-jars enabled causes runtime failures because zinc/sbt incorrectly detect interface changes. (See #4596). ### Solution Revert the commit making use-classpath-jars the default: 94775d9. ### Result #4596 is fixed.

stuhood · 2017-05-19T00:56:54Z

Fixed by #4607

### Problem With the 1.0.0-X5 version of sbt, there's a case where having use-classpath-jars enabled causes runtime failures because zinc/sbt incorrectly detect interface changes. (See #4596). ### Solution Revert the commit making use-classpath-jars the default: 94775d9. ### Result #4596 is fixed.

mateor · 2017-05-26T22:08:48Z

Arg...too bad. Is the hope that the 1.4.0 zinc upgrade could make this a possibility again?

stuhood · 2017-05-26T22:15:49Z

Yea, hopefully. See #4477. Also, there is some movement on this upstream.

### Problem We didn't have test coverage for source compatible, binary incompatible incremental changes for compiles with zinc. ### Solution Add a test case that covers it. ### Result If #4596 comes up again, it should be caught in the tests.

stuhood added bug jvm labels May 16, 2017

stuhood added this to the 1.3.0 milestone May 16, 2017

stuhood added the needs-cherrypick label May 16, 2017

baroquebobcat self-assigned this May 16, 2017

baroquebobcat mentioned this issue May 18, 2017

Revert "Enable --compile-zinc-use-classpath-jars by default (#4525)" #4607

Merged

baroquebobcat added a commit to baroquebobcat/pants that referenced this issue May 18, 2017

Regression test for pantsbuild#4596

30725da

baroquebobcat mentioned this issue May 18, 2017

Regression test for #4596 #4608

Merged

stuhood closed this as completed May 19, 2017

stuhood removed the needs-cherrypick label May 19, 2017

stuhood mentioned this issue Jun 7, 2017

Reapply the zinc upgrade in 1.6.0.dev #4477

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error compiling code that is source compatible, but binary incompatible, across library boundaries #4596

Error compiling code that is source compatible, but binary incompatible, across library boundaries #4596

bhmiller commented May 16, 2017

stuhood commented May 16, 2017 •

edited

Loading

bhmiller commented May 16, 2017

bhmiller commented May 16, 2017

bhmiller commented May 16, 2017

stuhood commented May 16, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017 •

edited

Loading

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

stuhood commented May 19, 2017

mateor commented May 26, 2017

stuhood commented May 26, 2017

Error compiling code that is source compatible, but binary incompatible, across library boundaries #4596

Error compiling code that is source compatible, but binary incompatible, across library boundaries #4596

Comments

bhmiller commented May 16, 2017

Repro

stuhood commented May 16, 2017 • edited Loading

bhmiller commented May 16, 2017

bhmiller commented May 16, 2017

bhmiller commented May 16, 2017

stuhood commented May 16, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017 • edited Loading

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

baroquebobcat commented May 18, 2017

stuhood commented May 19, 2017

mateor commented May 26, 2017

stuhood commented May 26, 2017

stuhood commented May 16, 2017 •

edited

Loading

baroquebobcat commented May 18, 2017 •

edited

Loading