Trait encoding vs the persnickety JVM 9 verifier: final fields only updatable from within <init>

[retronym]

Final Fields

JVM 9 closes a loophole in the verifier around final field updates outside of <init>. It is predicated on the new classfile version for backwards compat. Some JIT optimizations are disabled for such almost-final fields (at least, FoldStableValues).

Scala's trait encoding violates this rule. Below, we assign to x in T$_setter_$x_$eq.

⚡ scalac $(f "trait T { final val x: Int = 42 }; class C extends T") && javap -c -private C T
Compiled from "a.scala"
public class C implements T {
  private final int x;

  public final int x();
    Code:
       0: aload_0
       1: getfield      #15                 // Field x:I
       4: ireturn

  public final void T$_setter_$x_$eq(int);
    Code:
       0: aload_0
       1: iload_1
       2: putfield      #15                 // Field x:I
       5: return

  public C();
    Code:
       0: aload_0
       1: invokespecial #24                 // Method java/lang/Object."<init>":()V
       4: aload_0
       5: invokestatic  #28                 // InterfaceMethod T.$init$:(LT;)V
       8: return
}
Compiled from "a.scala"
public interface T {
  public abstract void T$_setter_$x_$eq(int);

  public abstract int x();

  public static void $init$(T);
    Code:
       0: aload_0
       1: bipush        42
       3: invokeinterface #18,  2           // InterfaceMethod T$_setter_$x_$eq:(I)V
       8: return
}

Option 1: Status Quo

Don't support emitting classes with the new classfile version. This robs users of a valuable mechanism to fail fast when running classes that depend on the Java 9 standard library. We're also likely to run into a case where some desirable VM feature is only available under the new version.

Option 2: Drop the final modifier in bytecode

We would lose a guarantee under the memory model, but we could get this back with an explicit VarHandle.xxxFence at the end of the constructor of a class with an "almost final" field.

We'd also miss out on some JIT optimizations. These could be regained if/when the @Stable annotation was exposed outside of the JDK, which is under discussion for a future JEP

Option 3: link time trait constructor inlining ?

If we could inline the T.$init$ into C.<init>, but somehow do this at link time so as not to violate separate compilation constraints, we'd be in better shape. C.<init> could use a bootstrap method that delegated to a bootstrap method helper in each trait that builds up the right sequence of calls.

Option 4: rearrange trait encoding, dotty style

Leave the assignments in the class constructor and have it call initializer methods for each val. Don't forget to execute side-effects (statements in the trait body) in the right order! Consider binary compatibility pros/cons!

[retronym]

/cc @adriaanm, this is one of the topics I'm thinking about ahead of JVMLS.

[DarkDimius]

I had a discussion about this behavior with Oracle folks on previous JVM language summit. They have verified that Dotty's trait encoding with initial methods does not have this issue. We perform option 3 at compile time, which has binary compatibility implications, and indeed should be better done via bootstrap method.

…

On 25 July 2017 7:02:33 am Jason Zaugg ***@***.***> wrote: /cc @adriaanm, this is one of the topics I'm thinking about ahead of JVMLS. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #408 (comment)

[DarkDimius]

There is also a similar issue with our module classes:

public final class C$ {
    public static final C$ MODULE$;

    public static {
        new C$();
    }

    public C$() {
        MODULE$ = this;
    }
}

• the $MODULE field is marked final
• we assign it outside of static constructor.

[lrytz]

For modules we no longer mark the field final since 2.12 scala/scala#5322

[sjrd]

FTR, Scala.js uses Option 2. Our IR has always been that strict, that "final" fields (vals in our terminology) are not allowed to be assigned anywhere but within the constructor.

[retronym]

Here's the dotty treatment for comparison.

dotc $(f 'trait T { println("1"); val x: Int = 42; println("2"); }; trait U extends T { override val x = 43 }; class C extends U') && for c in T U C; do cfr-decompiler $c.class; done

import scala.Predef;

public interface T {
    default public void $init$() {
        Predef..MODULE$.println((Object)"2");
    }

    public int x();

    default public int initial$x() {
        Predef..MODULE$.println((Object)"1");
        return 42;
    }
}
/*
 * Decompiled with CFR 0_121.
 */
public interface U
extends T {
    @Override
    default public void $init$() {
    }

    @Override
    public int x();

    @Override
    default public int initial$x() {
        return 43;
    }
}
/*
 * Decompiled with CFR 0_121.
 */
public class C
implements T,
U {
    private final int x;

    public C() {
        T.super.initial$x();
        T.super.$init$();
        this.x = U.super.initial$x();
    }

    @Override
    public int x() {
        return this.x;
    }
}

The binary fragilities here are things like:

• Adding or removing the override val in U requires recompilation of C
• reordering fields in a trait requires recompilation of implementing classes

That's the part that might be addressed by link time generation of the contents of C.<init>. We don't want to "bake in" the particulars of the trait initializers into the subclass constructor.

[retronym]

John Rose's discussion of the possibility of VM support for "larval objects" seem related here.

[retronym]

Also worth noting that Scala 2's trait encoding already has a very similar problem with non-final trait vals: adding/removing the override val in an intermediate trait requires recompilation of a subclass.

trait T {
  val x = 42
  println(x)
}

trait U extends T {
  override val x = 43
  println(x)
}

class C extends U

object Test {
  def main(args: Array[String]): Unit = {
    new C()
  }
}

scalac sandbox/test.scala && scala Test && javap -c T C
0
43
Compiled from "test.scala"
public interface T {
  public abstract void T$_setter_$x_$eq(int);

  public abstract int x();

  public static void $init$(T);
    Code:
       0: aload_0
       1: bipush        42
       3: invokeinterface #18,  2           // InterfaceMethod T$_setter_$x_$eq:(I)V
       8: getstatic     #24                 // Field scala/Predef$.MODULE$:Lscala/Predef$;
      11: aload_0
      12: invokeinterface #26,  1           // InterfaceMethod x:()I
      17: invokestatic  #32                 // Method scala/runtime/BoxesRunTime.boxToInteger:(I)Ljava/lang/Integer;
      20: invokevirtual #36                 // Method scala/Predef$.println:(Ljava/lang/Object;)V
      23: return
}
Compiled from "test.scala"
public class C implements U {
  public int x();
    Code:
       0: aload_0
       1: getfield      #15                 // Field x:I
       4: ireturn

  public void U$_setter_$x_$eq(int);
    Code:
       0: aload_0
       1: iload_1
       2: putfield      #15                 // Field x:I
       5: return

  public void T$_setter_$x_$eq(int);
    Code:
       0: return

  public C();
    Code:
       0: aload_0
       1: invokespecial #25                 // Method java/lang/Object."<init>":()V
       4: aload_0
       5: invokestatic  #31                 // InterfaceMethod T.$init$:(LT;)V
       8: aload_0
       9: invokestatic  #34                 // InterfaceMethod U.$init$:(LU;)V
      12: return
}

[retronym]

I'm not sure that "Option 3: link time trait constructor inlining" is actually feasible. The problems I see are:

• We can't assign a final field via java.lang.invoke, even if this is being done on behalf of an invokedynamic callsite with the class constructor.
• I'm not even sure how to expand an invokedynamic callsite into a statement sequence. There doesn't seem to be a "semi-colon" combinator in MethodHandles, only combinators to create expressions and loops.

[retronym]

I'm not even sure how to expand an invokedynamic callsite into a statement sequence

Here's how to do it: https://gist.github.com/77f2d51b8581d0f85d3d93b5ba195686.

We can't assign a final field via

Confirmed in discussions that this is the case :(

[lrytz]

I cannot follow the indirection from the gist Gist to your laptop's file system :)

[retronym]

D'oh! Updated.

[retronym]

WIP implementation of the change to Fields/Constructors to make the fields non-final and adding the fence is in one of the commits in: scala/scala#6425