Confusing debugger line numbers emitted by compiler around lambdas (change of behavior since Scala 2)

[ghost]

Compiler version

3.1.2 (but valid for all Scala 3 versions)

Minimized code

object main {
  def main(args: Array[String]): Unit = {
    Array(1).foreach { n =>
      val x = 123
      println(n)
    }
  }
}

Inspecting the bytecode produced by Scala 2 (matches between Scala 2.12 and Scala 2.13)

  public main([Ljava/lang/String;)V
    // parameter final  args
   L0
    GETSTATIC scala/collection/ArrayOps$.MODULE$ : Lscala/collection/ArrayOps$;
   L1
    LINENUMBER 3 L1
    GETSTATIC scala/Predef$.MODULE$ : Lscala/Predef$;
    ICONST_1
    NEWARRAY T_INT
    DUP
    ICONST_0
    ICONST_1
    IASTORE
    INVOKEVIRTUAL scala/Predef$.intArrayOps ([I)Ljava/lang/Object;
<---------- Line added by me, for emphasis, INVOKEDYNAMIC instruction is a part of the L1 label above
    INVOKEDYNAMIC apply$mcVI$sp()Lscala/runtime/java8/JFunction1$mcVI$sp; [
      // handle kind 0x6 : INVOKESTATIC
      java/lang/invoke/LambdaMetafactory.altMetafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;
      // arguments:
      (I)V, 
      // handle kind 0x6 : INVOKESTATIC
      main$.$anonfun$main$1(I)V, 
      (I)V, 
      1
    ]
    INVOKEVIRTUAL scala/collection/ArrayOps$.foreach$extension (Ljava/lang/Object;Lscala/Function1;)V
    RETURN
   L2
    LOCALVARIABLE this Lmain$; L0 L2 0
    LOCALVARIABLE args [Ljava/lang/String; L0 L2 1
    MAXSTACK = 6
    MAXLOCALS = 2

Notice the line added by me in the output. It signifies that the INVOKEDYNAMIC instruction which runs the lambda expression is part of the L1 label, where the array is initialized. This means that the creation of the array and the INVOKEDYNAMIC instruction are part of the same label. This is important later.

Inspecting the bytecode produced by Scala 3

  public main([Ljava/lang/String;)V
    // parameter final  args
   L0
    LINENUMBER 2 L0
    LINENUMBER 3 L0
    GETSTATIC scala/Predef$.MODULE$ : Lscala/Predef$;
    ICONST_1
    NEWARRAY T_INT
    DUP
    ICONST_0
    ICONST_1
    IASTORE
    INVOKEVIRTUAL scala/Predef$.intArrayOps ([I)Ljava/lang/Object;
    ASTORE 2
    GETSTATIC scala/collection/ArrayOps$.MODULE$ : Lscala/collection/ArrayOps$;
    ALOAD 2
   L1
    LINENUMBER 5 L1
    ALOAD 0
<---------- Line added by me, for emphasis, INVOKEDYNAMIC instruction is a part of the L1 label above
    INVOKEDYNAMIC apply$mcVI$sp(Lmain$;)Lscala/runtime/java8/JFunction1$mcVI$sp; [
      // handle kind 0x6 : INVOKESTATIC
      java/lang/invoke/LambdaMetafactory.altMetafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;
      // arguments:
      (I)V, 
      // handle kind 0x7 : INVOKESPECIAL
      main$.main$$anonfun$1(I)V, 
      (I)V, 
      1
    ]
    INVOKEVIRTUAL scala/collection/ArrayOps$.foreach$extension (Ljava/lang/Object;Lscala/Function1;)V
    RETURN
   L2
    LOCALVARIABLE this Lmain$; L0 L2 0
    LOCALVARIABLE args [Ljava/lang/String; L0 L2 1
    MAXSTACK = 5
    MAXLOCALS = 3

Notice the line added by me in the output. It signifies that the INVOKEDYNAMIC instruction which runs the lambda expression is part of the L1 label, but in this case, the array is initialized in the L0 label. Thus, the creation of the label and the INVOKEDYNAMIC instruction are parts of different labels, contrasting the Scala 2 case.

Now, this is not an inherent problem in and of itself.

The problem comes from the fact that the different Labels (L0 and L1 in the Scala 3 case) have different LINENUMBERs attached to them (2 and 3 for L0 and 5 for L1). Thus, the INVOKEDYNAMIC instruction is attached to LINENUMBER 5.

Compared to the Scala 2 case, the INVOKEDYNAMIC instruction does not have its own LINENUMBER. (Also notice that the whole bytecode of the main method does not mention LINENUMBER 5 anywhere. This will be important later.)

This distinction can clearly be seen in a debugger. I will give examples in both IntelliJ IDEA and Metals and they both have the same behavior, resulting in bad UX.

IntelliJ IDEA Debugger screenshot with Scala 2.13.8

Notice that the debugger is stopped on Line 5, which is in the stack frame of the $anonfun$main$1 method (the lambda body) and that the n = 1 and x = 123` local variables are visible.

Metals Debugger screenshot with Scala 2.13

Same exact situation in Metals, Line 5, stack frame of `$anonfun$main$1` method and `n = 1` and `x = 123` local variables visible.

IntelliJ IDEA Debugger screenshots with Scala 3.1.2

Notice that the debugger is stopped on Line 5, but this Line 5 is in the stack frame of the main method (stemming from the LINENUMBER 5 emitted in the bytecode above). The local variables n = 1 and x = 123 are not visible. (The red message that the local variable n cannot be found stems from this fact, but its presentation is otherwise a filtering bug in the UI and can be ignored).

Resuming the program gets us where we want to be (and used to be with Scala 2). Line 5 of main$$anonfun$1 (the lambda body) with n = 1 and x = 123 local variables visible.

Metals Debugger screenshots with Scala 3.1.2

Notice that the debugger is stopped on Line 5, but again, in the main method, with n = 1 and x = 123 again, not visible.

Again, resuming the program brings us to the proper breakpoint in the lambda body. Line 5 of main$$anonfun$1 with n = 1 and x = 123 local variables visible.

What if we stop on Line 4?

Line 4 is not an ambiguous line (it only exists in the lambda body bytecode) and thus the debugger stops fine on it on the first try. Both in IntelliJ and in Metals. In this case, we're correctly in the stack frame of main$$anonfun$1 and the lambda argument n = 1 is visible (the x = 123 local variable is not initialized yet, and thus not visible, this is by design).

Conclusion

Only the final line of a multi-line lambda is ambiguous, and stopping on it actually stops on 2 breakpoints, with the second one matching the user intentions more closely, resulting in bad UX. From a source code perspective, the final line of a lambda body should be the same as any other, and IMO, not ambiguous in the outer context.

Discussion

I'm not saying that the emitted LINENUMBER is irrelevant and can be dropped, I frankly don't have the whole picture to make a call like that. I would like to know some of the reasoning behind that decision and work with you to arrive at a solution that would benefit the end user the most, in all debuggers.

Thanks for reading and thanks in advance.