Emit trait method bodies in statics #5177

retronym · 2016-05-19T05:22:46Z

In 2.12.0-M4 / #5003, we changed the trait encoding to do away with trait implementation classes, in favour of leaving trait methods in place as interface default methods.

However, we've since discovered that we can no longer precisely target a particular statically resolved method with invokespecial due to impedance mismatch between JVM and Scala semantics.

This pull request emits a pair of methods for each concrete trait method. The default method remains, with the same signature as before. However, it now forwards to a same-named static method (with an explicit self param) that contains the method code.

This encoding allows us to precisely target a particular method with invokestatic.

This is binary incompatible, and will require bootstrapping of the modules.

(More details to follow)

retronym · 2016-05-19T08:40:03Z

Module bootstrap progressing well: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/424/console

=> I need to update some tests now.

Testsuite: scala.issues.BytecodeTest
Tests run: 10, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 5.357 sec
------------- Standard Output ---------------
Label(0)
LineNumber(4, Label(0))
VarOp(ALOAD, 1)
VarOp(ALOAD, 1)
Field(GETFIELD, a/B, t, I)
Op(ICONST_1)
Op(IADD)
Field(PUTFIELD, a/B, t, I)
Op(RETURN)
Label(9)
------------- ---------------- ---------------

Testcase: invocationReceiversProtected took 2.053 sec
Testcase: invocationReceivers took 0.213 sec
    FAILED
assertion failed: 
Expected: List(VarOp(ALOAD, 0), Invoke(INVOKESPECIAL, T, clone, ()Ljava/lang/Object;, false), Op(ARETURN))
Actual  : List(VarOp(ALOAD, 0), Invoke(INVOKESTATIC, T, clone, (LT;)Ljava/lang/Object;, false), Op(ARETURN))
junit.framework.AssertionFailedError: assertion failed: 
Expected: List(VarOp(ALOAD, 0), Invoke(INVOKESPECIAL, T, clone, ()Ljava/lang/Object;, false), Op(ARETURN))
Actual  : List(VarOp(ALOAD, 0), Invoke(INVOKESTATIC, T, clone, (LT;)Ljava/lang/Object;, false), Op(ARETURN))
    at scala.tools.nsc.backend.jvm.CodeGenTools$.assertSameCode(CodeGenTools.scala:169)
    at scala.tools.nsc.backend.jvm.CodeGenTools$.assertSameCode(CodeGenTools.scala:167)
    at scala.issues.BytecodeTest.invocationReceivers(BytecodeTest.scala:350)

Testcase: traitMethodForwarders took 0.412 sec
    FAILED
assertion failed: 
Expected: List(VarOp(ALOAD, 0), Invoke(INVOKESPECIAL, T3, f, ()I, false), Op(IRETURN))
Actual  : List(VarOp(ALOAD, 0), Invoke(INVOKESTATIC, T3, f, (LT3;)I, false), Op(IRETURN))
junit.framework.AssertionFailedError: assertion failed: 
Expected: List(VarOp(ALOAD, 0), Invoke(INVOKESPECIAL, T3, f, ()I, false), Op(IRETURN))
Actual  : List(VarOp(ALOAD, 0), Invoke(INVOKESTATIC, T3, f, (LT3;)I, false), Op(IRETURN))
    at scala.tools.nsc.backend.jvm.CodeGenTools$.assertSameCode(CodeGenTools.scala:169)
    at scala.tools.nsc.backend.jvm.CodeGenTools$.assertSameCode(CodeGenTools.scala:167)
    at scala.issues.BytecodeTest$forwarderTestUtils$.checkForwarder(BytecodeTest.scala:224)
    at scala.issues.BytecodeTest.traitMethodForwarders(BytecodeTest.scala:285)

Testcase: bytecodeForBranches took 0.643 sec
Testcase: noTraitMethodForwardersForOverloads took 0.087 sec
Testcase: t6288bJumpPosition took 0.328 sec
Testcase: t8731 took 0.123 sec
Testcase: t8926 took 0.078 sec
Testcase: traitMethodForwardersForJavaDefaultMethods took 0.125 sec
    FAILED
assertion failed: 
Expected: List(VarOp(ALOAD, 0), Invoke(INVOKESPECIAL, T2, f, ()I, false), Op(IRETURN))
Actual  : List(VarOp(ALOAD, 0), Invoke(INVOKESTATIC, T2, f, (LT2;)I, false), Op(IRETURN))
junit.framework.AssertionFailedError: assertion failed: 
Expected: List(VarOp(ALOAD, 0), Invoke(INVOKESPECIAL, T2, f, ()I, false), Op(IRETURN))
Actual  : List(VarOp(ALOAD, 0), Invoke(INVOKESTATIC, T2, f, (LT2;)I, false), Op(IRETURN))
    at scala.tools.nsc.backend.jvm.CodeGenTools$.assertSameCode(CodeGenTools.scala:169)
    at scala.tools.nsc.backend.jvm.CodeGenTools$.assertSameCode(CodeGenTools.scala:167)
    at scala.issues.BytecodeTest$forwarderTestUtils$.checkForwarder(BytecodeTest.scala:224)
    at scala.issues.BytecodeTest.traitMethodForwardersForJavaDefaultMethods(BytecodeTest.scala:343)

Testcase: specialInvocationReceivers took 0.056 sec

retronym · 2016-05-19T12:29:34Z

Another swing at the bootstrap: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/426/

retronym · 2016-05-20T06:12:24Z

@lrytz I'm a bit stuck trying to teach the inliner about the new setup. I'm working on InlinerTest.inlineArrayForeachEquivalent. Now, the invokevirtual is inlined, but only inlined the forwarder (an invokestatic), and it doesn't enable elimination of the closure.

I found the post inline request feature of the inliner, which seems to do the right thing. See the last commit for a hacky demo of this: InlinerTest.inlineArrayForeachEquivalent works after that change.

Is that the right place?
What's the right way to express isTraitVirtualForwarder ? Is that the sort of thing we need to add the the InlineInfo attribute, or could we figure it out from the bytecode + existing metadata available in the backend?

lrytz · 2016-05-20T06:17:01Z

i'll take a look today

lrytz · 2016-05-20T20:48:08Z

@retronym i think your approach works fine, here's a WIP that identifies trait forwarders invoking static impl meethods well enough (i'd say): lrytz@9f50d64 (needs cleaning up)

lrytz · 2016-05-20T20:50:15Z

lrytz@9f50d64 is the latest WIP

retronym · 2016-05-23T06:29:20Z

@lrytz thanks for that. I've cleaned it up by:

making it check that we're calling a Scala trait, not just any interface default method
applying it to the other heuristic strategies
refactoring the code a little bit.

I'll continue now with the job of adapting the JUnit tests for the optimizer, and then seeing how partest goes.

lrytz · 2016-05-23T07:59:52Z

src/compiler/scala/tools/nsc/backend/jvm/opt/BytecodeUtils.scala

@@ -123,6 +123,8 @@ object BytecodeUtils {

  def isReference(t: Type) = t.getSort == Type.OBJECT || t.getSort == Type.ARRAY

+  def isScala(classNode: ClassNode) = classNode.attrs != null && classNode.attrs.asScala.exists(a => a.`type` == BTypes.ScalaAttributeName || a.`type` == BTypes.ScalaSigAttributeName)


i just was a bit confused why this works, because the scala pickle is stored in the ScalaSignature annotation nowadays (no longer in the ScalaSig attribute), but it turns out we still put the attribute. do you know if we have a test for that?

scala/src/compiler/scala/tools/nsc/backend/jvm/BCodeHelpers.scala

Lines 1119 to 1121 in 2e40aa5

val ssa = getAnnotPickle(bType.internalName, moduleClass.companionSymbol)

mirrorClass.visitAttribute(if (ssa.isDefined) pickleMarkerLocal else pickleMarkerForeign)

emitAnnotations(mirrorClass, moduleClass.annotations ++ ssa)

The switch to annotation was made in ef1577a. I don't know whether leaving the attribute was intentional or an oversight.

retronym · 2016-05-25T02:24:08Z

@lrytz I found a different approach to fixing trait inlining that seems to work better, I changed the backend to use the inline metadata of the forward method for the static method, too.

This has got all the JUnit tests working again. I'm running a bootstrap build again to run the partest suite, too: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-bootstrap/433/console

lrytz · 2016-05-25T07:56:36Z

I found a problem:

➜  sandbox git:(pr5177) ✗ cat Test.scala
trait T {
  def f: Int = 0
  def f(t: T): Int = 1
}

object Test {
  def main(args: Array[String]): Unit = {
    new T { }
  }
}
➜  sandbox git:(pr5177) ✗ qsc Test.scala
➜  sandbox git:(pr5177) ✗ qs Test
java.lang.ClassFormatError: Duplicate method name&signature in class file T
    at java.lang.ClassLoader.defineClass1(Native Method)

lrytz · 2016-05-25T08:00:18Z

@retronym for your fix to the inliner metadata, I think it's a bit of a hack; we should start by adding an entry for the newly-generated static method to the ClassBType's methodInfos. I'll take a look.

lrytz · 2016-05-25T08:24:10Z

to expand: what happens in your fix is that the static method is inlined into the forwarder. the only reason this didn't happen before is because the MethodInlineInfo is missing. this way, inlining a call of the default method yields the actual implementation, enabling closure elimination.

We did rely on the same mechanics before, when we still had trait impl classes.

lrytz · 2016-05-25T13:27:24Z

@retronym I implemented the change to represent the static methods in the ScalaInlineInfo: retronym/scala@topic/trait-statics...lrytz:pr5177

There's first commit is necessary to fix a cross-talk between tests, the test introduced in the last commit would cause some other tests to fail otherwise.

The second commit is a similar fix, but not related to this PR.

lrytz · 2016-05-25T13:48:03Z

@retronym the implementation of isTraitMethodRequiringStaticImpl(sym: Symbol) looks a bit too brittle: retronym@e3535b4#diff-7a3aa9d1c9acd44da15dc536d4760e5aR53

The problem is that when building the InlineInfo, we don't have the method's DefDef at hand, so we cannot check dd.rhs.isEmpty. At least we can check that the two are consistent during code gen (retronym@e3535b4#diff-492148adc97fa1758c101777df204a20R493), but it's not a beauty..

retronym · 2016-05-26T06:00:30Z

@lrytz I'm going to try to move things back to mixin. To start with, I'm trying to add a symbol attachment with the isTraitMethodRequiringStaticImpl for the backend to pickup. The next step would be to create the trees for the static and forwarder pair at that point, too.

sjrd · 2016-05-26T06:13:53Z

The next step would be to create the trees for the static and forwarder pair at that point, too.

Oh. That would be a pity for Scala.js. the JS backend is very happy with the current trees. It doesn't need those static methods.

lrytz · 2016-05-26T06:27:11Z

I'm going to try to move things back to mixin

@retronym how about just adding $init to the static method names?

adriaanm · 2016-06-07T19:51:00Z

Great! Let's get that STARR published (I agree maven is the best location). (EDIT: I also experimented with publishing internally, but we'll go the same route as last time -- via maven)

lrytz · 2016-06-08T13:34:32Z

I tried to see how the community build performs here, but couldn't get it running.

The problem is that we fix scala-xml and scala-partest to 2.12.0-M4, which is not binary compatible. The result is crashes when using scaladoc:

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/416/consoleFull

[sbt-testng-interface] java.lang.NoSuchMethodError: scala.Product.$init$()V
[sbt-testng-interface]  at scala.util.parsing.json.JSONArray.<init>(Parser.scala:103)
[sbt-testng-interface]  at scala.tools.nsc.doc.html.page.IndexScript.$anonfun$packages$1(IndexScript.scala:54)
[sbt-testng-interface]  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
[sbt-testng-interface]  at scala.collection.immutable.Map$Map3.foreach(Map.scala:176)
[sbt-testng-interface]  at scala.collection.TraversableLike.map$(TraversableLike.scala:234)
[sbt-testng-interface]  at scala.collection.TraversableLike.map(TraversableLike.scala:227)
[sbt-testng-interface]  at scala.tools.nsc.doc.html.page.IndexScript.<init>(IndexScript.scala:29)

I tried a few hacks, including

comment-out extra.commands: "set scalaVersion := \"2.12.0-M4\"", but then it just seems to use what ever scala version is specified in the module's build file (2.12.0-M2).
manually set extra.commands: "set scalaVersion := \"2.12.0-dbuildxe6b32a79ee3b9b29b840ae9d4e9a0bc70fddbe92\"", but that didn't work, that scala version could not be resolved while trying to build the modules

Note that #4851 would fix the problem for scaladoc and parser-combinators, but we might still have it for projects using xml or parser-combinators in other ways.

retronym · 2016-06-09T01:11:45Z

Following the "bootstrap plan" we used in #5003

Rebased and dropped commits already present in 2.12.x
bootstrap build of the last commit with parameters to publish to a staging repository on Sonatype.
Community build (for expediency, based on modules built on 2.12.0-213c6d3-nightly, yesterdays version of this PR. Community build config in retronym/community-builds:topic/trait-statics

This corrects an error in the change to the trait encoding in scala#5003: getters in traits should have empty bodies and be emitted as abstract. ``` % ~/scala/2.12.0-M4/bin/scalac sandbox/test.scala && javap -c T Compiled from "test.scala" public interface T { public abstract void T$_setter_$x_$eq(int); public int x(); Code: 0: aload_0 1: invokeinterface #15, 1 // InterfaceMethod x:()I 6: ireturn public int y(); Code: 0: aload_0 1: invokeinterface #20, 1 // InterfaceMethod y:()I 6: ireturn public void y_$eq(int); Code: 0: aload_0 1: iload_1 2: invokeinterface #24, 2 // InterfaceMethod y_$eq:(I)V 7: return public void $init$(); Code: 0: aload_0 1: bipush 42 3: invokeinterface #29, 2 // InterfaceMethod T$_setter_$x_$eq:(I)V 8: aload_0 9: bipush 24 11: invokeinterface #24, 2 // InterfaceMethod y_$eq:(I)V 16: return } % qscalac sandbox/test.scala && javap -c T Compiled from "test.scala" public interface T { public abstract void T$_setter_$x_$eq(int); public abstract int x(); public abstract int y(); public abstract void y_$eq(int); public static void $init$(T); Code: 0: aload_0 1: bipush 42 3: invokeinterface #21, 2 // InterfaceMethod T$_setter_$x_$eq:(I)V 8: aload_0 9: bipush 24 11: invokeinterface #23, 2 // InterfaceMethod y_$eq:(I)V 16: return public void $init$(); Code: 0: aload_0 1: invokestatic #27 // Method $init$:(LT;)V 4: return } ```

This partially reverts the fix for SI-5278 made in 7a99c03. The original motivation for this case to avoid bytecode that stretched platform limitations in Android. For super calls to Scala defined trait methods, we won't use `invokespecial`, but rather use `invokestatic` to a static trait implementation method. As such, we can continue to prune redundant Scala interfaces. It might be worth considering removing the pruning of redundant parents altoghether, though: - We no longer include `ScalaObject` as a parent of every class, which was mentioned as a problem in SI-5728. - Scala 2.12 has left Android behind for the time being due to use of Java 8 facilities. - javac doesn't do this, so why should we?

lrytz · 2016-06-14T08:36:14Z

bootstrap build on latest rev https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-bootstrap/467/console

And use this as the target of the default methods or statically resolved super or $init calls. The call-site change is predicated on `-Yuse-trait-statics` as a stepping stone for experimentation / bootstrapping. I have performed this transformation in the backend, rather than trying to reflect this in the view from Scala symbols + ASTs. We also need to add an restriction related to invokespecial to Java parents: to support a super call to one of these to implement a super accessor, the interface must be listed as a direct parent of the class. The static method names has a trailing $ added to avoid duplicate name and signature errors in classfiles.

adriaanm · 2016-06-14T13:09:23Z

Community build using those bootstrapped modules, and fully bootstrapped scala by dbuild (can't skip locker due to binary incompat): https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/432/console

retronym · 2016-06-14T15:01:00Z

The process of restarring will be a tiny bit easier than last time after #5229 is merged.

adriaanm · 2016-06-29T00:46:50Z

Moving to the rebased PR

Emit trait method bodies in statics [rebase of #5177]

scala-jenkins added this to the 2.12.0-M5 milestone May 19, 2016

retronym changed the title ~~WIP Emit trait method bodies in statics~~ WIP Emit trait method bodies in statics [ci: last-only] May 19, 2016

retronym added the WIP label May 19, 2016

retronym self-assigned this May 19, 2016

retronym force-pushed the topic/trait-statics branch 3 times, most recently from 56fe5cd to 77dfcfe Compare May 19, 2016 08:12

retronym force-pushed the topic/trait-statics branch from c50b219 to 0daf45d Compare May 19, 2016 12:18

retronym force-pushed the topic/trait-statics branch from 892f325 to 3577041 Compare May 23, 2016 06:27

lrytz reviewed May 23, 2016
View reviewed changes

retronym force-pushed the topic/trait-statics branch 3 times, most recently from 87400b7 to 8aec58e Compare May 25, 2016 01:08

retronym force-pushed the topic/trait-statics branch from 12ab0f0 to ec6af82 Compare May 26, 2016 06:25

adriaanm added a commit to adriaanm/scala that referenced this pull request Jun 7, 2016

WIP: try starr from scala#5177

99be125

adriaanm added a commit to adriaanm/scala that referenced this pull request Jun 7, 2016

WIP: try starr from scala#5177

0c376e6

adriaanm added a commit to adriaanm/scala that referenced this pull request Jun 7, 2016

WIP: try starr from scala#5177

158f02b

adriaanm added a commit to adriaanm/scala that referenced this pull request Jun 7, 2016

WIP: try starr from scala#5177

847c928

retronym force-pushed the topic/trait-statics branch from 213c6d3 to 4e8f2f9 Compare June 9, 2016 01:06

retronym added 5 commits June 9, 2016 11:26

Better diagnostic for optimizer crashes

2b715e9

Remove stray .class file from version control

e529440

Don't minimize parents of java defined syms.

6f0db59

retronym force-pushed the topic/trait-statics branch from 4e8f2f9 to 6f7f32d Compare June 9, 2016 01:29

retronym force-pushed the topic/trait-statics branch from 6f7f32d to 4d20b81 Compare June 14, 2016 08:40

retronym force-pushed the topic/trait-statics branch from bd66724 to cb57729 Compare June 14, 2016 14:43

retronym force-pushed the topic/trait-statics branch from cb57729 to 4d20b81 Compare June 14, 2016 15:01

adriaanm added the release-notes worth highlighting in next release notes label Jun 18, 2016

adriaanm mentioned this pull request Jun 28, 2016

Emit trait method bodies in statics [rebase of #5177] #5251

Merged

adriaanm closed this Jun 29, 2016

lrytz added a commit that referenced this pull request Jun 29, 2016

Merge pull request #5251 from adriaanm/rebase-5177

79e24d5

Emit trait method bodies in statics [rebase of #5177]

SethTisue removed this from the 2.12.0-M5 milestone Sep 6, 2016

SethTisue removed the release-notes worth highlighting in next release notes label Sep 6, 2016

lrytz mentioned this pull request Sep 16, 2016

Consider eliminating static accessors for trait methods scala/scala-dev#228

Closed

scabug mentioned this pull request Apr 7, 2017

Substantial slowdown in groupBy (all collections) scala/bug#10049

Closed

		@@ -123,6 +123,8 @@ object BytecodeUtils {

		def isReference(t: Type) = t.getSort == Type.OBJECT \|\| t.getSort == Type.ARRAY

		def isScala(classNode: ClassNode) = classNode.attrs != null && classNode.attrs.asScala.exists(a => a.`type` == BTypes.ScalaAttributeName \|\| a.`type` == BTypes.ScalaSigAttributeName)

	val ssa = getAnnotPickle(bType.internalName, moduleClass.companionSymbol)
	mirrorClass.visitAttribute(if (ssa.isDefined) pickleMarkerLocal else pickleMarkerForeign)
	emitAnnotations(mirrorClass, moduleClass.annotations ++ ssa)

Emit trait method bodies in statics #5177

Emit trait method bodies in statics #5177

Uh oh!

Conversation

retronym commented May 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

retronym commented May 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

retronym commented May 19, 2016

Uh oh!

retronym commented May 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrytz commented May 20, 2016

Uh oh!

lrytz commented May 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrytz commented May 20, 2016

Uh oh!

retronym commented May 23, 2016

Uh oh!

lrytz May 23, 2016

Choose a reason for hiding this comment

Uh oh!

retronym May 24, 2016

Choose a reason for hiding this comment

Uh oh!

retronym commented May 25, 2016

Uh oh!

lrytz commented May 25, 2016

Uh oh!

lrytz commented May 25, 2016

Uh oh!

lrytz commented May 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrytz commented May 25, 2016

Uh oh!

lrytz commented May 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

retronym commented May 26, 2016

Uh oh!

sjrd commented May 26, 2016

Uh oh!

lrytz commented May 26, 2016

Uh oh!

adriaanm commented Jun 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrytz commented Jun 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

retronym commented Jun 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrytz commented Jun 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriaanm commented Jun 14, 2016

Uh oh!

retronym commented Jun 14, 2016

Uh oh!

adriaanm commented Jun 29, 2016

Uh oh!

Uh oh!

retronym commented May 19, 2016 •

edited

Loading

retronym commented May 19, 2016 •

edited

Loading

retronym commented May 20, 2016 •

edited

Loading

lrytz commented May 20, 2016 •

edited

Loading

lrytz commented May 25, 2016 •

edited

Loading

lrytz commented May 25, 2016 •

edited

Loading

adriaanm commented Jun 7, 2016 •

edited

Loading

lrytz commented Jun 8, 2016 •

edited

Loading

retronym commented Jun 9, 2016 •

edited

Loading

lrytz commented Jun 14, 2016 •

edited

Loading