Add a desugaring pass between the base linker and the optimizer. #5101

sjrd · 2025-01-04T17:32:14Z

Previously, the emitters and the optimizer all had to perform the same desugaring for LinkTimeProperty nodes. Instead, we now perform the desugaring in a dedicated phase, after the base linker.

The reachability analysis records whether each method needs desugaring or not. We mark those that do so that the desugaring pass knows what to process. Methods that do not require desugaring are not processed, and so incur no additional cost.

No caching is perfored in Desugarer. It processes so few methods that caching makes it (slightly) slower.

The machinery is heavy. It definitely outweighs the benefits in terms of duplication for LinkTimeProperty alone. However, the same machinery will be used to desugar NewLambda nodes. This commit serves as a stepping stone in that direction.

Alternative to #5096.

sjrd · 2025-01-09T10:48:46Z

This is now in a reviewable shape, I think.

The second commit might be over-engineered, though. 😅

gzm0 · 2025-01-17T17:29:06Z

I've had a look at this and the PRs on top of this. I think it fits the overall linker design reasonably nicely.

Some higher level comments:

IMO the way we flag for desugaring is not very nice at the moment. I'm wondering if just member methodsToDesugar: Map[MemberNamespace, Set[MethodName]] in LinkedClass would do the trick.
I'm wondering if we could do More DCE wrt case object equality test #2396 in this phase (fold _ === LoadModule(_) to false in case the module is not instantiated, the analyzer could flag for "desugaring" if it sees this shape, which it will need to recognize anyways).
How do Knowledge accessor utilities #5099 / Refactor: Introduce common caching utilities. #5098 factor into this picture? (I have not looked at them in very much detail).
The second commit might be over-engineered, though. 😅

I think there is value in having a better structured configuration value. However, I would not put any validation logic into it (and just "hard-code" that in the checkers, just like it is now).

I'll likely proceed with a more detailed review when I have time, unless you feel comparing this to #5096 needs more higher level discussion.

sjrd · 2025-01-17T23:35:36Z

IMO the way we flag for desugaring is not very nice at the moment. I'm wondering if just member methodsToDesugar: Map[MemberNamespace, Set[MethodName]] in LinkedClass would do the trick.

I can try something like that.

I'm wondering if we could do More DCE wrt case object equality test #2396 in this phase (fold _ === LoadModule(_) to false in case the module is not instantiated, the analyzer could flag for "desugaring" if it sees this shape, which it will need to recognize anyways).

I don't think we can. The problem is that we don't see the shape _ === Load module(_) at this stage. We see something like LoadModule.equals(y). We have to be able to inline the equals before we see the shape. That's not something we can do at the desugaring phase. We need the optimizer to do it.

How do Knowledge accessor utilities #5099 / Introduce common caching utilities. #5098 factor into this picture? (I have not looked at them in very much detail).

They don't. Those two PRs are unrelated. (I was prompted to look into that because I expected to have to write one more cache with clear after run for the desugaring phase. But it turned out it was so fast that a cache makes it worse, and so there is no relationship with the cache PRs.)

The second commit might be over-engineered, though. 😅

I think there is value in having a better structured configuration value. However, I would not put any validation logic into it (and just "hard-code" that in the checkers, just like it is now).

So basically making it a bare enum-like thing with an ordering, and have the checkers know the acceptable ranges? That's what I started with, but found it unwieldy. I can revert to that state to give a comparison.

I'll likely proceed with a more detailed review when I have time, unless you feel comparing this to #5096 needs more higher level discussion.

No I think it's fine. It seems that this variant wins over #5096 overall.

gzm0 · 2025-01-18T06:42:35Z

So basically making it a bare enum-like thing with an ordering, and have the checkers know the acceptable ranges? That's what I started with, but found it unwieldy. I can revert to that state to give a comparison.

Maybe consider giving the enum ordering to make the range checks easier to grok. Or even expand the features on top of the checker (val newLambdaAllowed = ...).

IMHO the important part is that the logic of what is allowed where is in the checkers and not the phase enum.

sjrd · 2025-01-18T12:27:44Z

I pushed a different version of the CheckingPhase refactoring. I removed logic from CheckingPhase; it is now a pure enum. However, since ClassDefChecker and IRChecker must agree on "which feature is available when", I still kept that logic in a common place, which is now FeatureSet. FeatureSet is private[checked], so it is only an implementation detail of the two checkers. From the checkers' perspective, we use "logical" features now.

This should make things clearer while leaving the responsibility to the checkers, as you mentioned (although factored out in FeatureSet.supportedBy).

sjrd · 2025-01-18T17:58:05Z

IMO the way we flag for desugaring is not very nice at the moment. I'm wondering if just member methodsToDesugar: Map[MemberNamespace, Set[MethodName]] in LinkedClass would do the trick.

I can try something like that.

I pushed a commit thast switches to that scheme. I'm not sure I like it better, though. It bothers me that we have fields in LinkedClass and LinkedTopLevelExport that are only useful between BaseLinker and Desugarer.

gzm0 · 2025-01-19T07:39:47Z

It bothers me that we have fields in LinkedClass and LinkedTopLevelExport that are only useful between BaseLinker and Desugarer.

Yes, I agree, this is not nice in our design. However, I don't think this is anything new: LinkedClass contains a lot of fields only useful for the frontend pipeline (e.g. fieldsRead, all the module dependency information).

Arguably, this is something we should fix. Ironically, one of the easiest way I know is using structural subtyping like in go (producer exposes concrete type, consumer defines an interface of what it needs). Unfortunately, that's not really available to us in Scala.

Other than that, I find the alternative much cleaner :-/ Do you think it makes sense to figure out how to better decouple the fields we have for individual phases?

sjrd · 2025-01-19T08:24:18Z

However, I don't think this is anything new: LinkedClass contains a lot of fields only useful for the frontend pipeline (e.g. fieldsRead, all the module dependency information).

Good point. The new desugaring info does not make the situation qualitatively worse.

This is probably fine as is, then.

We can try to improve that afterwards.

gzm0

More in-depth review of check refactoring.

How big is the effort to split this into a separate PR to ease review?

linker/shared/src/main/scala/org/scalajs/linker/checker/ClassDefChecker.scala

gzm0 · 2025-01-05T13:19:45Z

linker/shared/src/main/scala/org/scalajs/linker/frontend/Desugarer.scala

+        hasRuntimeTypeInfo,
+        fieldsRead,
+        staticFieldsRead,
+        staticDependencies,


Note to self: This will not be correct anymore when we add the NewLambda desguaring: there will be a dependency on the class.

linker/shared/src/main/scala/org/scalajs/linker/checker/ClassDefChecker.scala

linker/shared/src/main/scala/org/scalajs/linker/checker/CheckingPhase.scala

linker/shared/src/main/scala/org/scalajs/linker/checker/FeatureSet.scala

gzm0

Review of the new phase (as far as possible given the current commit situation).

gzm0 · 2025-01-19T09:08:10Z

linker/shared/src/main/scala/org/scalajs/linker/frontend/BaseLinker.scala

@@ -105,6 +107,13 @@ final class BaseLinker(config: CommonPhaseConfig, checkIR: Boolean) {

 private[frontend] object BaseLinker {

+  /** Takes a ClassDef and DCE infos to construct a stripped down LinkedClass.
+   */
+  private[frontend] def refineClassDef(classDef: ClassDef, version: Version,


Do we still need this extra method?

gzm0 · 2025-01-19T09:14:37Z

linker/shared/src/main/scala/org/scalajs/linker/frontend/Desugarer.scala

+      newJSPropertyDef.copy()(jsPropertyDef.version)(newJSPropertyDef.pos)
+    }
+
+    override def transformTopLevelExportDef(exportDef: TopLevelExportDef): TopLevelExportDef = {


IIUC this override is (now) unnecessary?

sjrd · 2025-01-19T16:15:41Z

How big is the effort to split this into a separate PR to ease review?

I can do that, but probably not before Wednesday.

gzm0 · 2025-01-19T16:48:43Z

I can do that, but probably not before Wednesday.

No worries. I will not have time before that for sure.

sjrd · 2025-01-25T10:37:05Z

Rebased on top of #5120.

sjrd · 2025-02-03T05:04:55Z

Rebased. This is now back to being standalone and re-reviewable.

gzm0

Just one comment regarding the name of desugaringInfo.

gzm0 · 2025-02-02T17:47:00Z

linker/shared/src/main/scala/org/scalajs/linker/standard/LinkedClass.scala

@@ -89,3 +92,19 @@ final class LinkedClass(

  def fullName: String = className.nameString
 }
+
+object LinkedClass {
+  final class DesugaringInfo(


This needs somewhere a comment at the very least what the info is. (the fact that desugaring is required).

Otherwise this IMO reads more like "information needed/useful for desugaring".

Ideally this of course goes into the names, but I realize that methodsNeedingDesugaring is probably a bit too long :-/

If only we had Latin's gérondif 😁 We could call it AbSaccharoTemperandum 😅

Perhaps DesugaringRequirements?

linker/shared/src/test/scala/org/scalajs/linker/IRCheckerTest.scala

sjrd · 2025-02-05T09:08:55Z

I went for DesugaringRequirements.

Also I changed how we construct instances of DesugaringRequirements, in a way that provides a fast isEmpty test. This allows Desugarer to entirely bypass classes that need no desugaring. It's now running in 3.5 ms on my machine.

Previously, the emitters and the optimizer all had to perform the same desugaring for `LinkTimeProperty` nodes. Instead, we now perform the desugaring in a dedicated phase, after the base linker. The reachability analysis records whether each method needs desugaring or not. We mark those that do so that the desugaring pass knows what to process. Methods that do not require desugaring are not processed, and so incur no additional cost. No caching is performed in `Desugarer`. It processes so few methods that caching makes it (slightly) *slower*. The machinery is heavy. It definitely outweighs the benefits in terms of duplication for `LinkTimeProperty` alone. However, the same machinery will be used to desugar `NewLambda` nodes. This commit serves as a stepping stone in that direction.

gzm0 · 2025-02-06T14:04:07Z

linker/shared/src/main/scala/org/scalajs/linker/frontend/Desugarer.scala

+  private def desugarClass(linkedClass: LinkedClass): LinkedClass = {
+    import linkedClass._
+
+    if (desugaringRequirements.isEmpty) {


sjrd mentioned this pull request Jan 4, 2025

Add a desugaring step in the base linker. #5096

Closed

gzm0 mentioned this pull request Jan 5, 2025

Remove coreSpec from ModuleSet / LinkingUnit and verifyModuleSet #5104

Merged

sjrd force-pushed the desugaring-separate-phase branch from ae9fdc5 to a738094 Compare January 9, 2025 10:47

sjrd marked this pull request as ready for review January 9, 2025 10:48

sjrd requested a review from gzm0 January 9, 2025 10:48

sjrd force-pushed the desugaring-separate-phase branch from a738094 to 808638e Compare January 18, 2025 12:24

gzm0 requested changes Jan 19, 2025

View reviewed changes

sjrd mentioned this pull request Jan 22, 2025

Refactoring: Generic notion of CheckingPhase for the IR checkers. #5120

Merged

sjrd force-pushed the desugaring-separate-phase branch from a98d50a to b21e091 Compare January 25, 2025 10:36

sjrd force-pushed the desugaring-separate-phase branch from b21e091 to a977798 Compare January 25, 2025 13:13

sjrd force-pushed the desugaring-separate-phase branch from a977798 to 2820d22 Compare February 3, 2025 05:04

sjrd requested a review from gzm0 February 3, 2025 05:04

gzm0 requested changes Feb 4, 2025

View reviewed changes

sjrd force-pushed the desugaring-separate-phase branch from 2820d22 to 4f75c5a Compare February 5, 2025 09:06

sjrd requested a review from gzm0 February 5, 2025 09:09

Bump the version to 1.19.0-SNAPSHOT for the upcoming changes.

38636cf

sjrd force-pushed the desugaring-separate-phase branch from 4f75c5a to 1987d87 Compare February 6, 2025 13:09

gzm0 approved these changes Feb 6, 2025

View reviewed changes

sjrd merged commit 76f7be7 into scala-js:main Feb 6, 2025
3 checks passed

sjrd deleted the desugaring-separate-phase branch February 6, 2025 15:46

sjrd mentioned this pull request Feb 6, 2025

Introduce NewLambda to synthesize instances of SAM types. #5003

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a desugaring pass between the base linker and the optimizer. #5101

Add a desugaring pass between the base linker and the optimizer. #5101

sjrd commented Jan 4, 2025

sjrd commented Jan 9, 2025

gzm0 commented Jan 17, 2025

sjrd commented Jan 17, 2025 •

edited

Loading

gzm0 commented Jan 18, 2025

sjrd commented Jan 18, 2025 •

edited

Loading

sjrd commented Jan 18, 2025

gzm0 commented Jan 19, 2025

sjrd commented Jan 19, 2025

gzm0 left a comment

gzm0 Jan 5, 2025

gzm0 left a comment

gzm0 Jan 19, 2025

gzm0 Jan 19, 2025

sjrd commented Jan 19, 2025

gzm0 commented Jan 19, 2025

sjrd commented Jan 25, 2025

sjrd commented Feb 3, 2025

gzm0 left a comment

gzm0 Feb 2, 2025

sjrd Feb 4, 2025

sjrd commented Feb 5, 2025

gzm0 Feb 6, 2025

Add a desugaring pass between the base linker and the optimizer. #5101

Add a desugaring pass between the base linker and the optimizer. #5101

Conversation

sjrd commented Jan 4, 2025

sjrd commented Jan 9, 2025

gzm0 commented Jan 17, 2025

sjrd commented Jan 17, 2025 • edited Loading

gzm0 commented Jan 18, 2025

sjrd commented Jan 18, 2025 • edited Loading

sjrd commented Jan 18, 2025

gzm0 commented Jan 19, 2025

sjrd commented Jan 19, 2025

gzm0 left a comment

Choose a reason for hiding this comment

gzm0 Jan 5, 2025

Choose a reason for hiding this comment

gzm0 left a comment

Choose a reason for hiding this comment

gzm0 Jan 19, 2025

Choose a reason for hiding this comment

gzm0 Jan 19, 2025

Choose a reason for hiding this comment

sjrd commented Jan 19, 2025

gzm0 commented Jan 19, 2025

sjrd commented Jan 25, 2025

sjrd commented Feb 3, 2025

gzm0 left a comment

Choose a reason for hiding this comment

gzm0 Feb 2, 2025

Choose a reason for hiding this comment

sjrd Feb 4, 2025

Choose a reason for hiding this comment

sjrd commented Feb 5, 2025

gzm0 Feb 6, 2025

Choose a reason for hiding this comment

sjrd commented Jan 17, 2025 •

edited

Loading

sjrd commented Jan 18, 2025 •

edited

Loading