[lit-labs/compiler] add element, attribute, boolean, event, property parts #4056

AndrewJakubowicz · 2023-07-28T22:17:46Z

Issue: #189
RFC: lit/rfcs#21
Full prototype PR: #3984

Why

Prior to this PR the compiler only handled child parts. This PR adds all the attribute parts, as well as element parts.

How

The logic is quite similar to what happens in the Template constructor code in lit-html.ts.

Because this is using parse5 and not TreeWalker, there are small differences in the API, but otherwise it's the same.

The compiler also needs to add imports for the various attribute parts, tracked in a separate data structure. During preparation if an attribute part is encountered (for the first time), a unique identifier is assigned for the attribute part constructor. This signals that the constructor needs to be imported and reassigned to the unique identifier. The unique identifier is used to avoid naming collisions.

While implementing this change, I also noticed that I messed up the tsconfig and it wasn't accessing any DOM types – which are needed for parse5 types to work correctly. Hence the modifications to tsconfig.

Test plan

Tested by adding a golden for each individual part. This ensures that a single part only generates a single constructor import. A golden was added containing every part as well.

Note I also manually tested the kitchen sink compiled golden in the lit playground, and checked that each part is correctly handled.

Missing features

Note: while comparing the preparation code to lit-html, note that the prepare phase is not completed by this change. Still to come: raw text elements handling, bindings in comments.

Thank you!

changeset-bot · 2023-07-28T22:17:49Z

🦋 Changeset detected

Latest commit: 54a40ce

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2023-07-28T22:18:01Z

📊 Tachometer Benchmark Results

Summary

nop-update

this-change, tip-of-tree, previous-release: unsure 🔍 -7% - +8% (-1.48ms - +1.68ms)
^{this-change vs tip-of-tree}

render

this-change: 69.80ms - 73.05ms
this-change, tip-of-tree, previous-release: slower ❌ 0% - 9% (0.05ms - 2.51ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -2% - +2% (-1.03ms - +0.80ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -3% - +1% (-1.32ms - +0.42ms)
^{this-change vs tip-of-tree}

update

this-change: 766.17ms - 776.53ms
this-change, tip-of-tree, previous-release: unsure 🔍 -4% - +6% (-2.76ms - +4.50ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -2% - +0% (-2.28ms - +0.55ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -1% - +0% (-10.81ms - +2.08ms)
^{this-change vs tip-of-tree}

update-reflect

this-change: 737.84ms - 747.95ms
this-change, tip-of-tree, previous-release: unsure 🔍 -0% - +1% (-1.76ms - +7.15ms)
^{this-change vs tip-of-tree}

Results

this-change

Browser: chrome-headless 115.0.5790.102
Sample size: 50
Built by: Benchmarks #5138
Commit: 54a40ce

render

Version	Avg time	vs
	69.80ms - 73.05ms	-

update

Version	Avg time	vs
	766.17ms - 776.53ms	-

update-reflect

Version	Avg time	vs
	737.84ms - 747.95ms	-

this-change, tip-of-tree, previous-release

Browser: chrome-headless 115.0.5790.102
Sample size: 30
Built by: Benchmarks #5138
Commit: 54a40ce

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	27.72ms - 29.38ms	-	slower ❌ 0% - 9% 0.05ms - 2.51ms	unsure 🔍 -3% - +6% -0.74ms - +1.66ms
tip-of-tree tip-of-tree	26.36ms - 28.18ms	faster ✔ 0% - 9% 0.05ms - 2.51ms	-	unsure 🔍 -7% - +1% -2.08ms - +0.44ms
previous-release previous-release	27.22ms - 28.96ms	unsure 🔍 -6% - +3% -1.66ms - +0.74ms	unsure 🔍 -2% - +8% -0.44ms - +2.08ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	71.79ms - 77.05ms	-	unsure 🔍 -4% - +6% -2.76ms - +4.50ms	unsure 🔍 -6% - +4% -4.80ms - +2.92ms
tip-of-tree tip-of-tree	71.06ms - 76.05ms	unsure 🔍 -6% - +4% -4.50ms - +2.76ms	-	unsure 🔍 -7% - +3% -5.58ms - +1.97ms
previous-release previous-release	72.53ms - 78.19ms	unsure 🔍 -4% - +6% -2.92ms - +4.80ms	unsure 🔍 -3% - +8% -1.97ms - +5.58ms	-

nop-update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	19.82ms - 22.14ms	-	unsure 🔍 -7% - +8% -1.48ms - +1.68ms	unsure 🔍 -8% - +7% -1.72ms - +1.45ms
tip-of-tree tip-of-tree	19.80ms - 21.94ms	unsure 🔍 -8% - +7% -1.68ms - +1.48ms	-	unsure 🔍 -8% - +6% -1.76ms - +1.29ms
previous-release previous-release	20.03ms - 22.19ms	unsure 🔍 -7% - +8% -1.45ms - +1.72ms	unsure 🔍 -6% - +8% -1.29ms - +1.76ms	-

this-change, tip-of-tree, previous-release

Browser: chrome-headless 115.0.5790.102
Sample size: 50
Built by: Benchmarks #5138
Commit: 54a40ce

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	50.28ms - 51.34ms	-	unsure 🔍 -2% - +2% -1.03ms - +0.80ms	unsure 🔍 -2% - +1% -0.88ms - +0.73ms
tip-of-tree tip-of-tree	50.18ms - 51.67ms	unsure 🔍 -2% - +2% -0.80ms - +1.03ms	-	unsure 🔍 -2% - +2% -0.92ms - +1.00ms
previous-release previous-release	50.28ms - 51.50ms	unsure 🔍 -1% - +2% -0.73ms - +0.88ms	unsure 🔍 -2% - +2% -1.00ms - +0.92ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	108.87ms - 110.82ms	-	unsure 🔍 -2% - +0% -2.28ms - +0.55ms	unsure 🔍 -3% - +0% -2.85ms - +0.19ms
tip-of-tree tip-of-tree	109.69ms - 111.74ms	unsure 🔍 -1% - +2% -0.55ms - +2.28ms	-	unsure 🔍 -2% - +1% -2.02ms - +1.09ms
previous-release previous-release	110.01ms - 112.34ms	unsure 🔍 -0% - +3% -0.19ms - +2.85ms	unsure 🔍 -1% - +2% -1.09ms - +2.02ms	-

this-change, tip-of-tree, previous-release

Browser: chrome-headless 115.0.5790.102
Sample size: 50
Built by: Benchmarks #5138
Commit: 54a40ce

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	44.87ms - 46.16ms	-	unsure 🔍 -3% - +1% -1.32ms - +0.42ms	unsure 🔍 -2% - +2% -0.90ms - +0.93ms
tip-of-tree tip-of-tree	45.38ms - 46.54ms	unsure 🔍 -1% - +3% -0.42ms - +1.32ms	-	unsure 🔍 -1% - +3% -0.40ms - +1.34ms
previous-release previous-release	44.84ms - 46.15ms	unsure 🔍 -2% - +2% -0.93ms - +0.90ms	unsure 🔍 -3% - +1% -1.34ms - +0.40ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	782.24ms - 790.66ms	-	unsure 🔍 -1% - +0% -10.81ms - +2.08ms	unsure 🔍 -1% - +0% -9.61ms - +2.93ms
tip-of-tree tip-of-tree	785.94ms - 795.70ms	unsure 🔍 -0% - +1% -2.08ms - +10.81ms	-	unsure 🔍 -1% - +1% -5.72ms - +7.76ms
previous-release previous-release	785.15ms - 794.44ms	unsure 🔍 -0% - +1% -2.93ms - +9.61ms	unsure 🔍 -1% - +1% -7.76ms - +5.72ms	-

update-reflect

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	783.55ms - 789.73ms	-	unsure 🔍 -0% - +1% -1.76ms - +7.15ms	unsure 🔍 -0% - +1% -3.60ms - +6.23ms
tip-of-tree tip-of-tree	780.73ms - 787.15ms	unsure 🔍 -1% - +0% -7.15ms - +1.76ms	-	unsure 🔍 -1% - +0% -6.37ms - +3.61ms
previous-release previous-release	781.50ms - 789.14ms	unsure 🔍 -1% - +0% -6.23ms - +3.60ms	unsure 🔍 -0% - +1% -3.61ms - +6.37ms	-

_{tachometer-reporter-action v2 for Benchmarks}

packages/labs/compiler/src/lib/ast-fragments.ts

justinfagnani · 2023-07-28T23:08:05Z

packages/labs/compiler/src/lib/ast-fragments.ts

+  const partsObjectBinding = partConstructors
+    .map((part) => {
+      const identifierAlias = partIdentifiers[part];
+      if (!identifierAlias) {


If I'm following this, you allow an object to be provided that renames the part constructors that are imported.

Who decides on the mapping, and is this needed? Can't you create a new unique name for the opt-level scope with TS APIs(factory.createUniqueName) and always rename the constructors?

You are following correctly. The prep stage is where we decide which parts to import (because we know which parts have been encountered). Therefore if a file contains no attribute parts, we do not need to generate any imports.

The unique names are to provide defense against naming collisions – in the edge case where a file already contains an AttributePart identifier.

Can't you just use factory.createUniqueName instead of taking a mapping?

The mapping is created with factory.createUniqueName. I think we're just ironing out where that happens, and therefore what state needs to be associated with the unique identifier.

Currently the order of operations, taken from CompiledTemplatePass.getTransformer() are:

Do a traversal of the AST to locate all the templates. pass.findTemplates

Rewrite the found templates. pass.rewriteTemplates. If an attribute part requires a constructor, the unique name is created here, and added to the part constructor mapping.

Finally, if any part constructors are used, add the part constructor import to the source file.

The benefit of this ordering is that we only add the import if a compiled template needs the constructors, and we easily can only emit the constructors that are needed.

An alternative order could be:

Generate all the unique constructor identifiers up front.

Do steps 1. and 2. above. While rewriting the templates track which constructors are actually used.

Then pass in a set of which ones we want to actually add to the file with the import.

Book keeping is still required if we don't want to naively add all the imports.

Yet another alternative is to not do any bookkeeping, and add the import and all the Attribute part constructors regardless of usage. Although that may require a downstream transform to tree shake or dead code eliminate them if we want the lowest file size. Maybe it's ok to simplify this PR and emit them all.

Ah, and going back to your top questions.

Who decides on the mapping, and is this needed?

The rewriteTemplate method only adds a uniqueIdentifier for an attribute part constructor if it is used. It's not necessarily needed. We could do one of the alternatives in my comment above.

Can't you create a new unique name for the opt-level scope with TS APIs(factory.createUniqueName) and always rename the constructors.

What is the opt-level scope? In the current implementation the constructors are always renamed. They're just instantiated with factory.createUniqueName during the rewriteTemplate pass; so we both track the unique name and which ones are used in a CompiledTemplate. Then we only import and pull the constructors off _$LH that are used.

Sorry, I meant "top-level" scope - ie, a name unique in that scope. But I mispoke a bit because I'm not sure that the TS API has scope-aware unique name generation, and the name has to be unique in all scopes that have templates, not just the top-level scope. Anyway, factory.createUniqueName() is fine.

packages/labs/compiler/src/lib/ast-fragments.ts

augustjk

Looks good to me overall.

packages/labs/compiler/tsconfig.json

packages/labs/compiler/test_files/part_attribute.js

augustjk

Thanks for adding the extra tests!

packages/labs/compiler/src/lib/ast-fragments.ts

rictic · 2023-08-01T17:45:26Z

packages/labs/compiler/src/lib/ast-fragments.ts

+/**
+ * Add Parts constructors import. To avoid identifier collisions, each part
+ * constructor is mapped to a unique identifier that must be provided. If no
+ * identifier is provided for a part, then the part will not be emitted.


If no identifier is provided for a part, then the part will not be emitted.

what does that mean? should we instead throw in that case?

This means that if a certain part constructor does not have a unique identifier provided, we ignore it and don't pull it off _$LH.

This prevents compiled files containing every part constructor. We only import what is used.

Oh, I see. I initially read it like "in [this case], [this parameter] is ignored", which set off my warning bells, but it's more like "add imports as requested by the parameters". The only things that are ignored are the values which aren't requested, i.e. that aren't in the attributePartConstructorNameMap

Exactly! I'll reword the comment to make that clearer

Reworded in 54a40ce

packages/labs/compiler/src/lib/ast-fragments.ts

packages/labs/compiler/src/lib/template-transform.ts

rictic · 2023-08-01T18:36:10Z

packages/labs/compiler/src/lib/ast-fragments.ts

+/**
+ * Add Parts constructors import. To avoid identifier collisions, each part
+ * constructor is mapped to a unique identifier that must be provided. If no
+ * identifier is provided for a part, then the part will not be emitted.


Oh, I see. I initially read it like "in [this case], [this parameter] is ignored", which set off my warning bells, but it's more like "add imports as requested by the parameters". The only things that are ignored are the values which aren't requested, i.e. that aren't in the attributePartConstructorNameMap

rictic · 2023-08-01T18:40:34Z

packages/labs/compiler/src/lib/ast-fragments.ts

+/**
+ * Add Parts constructors import. To avoid identifier collisions, each part
+ * constructor is mapped to a unique identifier that must be provided. If no
+ * identifier is provided for a part, then the part will not be emitted.
+ *
+ * This uses a namespace import for g3 compatibility.


Consider this rewrite:

Suggested change

/**

* Add Parts constructors import. To avoid identifier collisions, each part

* constructor is mapped to a unique identifier that must be provided. If no

* identifier is provided for a part, then the part will not be emitted.

*

* This uses a namespace import for g3 compatibility.

/**

* Add an import for the needed part constructors.

*

* By the time this function is called, we've already transformed the templates in the

* file, and discovered which part constructors we need, and those transforms have

* generated unique identifiers for each, so this function's job is to make sure the

* requested constructors are available as those identifiers.

*

* This uses a namespace import for g3 compatibility.

AndrewJakubowicz added 4 commits July 28, 2023 15:02

add ast-fragment changes. does not compile.

0c7d154

completed adding parts

e9e3b15

add empty changeset

1822026

Add comment to clear up the different nodeIndex.

ce64feb

AndrewJakubowicz requested a review from kevinpschaaf as a code owner July 28, 2023 22:17

AndrewJakubowicz requested review from augustjk and rictic July 28, 2023 22:17

update comments

b27f2a2

justinfagnani reviewed Jul 28, 2023

View reviewed changes

AndrewJakubowicz added 2 commits July 28, 2023 16:44

Address code review feedback

5e8a5b1

comment attributePartConstructorNames

727ba95

augustjk approved these changes Jul 31, 2023

View reviewed changes

packages/labs/compiler/tsconfig.json Outdated Show resolved Hide resolved

packages/labs/compiler/test_files/part_attribute.js Show resolved Hide resolved

code review feedback – add more attribute value cases

850bbcc

augustjk approved these changes Aug 1, 2023

View reviewed changes

rictic reviewed Aug 1, 2023

View reviewed changes

rictic code review feedback: improve the types a lot

25b2e7b

rictic reviewed Aug 1, 2023

View reviewed changes

rictic approved these changes Aug 1, 2023

View reviewed changes

AndrewJakubowicz added 2 commits August 1, 2023 11:42

update parse5/tools to 0.2.0 thank you @43081j

f1f6f77

clarify the attribute part imports comment

54a40ce

AndrewJakubowicz merged commit ff56677 into main Aug 1, 2023

AndrewJakubowicz deleted the compiler-add-attribute-parts branch August 1, 2023 19:10

[lit-labs/compiler] add element, attribute, boolean, event, property parts #4056

[lit-labs/compiler] add element, attribute, boolean, event, property parts #4056

Uh oh!

Conversation

AndrewJakubowicz commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Test plan

Missing features

Uh oh!

changeset-bot bot commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Jul 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Tachometer Benchmark Results

Summary

nop-update

render

update

update-reflect

Results

render

update

update-reflect

render

update

nop-update

render

update

render

update

update-reflect

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndrewJakubowicz Jul 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndrewJakubowicz Jul 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

augustjk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

augustjk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AndrewJakubowicz commented Jul 28, 2023 •

edited

Loading

changeset-bot bot commented Jul 28, 2023 •

edited

Loading

github-actions bot commented Jul 28, 2023 •

edited

Loading

AndrewJakubowicz Jul 29, 2023 •

edited

Loading

AndrewJakubowicz Jul 29, 2023 •

edited

Loading