[infra] improve error message for test:common #4059

AndrewJakubowicz · 2023-07-31T20:18:32Z

Context

There is an import rewriting error that can occasionally create a confusing situation for someone working on a PR. The confusing situation is that locally npm run test -w <package> passes, but then it fails in CI.

Most recently this issue blocked: #4013 (comment)

Features

Previous behavior:

The only logged error was: Unexpected dev request in prod mode: /packages/labs/task/development/deep-equals.js

New behavior:

A new log is added with:

❌ There may be a relative import in 'deep-equals_test.js' which is resolving to '/packages/labs/task/development/deep-equals.js'. Ensure the import is a bare module. Reproduce locally with: `MODE=prod npm run test:common -w @lit-internal/tests`

Testing

Low risk change. I tested locally by manually triggering the error.

changeset-bot · 2023-07-31T20:18:35Z

🦋 Changeset detected

Latest commit: 1a6f172

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2023-07-31T20:18:47Z

📊 Tachometer Benchmark Results

Summary

nop-update

this-change, tip-of-tree, previous-release: unsure 🔍 -8% - +6% (-1.96ms - +1.54ms)
^{this-change vs tip-of-tree}

render

this-change: 99.05ms - 104.21ms
this-change, tip-of-tree, previous-release: slower ❌ 0% - 10% (0.07ms - 3.71ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -3% - +3% (-2.45ms - +2.24ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -2% - +3% (-1.02ms - +1.89ms)
^{this-change vs tip-of-tree}

update

this-change: 1045.01ms - 1083.03ms
this-change, tip-of-tree, previous-release: unsure 🔍 -4% - +6% (-4.26ms - +5.36ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -3% - +5% (-4.69ms - +8.20ms)
^{this-change vs tip-of-tree}
this-change, tip-of-tree, previous-release: unsure 🔍 -1% - +1% (-8.46ms - +13.88ms)
^{this-change vs tip-of-tree}

update-reflect

this-change: 1001.93ms - 1031.69ms
this-change, tip-of-tree, previous-release: unsure 🔍 -1% - +1% (-5.62ms - +13.27ms)
^{this-change vs tip-of-tree}

Results

this-change

Browser: chrome-headless 115.0.5790.102
Sample size: 50
Built by: Benchmarks #5124
Commit: 1a6f172

render

Version	Avg time	vs
	99.05ms - 104.21ms	-

update

Version	Avg time	vs
	1045.01ms - 1083.03ms	-

update-reflect

Version	Avg time	vs
	1001.93ms - 1031.69ms	-

this-change, tip-of-tree, previous-release

Browser: chrome-headless 115.0.5790.102
Sample size: 30
Built by: Benchmarks #5124
Commit: 1a6f172

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	38.50ms - 41.27ms	-	slower ❌ 0% - 10% 0.07ms - 3.71ms	unsure 🔍 -3% - +6% -1.32ms - +2.52ms
tip-of-tree tip-of-tree	36.81ms - 39.18ms	faster ✔ 0% - 9% 0.07ms - 3.71ms	-	unsure 🔍 -8% - +1% -3.08ms - +0.49ms
previous-release previous-release	37.96ms - 40.62ms	unsure 🔍 -6% - +3% -2.52ms - +1.32ms	unsure 🔍 -1% - +8% -0.49ms - +3.08ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	93.14ms - 99.36ms	-	unsure 🔍 -4% - +6% -4.26ms - +5.36ms	unsure 🔍 -3% - +6% -2.70ms - +5.74ms
tip-of-tree tip-of-tree	92.03ms - 99.37ms	unsure 🔍 -6% - +4% -5.36ms - +4.26ms	-	unsure 🔍 -4% - +6% -3.67ms - +5.62ms
previous-release previous-release	91.87ms - 97.57ms	unsure 🔍 -6% - +3% -5.74ms - +2.70ms	unsure 🔍 -6% - +4% -5.62ms - +3.67ms	-

nop-update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	22.60ms - 24.07ms	-	unsure 🔍 -8% - +6% -1.96ms - +1.54ms	unsure 🔍 -6% - +6% -1.37ms - +1.29ms
tip-of-tree tip-of-tree	21.96ms - 25.13ms	unsure 🔍 -7% - +8% -1.54ms - +1.96ms	-	unsure 🔍 -8% - +9% -1.77ms - +2.10ms
previous-release previous-release	22.27ms - 24.48ms	unsure 🔍 -6% - +6% -1.29ms - +1.37ms	unsure 🔍 -9% - +7% -2.10ms - +1.77ms	-

this-change, tip-of-tree, previous-release

Browser: chrome-headless 115.0.5790.102
Sample size: 50
Built by: Benchmarks #5124
Commit: 1a6f172

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	69.08ms - 72.86ms	-	unsure 🔍 -3% - +3% -2.45ms - +2.24ms	unsure 🔍 -4% - +3% -2.53ms - +2.46ms
tip-of-tree tip-of-tree	69.68ms - 72.47ms	unsure 🔍 -3% - +3% -2.24ms - +2.45ms	-	unsure 🔍 -3% - +3% -2.08ms - +2.21ms
previous-release previous-release	69.38ms - 72.64ms	unsure 🔍 -3% - +4% -2.46ms - +2.53ms	unsure 🔍 -3% - +3% -2.21ms - +2.08ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	152.53ms - 161.91ms	-	unsure 🔍 -3% - +5% -4.69ms - +8.20ms	unsure 🔍 -4% - +4% -6.55ms - +6.12ms
tip-of-tree tip-of-tree	151.04ms - 159.88ms	unsure 🔍 -5% - +3% -8.20ms - +4.69ms	-	unsure 🔍 -5% - +3% -8.10ms - +4.17ms
previous-release previous-release	153.18ms - 161.69ms	unsure 🔍 -4% - +4% -6.12ms - +6.55ms	unsure 🔍 -3% - +5% -4.17ms - +8.10ms	-

this-change, tip-of-tree, previous-release

Browser: chrome-headless 115.0.5790.102
Sample size: 50
Built by: Benchmarks #5124
Commit: 1a6f172

render

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	63.72ms - 65.86ms	-	unsure 🔍 -2% - +3% -1.02ms - +1.89ms	unsure 🔍 -3% - +1% -1.89ms - +0.98ms
tip-of-tree tip-of-tree	63.37ms - 65.34ms	unsure 🔍 -3% - +2% -1.89ms - +1.02ms	-	unsure 🔍 -3% - +1% -2.27ms - +0.48ms
previous-release previous-release	64.29ms - 66.21ms	unsure 🔍 -2% - +3% -0.98ms - +1.89ms	unsure 🔍 -1% - +4% -0.48ms - +2.27ms	-

update

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	993.39ms - 1008.24ms	-	unsure 🔍 -1% - +1% -8.46ms - +13.88ms	unsure 🔍 -0% - +2% -2.63ms - +16.98ms
tip-of-tree tip-of-tree	989.76ms - 1006.46ms	unsure 🔍 -1% - +1% -13.88ms - +8.46ms	-	unsure 🔍 -1% - +2% -6.06ms - +14.99ms
previous-release previous-release	987.24ms - 1000.05ms	unsure 🔍 -2% - +0% -16.98ms - +2.63ms	unsure 🔍 -1% - +1% -14.99ms - +6.06ms	-

update-reflect

Version	Avg time	vs this-change	vs tip-of-tree tip-of-tree	vs previous-release previous-release
this-change	999.25ms - 1013.81ms	-	unsure 🔍 -1% - +1% -5.62ms - +13.27ms	unsure 🔍 -1% - +1% -5.81ms - +13.56ms
tip-of-tree tip-of-tree	996.69ms - 1008.72ms	unsure 🔍 -1% - +1% -13.27ms - +5.62ms	-	unsure 🔍 -1% - +1% -8.72ms - +8.82ms
previous-release previous-release	996.28ms - 1009.04ms	unsure 🔍 -1% - +1% -13.56ms - +5.81ms	unsure 🔍 -1% - +1% -8.82ms - +8.72ms	-

_{tachometer-reporter-action v2 for Benchmarks}

augustjk

Nice, thanks for adding this!

AndrewJakubowicz added 3 commits July 31, 2023 12:59

provide a better error message for relative imports in tests

6d7eed5

Reword error

7ef947a

add empty changeset

13c04a2

AndrewJakubowicz requested a review from kevinpschaaf as a code owner July 31, 2023 20:18

AndrewJakubowicz requested review from augustjk and justinfagnani July 31, 2023 20:19

add red cross to make the comment stand out

1a6f172

augustjk approved these changes Jul 31, 2023

View reviewed changes

AndrewJakubowicz merged commit 47b4891 into main Jul 31, 2023

AndrewJakubowicz deleted the improve-error-message-web-test-runner branch July 31, 2023 21:24

[infra] improve error message for test:common #4059

[infra] improve error message for test:common #4059

Uh oh!

Conversation

AndrewJakubowicz commented Jul 31, 2023

Context

Features

Testing

Uh oh!

changeset-bot bot commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Tachometer Benchmark Results

Summary

nop-update

render

update

update-reflect

Results

render

update

update-reflect

render

update

nop-update

render

update

render

update

update-reflect

Uh oh!

augustjk left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

changeset-bot bot commented Jul 31, 2023 •

edited

Loading

github-actions bot commented Jul 31, 2023 •

edited

Loading