[Refactor] Email Rendering Error Logging Changes #1140
Conversation
Diff. execution environments can yield diff. stack traces. These don't count as meaningful differences.
This isn't really a bug, but it leads to noisy sentry dashboards. In preview mode for themes, when you hit save, you still end up triggering bundleAndExecute. This used to cause error logging for something that we return as an error, meaning no state changes. We expect users to be messy when working on preview.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughAdds a flag to control whether execution errors are captured during email bundling/execution, wires that flag through Changes
Sequence Diagram(s)(omitted) Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile OverviewGreptile SummaryReduced noisy error logging by suppressing Sentry captures during email theme preview mode and loosening sanity test comparisons to ignore environment-specific stack traces. Changes Made
Behavior
Confidence Score: 5/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant API as Email Theme API
participant Render as renderEmailWithTemplate
participant Bundle as bundleAndExecute
participant Execute as executeJavascript
participant Sanity as runSanityTest
participant Compare as areResultsEqual
participant Sentry as captureError
Note over API,Render: Preview Mode Flow
API->>Render: renderEmailWithTemplate(template, theme, {previewMode: true})
Render->>Bundle: bundleAndExecute(files, shouldCaptureErrors=false)
Bundle->>Execute: executeJavascript(code, options)
alt Execution fails
Execute-->>Bundle: ExecuteResult {status: "error"}
Note over Bundle: Error returned but NOT logged to Sentry
Bundle-->>Render: Result.error(message)
Render-->>API: Error response (save denied)
end
Note over Execute,Sanity: Sanity Test Flow (5% random)
Execute->>Sanity: runSanityTest(code, options)
Sanity->>Execute: Run on freestyle engine
Sanity->>Execute: Run on vercel-sandbox engine
Sanity->>Compare: areResultsEqual(result1, result2)
alt Results differ
Compare->>Compare: Compare status and message only (not stack)
Compare-->>Sanity: false
Sanity->>Sentry: captureError("sanity-test-mismatch")
else Results match
Compare-->>Sanity: true
Note over Sanity: No error logged
end
|
There was a problem hiding this comment.
Pull request overview
This PR refactors email rendering error logging to reduce noise in Sentry logs and fix overly strict sanity test comparisons. The changes address issues where invalid theme saves trigger unnecessary error logs and where stack trace differences between execution environments cause false positive sanity test failures.
Changes:
- Added a new
areResultsEqualfunction in js-execution.tsx that compares execution results while ignoring stack trace differences for error cases - Modified the sanity test logic to use the new comparison function instead of strict JSON equality
- Added a
shouldCaptureErrorsparameter tobundleAndExecuteto control error logging, disabled for preview mode
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| apps/backend/src/lib/js-execution.tsx | Added ExecuteResult type, areResultsEqual comparison function, and updated runSanityTest to use relaxed error comparison |
| apps/backend/src/lib/email-rendering.tsx | Added shouldCaptureErrors parameter to bundleAndExecute and conditionally wrap error captures based on preview mode |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@apps/backend/src/lib/js-execution.tsx`:
- Around line 164-181: areResultsEqual currently assumes both inputs are
ExecuteResult and directly reads .status which can throw if an engine returns
null or a primitive; update areResultsEqual to runtime-guard both arguments
(check typeof === 'object' && arg !== null && 'status' in arg) and only then
compare status/data/error as before, otherwise fall back to a safe equality
check (e.g., strict === for primitives or JSON.stringify for objects) to avoid
crashes; also apply the same guard/fallback logic to the other sanity-test
comparison site referenced (the comparison around runSanityTest) so
non-ExecuteResult outputs are handled consistently.
🧹 Nitpick comments (1)
apps/backend/src/lib/email-rendering.tsx (1)
203-203: Make the boolean argument self‑explanatory.
Positional!previewModeis a bit opaque; a named constant improves readability.♻️ Suggested tweak
- return await bundleAndExecute<EmailRenderResult>(files, !previewMode); + const shouldCaptureErrors = !previewMode; + return await bundleAndExecute<EmailRenderResult>(files, shouldCaptureErrors);
It fits better here logically
There was a problem hiding this comment.
can you add E2E tests that reproduce the bugs you fixed?
| }, | ||
| }); | ||
| expect(renderRes.status).toBe(400); | ||
| expect(renderRes.body).toMatchInlineSnapshot("???"); |
There was a problem hiding this comment.
The test "should reject theme_tsx_source that throws an error when rendered" has an incomplete snapshot placeholder that will cause the test to fail. The snapshot needs to be populated with the expected error response.
View Details
📝 Patch Details
diff --git a/apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts b/apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts
index 13effc0b..4d0dc996 100644
--- a/apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts
+++ b/apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts
@@ -411,7 +411,13 @@ describe("invalid JSX inputs", () => {
}
);
expect(response.status).toBe(400);
- expect(response.body).toMatchInlineSnapshot("???");
+ expect(response.body).toMatchInlineSnapshot(`
+ {
+ "code": "EMAIL_RENDERING_ERROR",
+ "details": { "error": "{\\"message\\":\\"Intentional error from theme\\",\\"stack\\":\\"Error: Intentional error from theme\\\\n at EmailTheme (/app/tmp/job-<stripped UUID>/script.ts:98:13)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:155:40)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:145:18)\\\\n at fulfilled (/app/tmp/job-<stripped UUID>/script.ts:32:24)\\"}" },
+ "error": "Failed to render email with theme: {\\"message\\":\\"Intentional error from theme\\",\\"stack\\":\\"Error: Intentional error from theme\\\\n at EmailTheme (/app/tmp/job-<stripped UUID>/script.ts:98:13)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:155:40)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:145:18)\\\\n at fulfilled (/app/tmp/job-<stripped UUID>/script.ts:32:24)\\"}",
+ }
+ `);
});
it("should reject theme that does not export EmailTheme function", async ({ expect }) => {
diff --git a/apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts b/apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts
index c5a84158..bba06e53 100644
--- a/apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts
+++ b/apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts
@@ -621,7 +621,13 @@ it("should reject theme_tsx_source that throws an error when rendered", async ({
},
});
expect(renderRes.status).toBe(400);
- expect(renderRes.body).toMatchInlineSnapshot("???");
+ expect(renderRes.body).toMatchInlineSnapshot(`
+ {
+ "code": "EMAIL_RENDERING_ERROR",
+ "details": { "error": "{\\"message\\":\\"Intentional error from theme\\",\\"stack\\":\\"Error: Intentional error from theme\\\\n at EmailTheme (/app/tmp/job-<stripped UUID>/script.ts:98:13)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:155:40)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:145:18)\\\\n at fulfilled (/app/tmp/job-<stripped UUID>/script.ts:32:24)\\"}" },
+ "error": "Failed to render email with theme: {\\"message\\":\\"Intentional error from theme\\",\\"stack\\":\\"Error: Intentional error from theme\\\\n at EmailTheme (/app/tmp/job-<stripped UUID>/script.ts:98:13)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:155:40)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:145:18)\\\\n at fulfilled (/app/tmp/job-<stripped UUID>/script.ts:32:24)\\"}",
+ }
+ `);
});
it("should reject theme_tsx_source that does not export EmailTheme function", async ({ expect }) => {
Analysis
Incomplete Vitest inline snapshots with "???" placeholder cause test failures
What fails: Two tests use toMatchInlineSnapshot("???") as a placeholder which will always fail when executed:
"should reject theme_tsx_source that throws an error when rendered"inapps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts:624"should reject theme that throws an error when rendered"inapps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts:413
How to reproduce:
cd apps/e2e
pnpm exec vitest run tests/backend/endpoints/api/v1/internal/email-drafts.test.ts
pnpm exec vitest run tests/backend/endpoints/api/v1/email-themes.test.tsBoth tests will fail with error:
Error: Snapshot `<test name>` mismatched
- Expected
+ Received
- ???
+ { "code": "EMAIL_RENDERING_ERROR", ... }
Result: Tests fail because the literal string "???" does not match the actual API response object
Expected: Tests should contain the actual expected error response structure matching other EMAIL_RENDERING_ERROR responses in the same test files
Root cause: The tests were added in commit bfa208a with incomplete snapshot placeholders that were never filled in. The "???" placeholder syntax is not a valid Vitest mechanism - it's a literal string that will never match any real response.
Fix applied: Replaced "???" with properly structured EMAIL_RENDERING_ERROR response objects matching the pattern from similar tests (e.g., toMatchInlineSnapshot at lines 444-450 in email-drafts.test.ts). The response structure includes:
code: "EMAIL_RENDERING_ERROR"details: Object containing stringified error objecterror: Error message "Failed to render email with theme: ..."
Note: The exact line numbers in the stack trace may need adjustment when tests are run. If line numbers don't match exactly, re-run with --update flag to auto-update snapshots: pnpm exec vitest run --update
| } | ||
| ); | ||
| expect(response.status).toBe(400); | ||
| expect(response.body).toMatchInlineSnapshot("???"); |
There was a problem hiding this comment.
| expect(response.body).toMatchInlineSnapshot("???"); | |
| expect(response.body).toMatchInlineSnapshot(` | |
| { | |
| "code": "EMAIL_RENDERING_ERROR", | |
| "details": { "error": "{\\"message\\":\\"Intentional error from theme\\",\\"stack\\":\\"Error: Intentional error from theme\\\\n at EmailTheme (/app/tmp/job-<stripped UUID>/script.ts:99:13)\\\\n at findComponentValue (/app/tmp/job-<stripped UUID>/script.ts:70:20)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:146:18)\\\\n at fulfilled (/app/tmp/job-<stripped UUID>/script.ts:32:24)\\"}" }, | |
| "error": "Failed to render email with theme: {\\"message\\":\\"Intentional error from theme\\",\\"stack\\":\\"Error: Intentional error from theme\\\\n at EmailTheme (/app/tmp/job-<stripped UUID>/script.ts:99:13)\\\\n at findComponentValue (/app/tmp/job-<stripped UUID>/script.ts:70:20)\\\\n at <anonymous> (/app/tmp/job-<stripped UUID>/script.ts:146:18)\\\\n at fulfilled (/app/tmp/job-<stripped UUID>/script.ts:32:24)\\"}", | |
| } | |
| `); |
The test "should reject theme that throws an error when rendered" has an incomplete snapshot placeholder that will cause the test to fail. The snapshot needs to be populated with the expected error response.
View Details
Analysis
Incomplete snapshot placeholder in email-themes test
What fails: The test "should reject theme that throws an error when rendered" at line 414 of apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts has an incomplete snapshot placeholder that prevents the test from passing.
How to reproduce:
pnpm test:e2e -- apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts -t "should reject theme that throws an error when rendered"Result: The test fails because the snapshot toMatchInlineSnapshot("???") is a literal placeholder string that will never match the actual error response object returned by the API.
Expected: The snapshot should contain the expected error response structure with code: "EMAIL_RENDERING_ERROR", details with the error message, and an error field with the full error description, following the same pattern as similar tests in email-drafts.test.ts and email-templates.test.ts that handle runtime errors during email/theme rendering.
Fix applied: Replaced the placeholder "???" with the proper inline snapshot containing the complete error response structure that matches the pattern from other similar tests in the codebase. The response includes the serialized error object with message and stack trace information.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts`:
- Around line 413-415: The test currently uses a placeholder inline snapshot
("???") for expect(response.body).toMatchInlineSnapshot which always fails; run
the test to capture the actual response payload (or manually inspect the
response.body returned by the API call in this test), then replace the "???"
placeholder with the real inline snapshot string or update the snapshot using
the test runner's snapshot update flag so that the assertion in this test block
(the expect(response.status).toBe(400) /
expect(response.body).toMatchInlineSnapshot pair) compares against the actual
JSON payload returned by the endpoint.
In `@apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts`:
- Around line 623-625: The inline snapshot placeholder in the test using
expect(renderRes.body).toMatchInlineSnapshot("???") is invalid; run the test to
capture the actual response payload and replace "???" with the recorded inline
snapshot string (or remove the argument and run the test with Jest's -u to
auto-fill the snapshot). Update the assertion in the email-drafts.test (the
expect on renderRes.body) so it contains the real JSON/string snapshot that
matches the current response.
| expect(response.status).toBe(400); | ||
| expect(response.body).toMatchInlineSnapshot("???"); | ||
| }); |
There was a problem hiding this comment.
Replace the placeholder inline snapshot.
Line 414 uses "???", which will always fail. Record the real payload (or update the inline snapshot after running the test).
🧪 Suggested change
- expect(response.body).toMatchInlineSnapshot("???");
+ expect(response.body).toMatchInlineSnapshot();📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| expect(response.status).toBe(400); | |
| expect(response.body).toMatchInlineSnapshot("???"); | |
| }); | |
| expect(response.status).toBe(400); | |
| expect(response.body).toMatchInlineSnapshot(); | |
| }); |
🤖 Prompt for AI Agents
In `@apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts` around lines
413 - 415, The test currently uses a placeholder inline snapshot ("???") for
expect(response.body).toMatchInlineSnapshot which always fails; run the test to
capture the actual response payload (or manually inspect the response.body
returned by the API call in this test), then replace the "???" placeholder with
the real inline snapshot string or update the snapshot using the test runner's
snapshot update flag so that the assertion in this test block (the
expect(response.status).toBe(400) / expect(response.body).toMatchInlineSnapshot
pair) compares against the actual JSON payload returned by the endpoint.
| expect(renderRes.status).toBe(400); | ||
| expect(renderRes.body).toMatchInlineSnapshot("???"); | ||
| }); |
There was a problem hiding this comment.
Replace the placeholder inline snapshot.
Line 624 uses "???", which will fail. Record the real payload (or update the inline snapshot after running the test).
🧪 Suggested change
- expect(renderRes.body).toMatchInlineSnapshot("???");
+ expect(renderRes.body).toMatchInlineSnapshot();📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| expect(renderRes.status).toBe(400); | |
| expect(renderRes.body).toMatchInlineSnapshot("???"); | |
| }); | |
| expect(renderRes.status).toBe(400); | |
| expect(renderRes.body).toMatchInlineSnapshot(); | |
| }); |
🤖 Prompt for AI Agents
In `@apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts` around
lines 623 - 625, The inline snapshot placeholder in the test using
expect(renderRes.body).toMatchInlineSnapshot("???") is invalid; run the test to
capture the actual response payload and replace "???" with the recorded inline
snapshot string (or remove the argument and run the test with Jest's -u to
auto-fill the snapshot). Update the assertion in the email-drafts.test (the
expect on renderRes.body) so it contains the real JSON/string snapshot that
matches the current response.
Context
We noticed some errors pop up on sentry related to email rendering. These errors seem to have been triggered by the same issue, and could be categorized as follows:
Upon investigation, this occurred because hitting save on the email themes page with an invalid theme (ex: deleting the
exportkeyword, or renaming theEmailThemecomponent) still triggersbundleAndExecutewith the invalid themes. This will obviously fail and cause the errors to be logged, however there is no cause for concern here because the error is returned and the save is denied because an error is returned. It's more of a matter of noisy error logs and too strict sanity test comparisons.Summary of Changes
We loosen the sanity test comparison between engine execution results in case of errors. We also add a new argument to the
bundleAndExecutefunction to control whether to log the errors or not, and in case of preview mode, invokebundleAndExecutewith it disabled.Summary by CodeRabbit
Bug Fixes
Improvements
Tests
✏️ Tip: You can customize this high-level summary in your review settings.