-
Notifications
You must be signed in to change notification settings - Fork 498
[Refactor] [Fix] Email Rendering Pipeline Refactor, Error Handling, and Bug Fixes #1140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Conversation
Diff. execution environments can yield diff. stack traces. These don't count as meaningful differences.
This isn't really a bug, but it leads to noisy sentry dashboards. In preview mode for themes, when you hit save, you still end up triggering bundleAndExecute. This used to cause error logging for something that we return as an error, meaning no state changes. We expect users to be messy when working on preview.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughReplaces multi-runtime execution/fallback with a single ExecuteResult-driven execution path, adds the exported ExecuteResult type and equality logic, updates js-execution APIs, simplifies email bundling/execution error handling, replaces Bun freestyle mock with Node-based runtime, and adds extensive E2E tests and updated email HTML snapshots. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant EmailRenderer as EmailRenderer (API)
participant JsExec as JsExecution
participant Engine as JsEngine
Client->>EmailRenderer: POST /render-email (code, theme, options)
EmailRenderer->>JsExec: bundleAndExecute(code, options)
JsExec->>Engine: execute(bundledCode, options)
alt Engine success
Engine-->>JsExec: ExecuteResult {status: "ok", data}
else Engine error
Engine-->>JsExec: ExecuteResult {status: "error", error}
end
JsExec-->>EmailRenderer: ExecuteResult
alt ExecuteResult.status == "ok"
EmailRenderer-->>Client: 200 + rendered HTML
else
EmailRenderer-->>Client: 400 + EMAIL_RENDERING_ERROR payload
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile OverviewGreptile SummaryReduced noisy error logging by suppressing Sentry captures during email theme preview mode and loosening sanity test comparisons to ignore environment-specific stack traces. Changes Made
Behavior
Confidence Score: 5/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant API as Email Theme API
participant Render as renderEmailWithTemplate
participant Bundle as bundleAndExecute
participant Execute as executeJavascript
participant Sanity as runSanityTest
participant Compare as areResultsEqual
participant Sentry as captureError
Note over API,Render: Preview Mode Flow
API->>Render: renderEmailWithTemplate(template, theme, {previewMode: true})
Render->>Bundle: bundleAndExecute(files, shouldCaptureErrors=false)
Bundle->>Execute: executeJavascript(code, options)
alt Execution fails
Execute-->>Bundle: ExecuteResult {status: "error"}
Note over Bundle: Error returned but NOT logged to Sentry
Bundle-->>Render: Result.error(message)
Render-->>API: Error response (save denied)
end
Note over Execute,Sanity: Sanity Test Flow (5% random)
Execute->>Sanity: runSanityTest(code, options)
Sanity->>Execute: Run on freestyle engine
Sanity->>Execute: Run on vercel-sandbox engine
Sanity->>Compare: areResultsEqual(result1, result2)
alt Results differ
Compare->>Compare: Compare status and message only (not stack)
Compare-->>Sanity: false
Sanity->>Sentry: captureError("sanity-test-mismatch")
else Results match
Compare-->>Sanity: true
Note over Sanity: No error logged
end
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2 files reviewed, no comments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR refactors email rendering error logging to reduce noise in Sentry logs and fix overly strict sanity test comparisons. The changes address issues where invalid theme saves trigger unnecessary error logs and where stack trace differences between execution environments cause false positive sanity test failures.
Changes:
- Added a new
areResultsEqualfunction in js-execution.tsx that compares execution results while ignoring stack trace differences for error cases - Modified the sanity test logic to use the new comparison function instead of strict JSON equality
- Added a
shouldCaptureErrorsparameter tobundleAndExecuteto control error logging, disabled for preview mode
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| apps/backend/src/lib/js-execution.tsx | Added ExecuteResult type, areResultsEqual comparison function, and updated runSanityTest to use relaxed error comparison |
| apps/backend/src/lib/email-rendering.tsx | Added shouldCaptureErrors parameter to bundleAndExecute and conditionally wrap error captures based on preview mode |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@apps/backend/src/lib/js-execution.tsx`:
- Around line 164-181: areResultsEqual currently assumes both inputs are
ExecuteResult and directly reads .status which can throw if an engine returns
null or a primitive; update areResultsEqual to runtime-guard both arguments
(check typeof === 'object' && arg !== null && 'status' in arg) and only then
compare status/data/error as before, otherwise fall back to a safe equality
check (e.g., strict === for primitives or JSON.stringify for objects) to avoid
crashes; also apply the same guard/fallback logic to the other sanity-test
comparison site referenced (the comparison around runSanityTest) so
non-ExecuteResult outputs are handled consistently.
🧹 Nitpick comments (1)
apps/backend/src/lib/email-rendering.tsx (1)
203-203: Make the boolean argument self‑explanatory.
Positional!previewModeis a bit opaque; a named constant improves readability.♻️ Suggested tweak
- return await bundleAndExecute<EmailRenderResult>(files, !previewMode); + const shouldCaptureErrors = !previewMode; + return await bundleAndExecute<EmailRenderResult>(files, shouldCaptureErrors);
It fits better here logically
apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts`:
- Around line 413-415: The test currently uses a placeholder inline snapshot
("???") for expect(response.body).toMatchInlineSnapshot which always fails; run
the test to capture the actual response payload (or manually inspect the
response.body returned by the API call in this test), then replace the "???"
placeholder with the real inline snapshot string or update the snapshot using
the test runner's snapshot update flag so that the assertion in this test block
(the expect(response.status).toBe(400) /
expect(response.body).toMatchInlineSnapshot pair) compares against the actual
JSON payload returned by the endpoint.
In `@apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts`:
- Around line 623-625: The inline snapshot placeholder in the test using
expect(renderRes.body).toMatchInlineSnapshot("???") is invalid; run the test to
capture the actual response payload and replace "???" with the recorded inline
snapshot string (or remove the argument and run the test with Jest's -u to
auto-fill the snapshot). Update the assertion in the email-drafts.test (the
expect on renderRes.body) so it contains the real JSON/string snapshot that
matches the current response.
Logging errors hsould generally not be gated behind conditionals.
throwing an error inside the emailthemes component was not caught as an error. This is because react-email/components didnt handle it correctly and fell back to client side rendering. This was outside of our try catch code, it was in the actual rendering. So it came back as status-ok. We bump versions to the latest version which does throw it. We also have to update the freestyle mock. Bun cannot handle these kinds of errors. they are treated as uncaught exceptions by it. So we switch to node. We also refactor some tests because the react-email components now return slightly different messages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@docker/dependencies/freestyle-mock/Dockerfile`:
- Around line 95-140: The code mutates global console and process.env (console
override via logMethods and previousEnv) while handling requests in server,
causing race conditions; to fix, stop mutating globals and run each request in
an isolated Worker thread instead: spawn a worker (worker_threads.Worker)
passing scriptFile, envVars and workDir as workerData, have the worker set its
own process.env and local console override, import the user module and
postMessage back { result, logs, error }, and in the main server thread listen
for worker messages/errors to respond and perform rm(workDir) cleanup; replace
the console/logMethods/previousEnv manipulation and the direct dynamic import of
scriptFile with this per-request worker approach (referencing server,
logMethods, previousEnv, scriptFile, workDir, and the import/result handling).
- Around line 40-93: The handler in createServer currently calls
JSON.parse(body) and awaits the npm install Promise without error handling which
can crash the process and leak workDir; wrap JSON.parse(body) in a try-catch and
respond with a 400 on parse errors, and wrap the install flow (the Promise that
waits on installProcess close and the spawn/await block around npm install) in
try-catch to respond with 500 on install failures, ensure you cleanup the
created workDir on any failure (delete workDir) and propagate meaningful error
messages via res.writeHead/res.end; also add minimal try-catch around mkdir and
writeFile operations (for workDir, scriptFile, packageJsonFile) to return 500
and cleanup on filesystem errors. Ensure you reference the same variables:
body/JSON.parse, workDir, baseWorkDir, scriptFile, packageJsonFile,
requestedNodeModules, preinstalledNodeModules, installProcess.
🧹 Nitpick comments (6)
apps/e2e/tests/backend/endpoints/api/v1/internal/email-templates.test.ts (1)
92-127: Consider using.toMatchInlineSnapshot()for the full response for consistency.Per coding guidelines,
.toMatchInlineSnapshot()is preferred. The first test (lines 33-45) uses a full response snapshot, but this test separates assertions. If the error message format is stable enough, consider consolidating:expect(updateResponse).toMatchInlineSnapshot(` NiceResponse { "status": 400, "body": { "code": "EMAIL_RENDERING_ERROR", "error": "...", // with actual error }, ... } `);However, if the exact error message varies by environment (similar to the comment at line 163), the current approach with
.toContain()is reasonable. Based on learnings: "Prefer .toMatchInlineSnapshot over other selectors when writing tests."apps/backend/src/lib/js-execution.tsx (2)
142-142: Use strict equality comparison.Line 142 uses loose equality (
!=) instead of strict equality (!==).🔧 Suggested fix
- if (getEnvVariable("STACK_VERCEL_SANDBOX_TOKEN","") != "") { + if (getEnvVariable("STACK_VERCEL_SANDBOX_TOKEN","") !== "") {
212-214: Consider using defensive coding pattern for Map access.The non-null assertions on
engineMap.get()work because the keys are hardcoded and the map is initialized at module scope. However, per coding guidelines, prefer?? throwErr(...)for defensive coding.🛡️ Optional: Defensive access pattern
async function runWithFallback(code: string, options: ExecuteJavascriptOptions): Promise<ExecuteResult> { - const freestyleEngine = engineMap.get("freestyle")!; - const vercelSandboxEngine = engineMap.get("vercel-sandbox")!; + const freestyleEngine = engineMap.get("freestyle") ?? throwErr("freestyle engine not found in engineMap"); + const vercelSandboxEngine = engineMap.get("vercel-sandbox") ?? throwErr("vercel-sandbox engine not found in engineMap");Apply similar change at line 252 for
runWithoutFallback.Based on learnings: "Code defensively. Prefer
?? throwErr(...)over non-null assertions with good error messages"apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts (1)
406-448: UnuseddraftIdvariable in test.The test creates a draft and stores its ID in
draftId(line 427), but then the render call at line 430 usestemplate_tsx_sourcedirectly instead of referencing the draft. Either:
- The draft creation is unnecessary and can be removed, or
- The render call should use the draft ID via a different endpoint/parameter.
This doesn't affect test correctness since the inline source matches, but the draft creation appears to serve no purpose.
♻️ Option 1: Remove unused draft creation
it("should reject draft that throws an error when rendered", async ({ expect }) => { await Project.createAndSwitch({ display_name: "Email Drafts Invalid JSX Project", config: { email_config: customEmailConfig }, }); - const createRes = await niceBackendFetch("/api/v1/internal/email-drafts", { - method: "POST", - accessType: "admin", - body: { - display_name: "Throwing Draft", - theme_id: false, - tsx_source: ` - import { Subject, NotificationCategory } from "@stackframe/emails"; - export function EmailTemplate() { - throw new Error('Intentional error from draft'); - } - `, - }, - }); - expect(createRes.status).toBe(200); - const draftId = createRes.body.id as string; - // Attempt to render the draft const renderRes = await niceBackendFetch("/api/v1/emails/render-email", {docker/dependencies/freestyle-mock/Dockerfile (2)
1-1: Verify Node base image support (optionally pin digest).
Line 1 switches tonode:22-slim; please confirm it aligns with your supported runtime and security patch cadence. Optional: pin the digest for reproducible builds.
7-36: Avoid dependency version drift between/app/package.jsonandpreinstalledNodeModules.
Lines 7-36 duplicate versions in two places; if they diverge,needsInstallcan be wrong. Consider deriving the map from/app/package.jsonat startup.♻️ Suggested refactor to derive preinstalled deps from /app/package.json
-import { mkdir, writeFile, rm } from "fs/promises"; +import { mkdir, writeFile, rm, readFile } from "fs/promises"; @@ -const preinstalledNodeModules = new Map([ - ["arktype", "2.1.20"], - ["react-dom", "19.1.1"], - ["react", "19.1.1"], - ["@react-email/components", "1.0.6"], -]); +const preinstalledNodeModules = new Map( + Object.entries( + JSON.parse(await readFile("/app/package.json", "utf8")).dependencies ?? {} + ) +);
Since react-email has been bumped, the rendering is slightly different.
Context
We noticed some errors pop up on sentry related to email rendering. These errors seem to have been triggered by the same issue, and could be categorized as follows:
Upon investigation, this occurred because hitting save on the email themes page with an invalid theme (ex: deleting the
exportkeyword, or renaming theEmailThemecomponent) still triggersbundleAndExecutewith the invalid themes. This will obviously fail and cause the errors to be logged, however there is no cause for concern here because the error is returned and the save is denied because an error is returned. It's more of a matter of noisy error logs and too strict sanity test comparisons.Beyond that,
js-executionis a little opaque and hard to understand, and this can mask errors in logic.We also noticed a new issue: manually throwing an error in the email theme code editor, and then trying to save was actually successful. This was because the version of
react-email/componentswe were using had faulty error handling, and fell back to client side rendering, masking the error. This wasn't caught by ourtry-catchsafeguards because it was a render time issue that was masked. More specifically, this was whatreact-emailwas doing:Switched to client rendering because the server rendering errored.Summary of Changes
We loosen the sanity test comparison between engine execution results in case of errors. We then refactor the
js-executionandemail-renderingfiles to read better, and to onlycaptureErrorwhen a service is down, but not for runtime errors in the user submitted code.To deal with the other bug, we bumped
react-email/componentsto the latest version. However, doing so exposed a gap between realfreestyleand ourfreestyle-mock: with the mock, the errors that were now raised were treated as uncaught exceptions, crashing the mock server. Consequently, we switched to usingnodeoverbun.We also expanded test coverage to account for different error paths.
Summary by CodeRabbit
New Features
Bug Fixes
Tests
✏️ Tip: You can customize this high-level summary in your review settings.