Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@nams1570
Copy link
Collaborator

@nams1570 nams1570 commented Jan 28, 2026

Context

We noticed some errors pop up on sentry related to email rendering. These errors seem to have been triggered by the same issue, and could be categorized as follows:

  1. Sanity test mismatch, even when the errors from freestyle and vercel sandbox were broadly similar. This occurred due to stack traces differing in different execution environments.
  2. Rendering errors from freestyle and vercel sandbox caused by the theme not being imported/ empty theme component.

Upon investigation, this occurred because hitting save on the email themes page with an invalid theme (ex: deleting the export keyword, or renaming the EmailTheme component) still triggers bundleAndExecute with the invalid themes. This will obviously fail and cause the errors to be logged, however there is no cause for concern here because the error is returned and the save is denied because an error is returned. It's more of a matter of noisy error logs and too strict sanity test comparisons.

Beyond that, js-execution is a little opaque and hard to understand, and this can mask errors in logic.

We also noticed a new issue: manually throwing an error in the email theme code editor, and then trying to save was actually successful. This was because the version of react-email/components we were using had faulty error handling, and fell back to client side rendering, masking the error. This wasn't caught by our try-catch safeguards because it was a render time issue that was masked. More specifically, this was what react-email was doing: Switched to client rendering because the server rendering errored.

Summary of Changes

We loosen the sanity test comparison between engine execution results in case of errors. We then refactor the js-execution and email-rendering files to read better, and to only captureError when a service is down, but not for runtime errors in the user submitted code.

To deal with the other bug, we bumped react-email/components to the latest version. However, doing so exposed a gap between real freestyle and our freestyle-mock: with the mock, the errors that were now raised were treated as uncaught exceptions, crashing the mock server. Consequently, we switched to using node over bun.

We also expanded test coverage to account for different error paths.

Summary by CodeRabbit

  • New Features

    • Structured error reporting for email rendering failures with detailed diagnostic messages.
  • Bug Fixes

    • Improved email rendering reliability and error handling robustness.
    • Enhanced fallback mechanisms for rendering failures.
  • Tests

    • Expanded test coverage for email rendering error scenarios and invalid inputs.

✏️ Tip: You can customize this high-level summary in your review settings.

Diff. execution environments can yield diff.
 stack traces.
These don't count as meaningful differences.
This isn't really a bug, but it leads to noisy sentry dashboards.
In preview mode for themes, when you hit save, you still end up triggering bundleAndExecute.
This used to cause error logging for something that we return as an error, meaning no state changes.
We expect users to be messy when working on preview.
Copilot AI review requested due to automatic review settings January 28, 2026 19:57
@vercel
Copy link

vercel bot commented Jan 28, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
stack-backend Ready Ready Preview, Comment Jan 31, 2026 2:33am
stack-dashboard Ready Ready Preview, Comment Jan 31, 2026 2:33am
stack-demo Ready Ready Preview, Comment Jan 31, 2026 2:33am
stack-docs Ready Ready Preview, Comment Jan 31, 2026 2:33am

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jan 28, 2026

📝 Walkthrough

Walkthrough

Replaces multi-runtime execution/fallback with a single ExecuteResult-driven execution path, adds the exported ExecuteResult type and equality logic, updates js-execution APIs, simplifies email bundling/execution error handling, replaces Bun freestyle mock with Node-based runtime, and adds extensive E2E tests and updated email HTML snapshots.

Changes

Cohort / File(s) Summary
Core JS execution
apps/backend/src/lib/js-execution.tsx
Add exported ExecuteResult union type; change JsEngine.execute and executeJavascript to return ExecuteResult; add areResultsEqual; refactor sanity tests, engine iteration, and structured error/fallback messages.
Email rendering
apps/backend/src/lib/email-rendering.tsx
Remove multi-runtime branches and captureError usage; call single executeJavascript path and return JSON-stringified errors on execution failure; minor signature tweak to bundle/execute.
E2E tests — rendering error coverage & snapshots
apps/e2e/tests/backend/endpoints/api/v1/.../email-themes.test.ts, .../email-drafts.test.ts, .../email-templates.test.ts, .../render-email.test.ts, .../emails/email-queue.test.ts, .../integrations/credential-scanning/revoke.test.ts, .../unsubscribe-link.test.ts
Add many tests asserting 400 + EMAIL_RENDERING_ERROR for invalid TSX, missing exports, runtime throws, non-function exports, and placeholders for infinite-loop/memory cases; update rendered HTML snapshots to new table-based output.
Freestyle mock runtime container
docker/dependencies/freestyle-mock/Dockerfile, docker/dependencies/freestyle-mock/server.mjs
Replace Bun-based image/server with Node 22 slim and Node HTTP server; switch scripts to .mjs; use npm install; implement per-request workdir, dynamic import execution, env var scoping, log capture, and structured JSON responses; ensure cleanup.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant EmailRenderer as EmailRenderer (API)
  participant JsExec as JsExecution
  participant Engine as JsEngine

  Client->>EmailRenderer: POST /render-email (code, theme, options)
  EmailRenderer->>JsExec: bundleAndExecute(code, options)
  JsExec->>Engine: execute(bundledCode, options)
  alt Engine success
    Engine-->>JsExec: ExecuteResult {status: "ok", data}
  else Engine error
    Engine-->>JsExec: ExecuteResult {status: "error", error}
  end
  JsExec-->>EmailRenderer: ExecuteResult
  alt ExecuteResult.status == "ok"
    EmailRenderer-->>Client: 200 + rendered HTML
  else
    EmailRenderer-->>Client: 400 + EMAIL_RENDERING_ERROR payload
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • BilalG1
  • N2D4
  • Developing-Gamer

Poem

🐇 I hopped through code at break of light,

Bundled modules snug and bright.
Engines hum, results aligned,
Tests now catch what used to hide.
Hop, commit — and nibble on a byte!

🚥 Pre-merge checks | ✅ 1 | ❌ 2
❌ Failed checks (1 warning, 1 inconclusive)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title mentions 'Refactor', 'Fix', and 'Email Rendering Pipeline' but is overly broad and generic, not clearly specifying the core issue (noisy Sentry errors, sanity test mismatches, or error handling improvements) addressed in the PR. Revise the title to be more specific about the main change, such as 'Reduce noisy Sentry errors in email rendering pipeline' or 'Simplify email rendering error handling and sanity tests'.
✅ Passed checks (1 passed)
Check name Status Explanation
Description check ✅ Passed The PR description is comprehensive, clearly explaining the context, investigation findings, and all key changes made, including the sanity test loosening, refactoring, react-email bump, Bun-to-Node switch, and test expansions.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch sentry-email-rendering

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Jan 28, 2026

Greptile Overview

Greptile Summary

Reduced noisy error logging by suppressing Sentry captures during email theme preview mode and loosening sanity test comparisons to ignore environment-specific stack traces.

Changes Made

  • Email Rendering: Added shouldCaptureErrors parameter to bundleAndExecute() function (defaults to true). When previewMode is enabled, errors are not logged to Sentry but are still returned to deny the save operation.

  • JS Execution: Introduced areResultsEqual() helper that compares execution results by status and error message only, excluding stack traces which differ across execution environments (freestyle vs vercel-sandbox).

Behavior

  • Invalid theme saves (missing exports, renamed components) still fail correctly and return errors
  • Preview mode no longer generates noisy Sentry alerts for expected validation failures
  • Sanity tests compare error semantics rather than exact stack trace strings
  • Production email rendering continues to capture all errors normally

Confidence Score: 5/5

  • This PR is safe to merge with no risk
  • The changes are simple refactors that improve error logging behavior without changing core functionality. The logic correctly suppresses logs during preview while maintaining validation, and the sanity test improvement is a sensible relaxation of overly strict comparisons.
  • No files require special attention

Important Files Changed

Filename Overview
apps/backend/src/lib/email-rendering.tsx Added shouldCaptureErrors parameter to bundleAndExecute to suppress error logging in preview mode
apps/backend/src/lib/js-execution.tsx Introduced areResultsEqual function to compare execution results by message only, not stack traces

Sequence Diagram

sequenceDiagram
    participant API as Email Theme API
    participant Render as renderEmailWithTemplate
    participant Bundle as bundleAndExecute
    participant Execute as executeJavascript
    participant Sanity as runSanityTest
    participant Compare as areResultsEqual
    participant Sentry as captureError

    Note over API,Render: Preview Mode Flow
    API->>Render: renderEmailWithTemplate(template, theme, {previewMode: true})
    Render->>Bundle: bundleAndExecute(files, shouldCaptureErrors=false)
    Bundle->>Execute: executeJavascript(code, options)
    
    alt Execution fails
        Execute-->>Bundle: ExecuteResult {status: "error"}
        Note over Bundle: Error returned but NOT logged to Sentry
        Bundle-->>Render: Result.error(message)
        Render-->>API: Error response (save denied)
    end

    Note over Execute,Sanity: Sanity Test Flow (5% random)
    Execute->>Sanity: runSanityTest(code, options)
    Sanity->>Execute: Run on freestyle engine
    Sanity->>Execute: Run on vercel-sandbox engine
    Sanity->>Compare: areResultsEqual(result1, result2)
    
    alt Results differ
        Compare->>Compare: Compare status and message only (not stack)
        Compare-->>Sanity: false
        Sanity->>Sentry: captureError("sanity-test-mismatch")
    else Results match
        Compare-->>Sanity: true
        Note over Sanity: No error logged
    end
Loading

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors email rendering error logging to reduce noise in Sentry logs and fix overly strict sanity test comparisons. The changes address issues where invalid theme saves trigger unnecessary error logs and where stack trace differences between execution environments cause false positive sanity test failures.

Changes:

  • Added a new areResultsEqual function in js-execution.tsx that compares execution results while ignoring stack trace differences for error cases
  • Modified the sanity test logic to use the new comparison function instead of strict JSON equality
  • Added a shouldCaptureErrors parameter to bundleAndExecute to control error logging, disabled for preview mode

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
apps/backend/src/lib/js-execution.tsx Added ExecuteResult type, areResultsEqual comparison function, and updated runSanityTest to use relaxed error comparison
apps/backend/src/lib/email-rendering.tsx Added shouldCaptureErrors parameter to bundleAndExecute and conditionally wrap error captures based on preview mode

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@apps/backend/src/lib/js-execution.tsx`:
- Around line 164-181: areResultsEqual currently assumes both inputs are
ExecuteResult and directly reads .status which can throw if an engine returns
null or a primitive; update areResultsEqual to runtime-guard both arguments
(check typeof === 'object' && arg !== null && 'status' in arg) and only then
compare status/data/error as before, otherwise fall back to a safe equality
check (e.g., strict === for primitives or JSON.stringify for objects) to avoid
crashes; also apply the same guard/fallback logic to the other sanity-test
comparison site referenced (the comparison around runSanityTest) so
non-ExecuteResult outputs are handled consistently.
🧹 Nitpick comments (1)
apps/backend/src/lib/email-rendering.tsx (1)

203-203: Make the boolean argument self‑explanatory.
Positional !previewMode is a bit opaque; a named constant improves readability.

♻️ Suggested tweak
-  return await bundleAndExecute<EmailRenderResult>(files, !previewMode);
+  const shouldCaptureErrors = !previewMode;
+  return await bundleAndExecute<EmailRenderResult>(files, shouldCaptureErrors);

It fits better here logically
@nams1570 nams1570 requested a review from N2D4 January 28, 2026 22:23
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@apps/e2e/tests/backend/endpoints/api/v1/email-themes.test.ts`:
- Around line 413-415: The test currently uses a placeholder inline snapshot
("???") for expect(response.body).toMatchInlineSnapshot which always fails; run
the test to capture the actual response payload (or manually inspect the
response.body returned by the API call in this test), then replace the "???"
placeholder with the real inline snapshot string or update the snapshot using
the test runner's snapshot update flag so that the assertion in this test block
(the expect(response.status).toBe(400) /
expect(response.body).toMatchInlineSnapshot pair) compares against the actual
JSON payload returned by the endpoint.

In `@apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts`:
- Around line 623-625: The inline snapshot placeholder in the test using
expect(renderRes.body).toMatchInlineSnapshot("???") is invalid; run the test to
capture the actual response payload and replace "???" with the recorded inline
snapshot string (or remove the argument and run the test with Jest's -u to
auto-fill the snapshot). Update the assertion in the email-drafts.test (the
expect on renderRes.body) so it contains the real JSON/string snapshot that
matches the current response.

Logging errors hsould generally not be gated behind conditionals.
throwing an error inside the emailthemes component was not caught as an error.
This is because react-email/components
 didnt handle it correctly and fell back to client side rendering.
This was outside of our try catch code, it was in the actual rendering.
So it came back as status-ok.
We bump versions to the latest version which does throw it.
We also  have to update the freestyle mock.
Bun cannot handle these kinds of errors.
they are treated as uncaught exceptions by it.
So we switch to node.
We also refactor some tests because the react-email components now return slightly different messages.
Copy link

@vercel vercel bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:

Unhandled exceptions from executeJavascript when both engines fail cause unhandled promise rejections

Fix on Vercel

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@docker/dependencies/freestyle-mock/Dockerfile`:
- Around line 95-140: The code mutates global console and process.env (console
override via logMethods and previousEnv) while handling requests in server,
causing race conditions; to fix, stop mutating globals and run each request in
an isolated Worker thread instead: spawn a worker (worker_threads.Worker)
passing scriptFile, envVars and workDir as workerData, have the worker set its
own process.env and local console override, import the user module and
postMessage back { result, logs, error }, and in the main server thread listen
for worker messages/errors to respond and perform rm(workDir) cleanup; replace
the console/logMethods/previousEnv manipulation and the direct dynamic import of
scriptFile with this per-request worker approach (referencing server,
logMethods, previousEnv, scriptFile, workDir, and the import/result handling).
- Around line 40-93: The handler in createServer currently calls
JSON.parse(body) and awaits the npm install Promise without error handling which
can crash the process and leak workDir; wrap JSON.parse(body) in a try-catch and
respond with a 400 on parse errors, and wrap the install flow (the Promise that
waits on installProcess close and the spawn/await block around npm install) in
try-catch to respond with 500 on install failures, ensure you cleanup the
created workDir on any failure (delete workDir) and propagate meaningful error
messages via res.writeHead/res.end; also add minimal try-catch around mkdir and
writeFile operations (for workDir, scriptFile, packageJsonFile) to return 500
and cleanup on filesystem errors. Ensure you reference the same variables:
body/JSON.parse, workDir, baseWorkDir, scriptFile, packageJsonFile,
requestedNodeModules, preinstalledNodeModules, installProcess.
🧹 Nitpick comments (6)
apps/e2e/tests/backend/endpoints/api/v1/internal/email-templates.test.ts (1)

92-127: Consider using .toMatchInlineSnapshot() for the full response for consistency.

Per coding guidelines, .toMatchInlineSnapshot() is preferred. The first test (lines 33-45) uses a full response snapshot, but this test separates assertions. If the error message format is stable enough, consider consolidating:

expect(updateResponse).toMatchInlineSnapshot(`
  NiceResponse {
    "status": 400,
    "body": {
      "code": "EMAIL_RENDERING_ERROR",
      "error": "...",  // with actual error
    },
    ...
  }
`);

However, if the exact error message varies by environment (similar to the comment at line 163), the current approach with .toContain() is reasonable. Based on learnings: "Prefer .toMatchInlineSnapshot over other selectors when writing tests."

apps/backend/src/lib/js-execution.tsx (2)

142-142: Use strict equality comparison.

Line 142 uses loose equality (!=) instead of strict equality (!==).

🔧 Suggested fix
-    if (getEnvVariable("STACK_VERCEL_SANDBOX_TOKEN","") != "") {
+    if (getEnvVariable("STACK_VERCEL_SANDBOX_TOKEN","") !== "") {

212-214: Consider using defensive coding pattern for Map access.

The non-null assertions on engineMap.get() work because the keys are hardcoded and the map is initialized at module scope. However, per coding guidelines, prefer ?? throwErr(...) for defensive coding.

🛡️ Optional: Defensive access pattern
 async function runWithFallback(code: string, options: ExecuteJavascriptOptions): Promise<ExecuteResult> {
-  const freestyleEngine = engineMap.get("freestyle")!;
-  const vercelSandboxEngine = engineMap.get("vercel-sandbox")!;
+  const freestyleEngine = engineMap.get("freestyle") ?? throwErr("freestyle engine not found in engineMap");
+  const vercelSandboxEngine = engineMap.get("vercel-sandbox") ?? throwErr("vercel-sandbox engine not found in engineMap");

Apply similar change at line 252 for runWithoutFallback.

Based on learnings: "Code defensively. Prefer ?? throwErr(...) over non-null assertions with good error messages"

apps/e2e/tests/backend/endpoints/api/v1/internal/email-drafts.test.ts (1)

406-448: Unused draftId variable in test.

The test creates a draft and stores its ID in draftId (line 427), but then the render call at line 430 uses template_tsx_source directly instead of referencing the draft. Either:

  1. The draft creation is unnecessary and can be removed, or
  2. The render call should use the draft ID via a different endpoint/parameter.

This doesn't affect test correctness since the inline source matches, but the draft creation appears to serve no purpose.

♻️ Option 1: Remove unused draft creation
 it("should reject draft that throws an error when rendered", async ({ expect }) => {
   await Project.createAndSwitch({
     display_name: "Email Drafts Invalid JSX Project",
     config: { email_config: customEmailConfig },
   });

-  const createRes = await niceBackendFetch("/api/v1/internal/email-drafts", {
-    method: "POST",
-    accessType: "admin",
-    body: {
-      display_name: "Throwing Draft",
-      theme_id: false,
-      tsx_source: `
-        import { Subject, NotificationCategory } from "@stackframe/emails";
-        export function EmailTemplate() {
-          throw new Error('Intentional error from draft');
-        }
-      `,
-    },
-  });
-  expect(createRes.status).toBe(200);
-  const draftId = createRes.body.id as string;
-
   // Attempt to render the draft
   const renderRes = await niceBackendFetch("/api/v1/emails/render-email", {
docker/dependencies/freestyle-mock/Dockerfile (2)

1-1: Verify Node base image support (optionally pin digest).
Line 1 switches to node:22-slim; please confirm it aligns with your supported runtime and security patch cadence. Optional: pin the digest for reproducible builds.


7-36: Avoid dependency version drift between /app/package.json and preinstalledNodeModules.
Lines 7-36 duplicate versions in two places; if they diverge, needsInstall can be wrong. Consider deriving the map from /app/package.json at startup.

♻️ Suggested refactor to derive preinstalled deps from /app/package.json
-import { mkdir, writeFile, rm } from "fs/promises";
+import { mkdir, writeFile, rm, readFile } from "fs/promises";
@@
-const preinstalledNodeModules = new Map([
-  ["arktype", "2.1.20"],
-  ["react-dom", "19.1.1"],
-  ["react", "19.1.1"],
-  ["@react-email/components", "1.0.6"],
-]);
+const preinstalledNodeModules = new Map(
+  Object.entries(
+    JSON.parse(await readFile("/app/package.json", "utf8")).dependencies ?? {}
+  )
+);

Since react-email has been bumped, the rendering is slightly different.
@nams1570 nams1570 requested a review from N2D4 January 31, 2026 02:21
@nams1570 nams1570 changed the title [Refactor] Email Rendering Error Logging Changes [Refactor] [Fix] Email Rendering Pipeline Refactor, Error Handling, and Bug Fixes Jan 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants