-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Async partner export #3013
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Async partner export #3013
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Caution Review failedThe pull request is closed. WalkthroughAdds batched partner streaming, a QStash-triggered cron export route, a consolidated partner-count helper, export endpoint backgrounding for large exports, CSV formatting utilities, UI handling for async exports, and an email template for completed exports. Changes
Sequence Diagram(s)sequenceDiagram
actor User
participant Frontend as Export Modal
participant API as /api/partners/export (GET)
participant Counter as getPartnersCount
participant QStash
participant Cron as /api/cron/partners/export (POST)
participant BatchGen as fetchPartnersBatch
participant Formatter as formatPartnersForExport
participant R2 as Storage (R2)
participant Email
participant Inbox as User Email
User->>Frontend: Request export
Frontend->>API: GET export with filters
API->>Counter: count matching partners
Counter-->>API: totalCount
alt totalCount β€ MAX_PARTNERS_TO_EXPORT
API->>API: generate CSV synchronously (format & stream)
API-->>Frontend: CSV blob (200)
Frontend-->>User: prompt download
else totalCount > MAX_PARTNERS_TO_EXPORT
API->>QStash: enqueue POST job (user + filters)
API-->>Frontend: HTTP 202
Frontend->>User: show toast (email confirmed)
QStash->>Cron: trigger POST job
Cron->>Cron: validate QStash + payload
Cron->>BatchGen: iterate batches via fetchPartnersBatch
BatchGen-->>Cron: partner batches
Cron->>Formatter: map & validate rows
Cron->>R2: upload CSV
Cron->>Email: send PartnerExportReady(downloadUrl)
Email-->>Inbox: deliver download link
end
Estimated code review effortπ― 4 (Complex) | β±οΈ ~45 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touchesβ Failed checks (1 warning)
β Passed checks (2 passed)
π Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro π Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 3
Caution
Some comments are outside the diff and canβt be posted inline due to platform limitations.
β οΈ Outside diff range comments (2)
apps/web/ui/modals/export-partners-modal.tsx (1)
107-111: Pass a string to toast.error.Passing an Error object can render β[object Object]β. Coerce to a message.
- } catch (error) { - toast.error(error); + } catch (error) { + const message = + error instanceof Error ? error.message : typeof error === "string" ? error : "Something went wrong"; + toast.error(message);apps/web/app/(ee)/api/partners/export/route.ts (1)
67-85: Validate and sanitize requested columns against the allowlist.Unvalidated columns can produce undefined labels and zod schema keys, causing runtime errors or malformed CSV. Filter and de-duplicate before building the schema.
- const schemaFields = {}; - columns.forEach((column) => { + // Enforce allowlist and stable order + const allowed = new Set(exportPartnerColumns.map((c) => c.id)); + columns = Array.from(new Set(columns.filter((c) => allowed.has(c)))); + + const schemaFields = {} as Record<string, z.ZodTypeAny>; + columns.forEach((column) => { if (numericColumns.includes(column)) { schemaFields[columnIdToLabel[column]] = z.coerce .number() .optional() .default(0); } else { schemaFields[columnIdToLabel[column]] = z .string() .optional() .default(""); } });
π§Ή Nitpick comments (8)
apps/web/ui/modals/export-partners-modal.tsx (1)
89-95: Guard against missing email in success toast.If session.user.email is undefined, users will see βundefinedβ. Add a fallback or conditionally omit the email.
- toast.success( - `Your export is being processed and we'll send you an email (${session?.user?.email}) when it's ready to download.`, - ); + const emailSuffix = session?.user?.email ? ` (${session.user.email})` : ""; + toast.success( + `Your export is being processed and we'll email you${emailSuffix} when it's ready.`, + );apps/web/app/(ee)/api/cron/partners/export/fetch-partners-batch.ts (1)
12-34: Generator logic looks solid. Consider cancellability.Add optional AbortSignal to stop long-running exports when jobs are canceled.
-export async function* fetchPartnersBatch( - filters: PartnerFilters, - batchSize: number = 1000, -) { +export async function* fetchPartnersBatch( + filters: PartnerFilters, + batchSize: number = 1000, + signal?: AbortSignal, +) { let page = 1; let hasMore = true; while (hasMore) { + if (signal?.aborted) break; const partners = await getPartners({ ...filters, page, pageSize: batchSize, });packages/email/src/templates/partner-export-ready.tsx (1)
48-57: Improve link UX for email clients.Open in new tab and include a plaintext fallback link for clients that strip button styles.
- <Link - className="rounded-lg bg-black px-6 py-3 text-center text-[12px] font-semibold text-white no-underline" - href={downloadUrl} - > + <Link + className="rounded-lg bg-black px-6 py-3 text-center text-[12px] font-semibold text-white no-underline" + href={downloadUrl} + target="_blank" + rel="noopener noreferrer" + > Download Export </Link> + <Text className="mt-3 text-[12px] text-neutral-500"> + If the button doesnβt work, copy and paste this URL into your browser: {downloadUrl} + </Text>apps/web/app/(ee)/api/partners/export/route.ts (1)
106-111: Include a filename in Content-Disposition.Improves UX when calling the route directly (without the UI blob name).
- return new Response(convertToCSV(formattedPartners), { + const filename = `dub-partners-export-${new Date().toISOString()}.csv`; + return new Response(convertToCSV(formattedPartners), { headers: { "Content-Type": "text/csv", - "Content-Disposition": "attachment", + "Content-Disposition": `attachment; filename="${filename}"`, }, });apps/web/app/(ee)/api/partners/count/route.ts (1)
13-16: Response shape may vary; consider normalizing or typing.getPartnersCount returns either a number or grouped rows. If clients consume this route generically, standardize the JSON shape (e.g., { total } vs { groups, groupBy }) or at least annotate the union type for clarity.
apps/web/app/(ee)/api/cron/partners/export/route.ts (3)
145-153: Set a filename in Content-Disposition.Improve download UX and avoid generic names.
- headers: { - "Content-Type": "text/csv", - "Content-Disposition": "attachment", - }, + headers: { + "Content-Type": "text/csv", + // Example: partners-2025-10-27.csv + "Content-Disposition": `attachment; filename="partners-${new Date().toISOString().slice(0,10)}.csv"`, + },
126-136: Currency/Decimal precision: avoid lossy number coercion.Fields like totalSaleAmount/netRevenue may be Decimal/BigInt; z.coerce.number can lose precision. Prefer strings (or fixed 2βdp formatting) for CSV.
- if (numericColumns.includes(column)) { - schemaFields[columnIdToLabel[column]] = z.coerce.number().optional().default(0); - } else { + if (numericColumns.includes(column)) { + // Keep as string to preserve precision; format upstream if needed. + schemaFields[columnIdToLabel[column]] = z.string().optional().default("0"); + } else {And stringify Decimals when mapping:
- let value = partner[column] || ""; + let value = partner[column] ?? ""; + if (numericColumns.includes(column) && value != null) { + value = typeof value === "object" && "toString" in value ? String(value) : String(value); + }
175-180: Defensive logging of unknown errors.error may be non-Error. Use a safe message fallback and include context (programId, userId) when available.
- await log({ - message: `Error exporting partners: ${error.message}`, + const msg = error instanceof Error ? error.message : JSON.stringify(error); + await log({ + message: `Error exporting partners: ${msg}`, type: "cron", });
π Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
π Files selected for processing (7)
apps/web/app/(ee)/api/cron/partners/export/fetch-partners-batch.ts(1 hunks)apps/web/app/(ee)/api/cron/partners/export/route.ts(1 hunks)apps/web/app/(ee)/api/partners/count/route.ts(1 hunks)apps/web/app/(ee)/api/partners/export/route.ts(2 hunks)apps/web/lib/api/partners/get-partners-count.ts(1 hunks)apps/web/ui/modals/export-partners-modal.tsx(3 hunks)packages/email/src/templates/partner-export-ready.tsx(1 hunks)
π§° Additional context used
𧬠Code graph analysis (6)
apps/web/app/(ee)/api/partners/count/route.ts (2)
apps/web/lib/zod/schemas/partners.ts (1)
partnersCountQuerySchema(182-191)apps/web/lib/api/partners/get-partners-count.ts (1)
getPartnersCount(10-149)
apps/web/app/(ee)/api/cron/partners/export/fetch-partners-batch.ts (2)
apps/web/lib/zod/schemas/partners.ts (1)
partnersExportQuerySchema(171-180)apps/web/lib/api/partners/get-partners.ts (1)
getPartners(9-74)
apps/web/app/(ee)/api/partners/export/route.ts (4)
apps/web/lib/auth/workspace.ts (1)
withWorkspace(42-436)apps/web/lib/zod/schemas/partners.ts (1)
partnersExportQuerySchema(171-180)apps/web/lib/api/partners/get-partners-count.ts (1)
getPartnersCount(10-149)apps/web/lib/api/partners/get-partners.ts (1)
getPartners(9-74)
apps/web/app/(ee)/api/cron/partners/export/route.ts (6)
apps/web/lib/zod/schemas/partners.ts (2)
partnersExportQuerySchema(171-180)exportPartnerColumns(27-65)apps/web/app/(ee)/api/cron/partners/export/fetch-partners-batch.ts (1)
fetchPartnersBatch(12-34)apps/web/lib/api/utils/generate-random-string.ts (1)
generateRandomString(3-14)packages/email/src/index.ts (1)
sendEmail(6-29)packages/email/src/templates/partner-export-ready.tsx (1)
PartnerExportReady(17-64)apps/web/lib/api/errors.ts (1)
handleAndReturnErrorResponse(175-178)
packages/email/src/templates/partner-export-ready.tsx (2)
packages/email/src/react-email.d.ts (11)
Html(4-4)Head(5-5)Preview(17-17)Tailwind(18-18)Body(6-6)Container(7-7)Section(8-8)Img(13-13)Heading(16-16)Text(15-15)Link(14-14)packages/ui/src/footer.tsx (1)
Footer(106-344)
apps/web/lib/api/partners/get-partners-count.ts (3)
apps/web/lib/zod/schemas/partners.ts (1)
partnersCountQuerySchema(182-191)packages/prisma/client.ts (2)
Prisma(27-27)ProgramEnrollmentStatus(28-28)packages/prisma/index.ts (2)
sanitizeFullTextSearch(19-22)prisma(3-9)
β° Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
π Additional comments (5)
apps/web/app/(ee)/api/partners/export/route.ts (1)
40-53: Review comment is incorrect.The
APP_DOMAIN_WITH_NGROKconstant is production-safe by designβit returns the production app domain whenNEXT_PUBLIC_VERCEL_ENV === "production". The code does not break in production.Additionally, the suggested environment variables (
DUB_PUBLIC_API_URL,DUB_APP_URL,VERCEL_PROJECT_PRODUCTION_URL) do not exist in the codebase. This same pattern of usingAPP_DOMAIN_WITH_NGROKfor QStash callbacks is used consistently throughout the entire codebase across 60+ files, establishing it as the standard approach.Likely an incorrect or invalid review comment.
apps/web/app/(ee)/api/partners/count/route.ts (2)
1-1: Centralizing count logic is a good move.Importing and delegating to getPartnersCount reduces duplication and keeps filters consistent.
11-11: The review comment is incorrect.searchParamsis already a plain object, not URLSearchParams.The
getSearchParams()utility function (packages/utils/src/functions/urls.ts:40-49) explicitly converts URLSearchParams to a plain object by iterating through search parameters and building aRecord<string, string>. This conversion happens withinwithWorkspacebefore passingsearchParamsto the handler, so the code at line 8-10 is already correct and requires no changes.Likely an incorrect or invalid review comment.
apps/web/app/(ee)/api/cron/partners/export/route.ts (2)
35-44: Good: verify QStash signature before parsing and side effects.Ordering avoids TOCTOU risks and prevents processing spoofed requests.
110-141: Storage API does not support streaming; review's suggested approach is not feasible with current implementation.The memory concern is validβaccumulating all partners and then converting to a full CSV string risks OOM on large exports. However,
storage.upload()only acceptsBlob | Buffer | string(no streams), andconvertToCSV()requires the entire object array upfront.The suggested temp file + pipe approach won't work because:
storage.upload()does not accept Node.js streams or Readable typesaws4fetchAwsClient expects body to have.sizeproperty (Blob/Buffer only)- Temp file would still need to be read as Blob before uploading
Better alternatives:
- Accept bounded memory risk if typical exports are < 50k partners
- Batch-convert smaller chunks (e.g., per 1k records) and concatenate CSV strings after headers, then Blob upload
- Implement S3 multipart upload with presigned URLs (requires new upload flow)
For now, document the memory constraint or refactor to batch CSV generation per 1β2k records to keep peak memory lower.
| await sendEmail({ | ||
| to: user.email, | ||
| subject: "Your partner export is ready", | ||
| react: PartnerExportReady({ | ||
| email: user.email, | ||
| downloadUrl: uploadResult.url, | ||
| program: { | ||
| name: program.name, | ||
| }, | ||
| }), | ||
| }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
π§© Analysis chain
Email says βlink expires in 7 daysβ but uploaded URL likely doesnβt.
storage.upload returns a URL that may be public and non-expiring. Either generate a signed URL with 7βday TTL, or set an object TTL/lifecycle and send a signed download route. Otherwise the email copy is misleading and creates privacy/retention risk.
If signed URLs are supported:
- await sendEmail({
+ // const signedUrl = await storage.getSignedUrl(fileKey, { expiresIn: 7 * 24 * 60 * 60 });
+ await sendEmail({
to: user.email,
subject: "Your partner export is ready",
react: PartnerExportReady({
email: user.email,
- downloadUrl: uploadResult.url,
+ downloadUrl: signedUrl ?? uploadResult.url,
program: {
name: program.name,
},
}),
});Otherwise, schedule deletion after 7 days and adjust copy.
π Script executed:
#!/bin/bash
# Verify if storage supports signed URLs or object TTL/lifecycle.
rg -nC2 'getSignedUrl|signed|ttl|lifecycle|upload\(' -g 'apps/** lib/** packages/**'Length of output: 148
I need to search the codebase more effectively. Let me try a different approach to find the relevant files and implementations.
π Script executed:
#!/bin/bash
# Find the specific file being reviewed
find . -name "route.ts" -path "*/api/cron/partners/export/*" 2>/dev/null | head -5
# Also search more broadly for storage and signed URL patterns
echo "=== Searching for storage.upload usage ==="
rg -n "storage\.upload\|getSignedUrl\|SignedUrl" --type ts --type js 2>/dev/null | head -20
# Search for PartnerExportReady component/template
echo "=== Searching for PartnerExportReady ==="
rg -n "PartnerExportReady" --type ts --type js --type tsx 2>/dev/null | head -10
# Search for email template or component with "expires" or "7 days"
echo "=== Searching for expiration mentions in emails ==="
rg -n "expires|7.days|7-day" --type ts --type js --type tsx 2>/dev/null | head -20Length of output: 189
Let me examine the actual file and search for the storage implementation more broadly:
π Script executed:
#!/bin/bash
# Read the specific file
echo "=== File content: route.ts ==="
cat ./apps/web/app/\(ee\)/api/cron/partners/export/route.ts | head -200
# Search for storage implementations by looking at imports and common patterns
echo "=== Searching for storage implementations ==="
rg -n "storage|Storage|upload|Upload" --type ts --type js --type tsx --max-count 50 2>/dev/null | grep -i "import\|export\|function\|const.*=" | head -30
# Find PartnerExportReady in all file types
echo "=== Searching for PartnerExportReady more broadly ==="
find . -type f \( -name "*.ts" -o -name "*.tsx" -o -name "*.js" -o -name "*.jsx" \) 2>/dev/null | xargs grep -l "PartnerExportReady" 2>/dev/null | head -10
# Search for email components
echo "=== Searching for email templates/components ==="
find . -path "*/emails/*" -o -path "*/email/*" 2>/dev/null | grep -E "\.(ts|tsx|js)$" | head -20Length of output: 6024
Now let me check the storage implementation and the email template:
π Script executed:
#!/bin/bash
# Find and read storage implementation
echo "=== Finding storage implementation ==="
find . -path "*/lib/storage*" -name "*.ts" -o -path "*/lib/storage*" -name "*.js" 2>/dev/null | head -10
# Read the storage file(s)
echo "=== Storage module content ==="
find . -path "*/lib/storage.ts" -o -path "*/lib/storage/index.ts" 2>/dev/null | head -1 | xargs cat 2>/dev/null
# Read PartnerExportReady template
echo "=== PartnerExportReady email template ==="
cat ./packages/email/src/templates/partner-export-ready.tsx 2>/dev/null
# Search for getSignedUrl in the entire lib/storage path
echo "=== Searching for signed URL support ==="
find . -path "*/lib/storage*" -type f 2>/dev/null | xargs grep -l "sign\|Sign" 2>/dev/nullLength of output: 6505
Use signed URL with 7-day TTL for download links to match email promise.
The email template states "This download link will expire in 7 days," but the code passes a plain public URL from storage.upload() that doesn't expire. The storage.getSignedUrl() method exists but is hardcoded to 10-minute expiration and not being called.
Update the route to generate a signed URL with proper 7-day TTL:
const uploadResult = await storage.upload(fileKey, csvBlob, {
contentType: "text/csv",
headers: {
"Content-Type": "text/csv",
"Content-Disposition": "attachment",
},
});
if (!uploadResult || !uploadResult.url) {
throw new Error("Failed to upload CSV to storage.");
}
await sendEmail({
to: user.email,
subject: "Your partner export is ready",
react: PartnerExportReady({
email: user.email,
- downloadUrl: uploadResult.url,
+ downloadUrl: await storage.getSignedUrl(fileKey),
program: {
name: program.name,
},
}),
});Additionally, update storage.getSignedUrl() to accept a TTL parameter (currently hardcoded to 600 seconds) and call it with 7 days (604800 seconds). Verify the signed URL is configured for GET requests, not PUT.
Committable suggestion skipped: line range outside the PR's diff.
π€ Prompt for AI Agents
In apps/web/app/(ee)/api/cron/partners/export/route.ts around lines 159 to 169,
the code sends the plain public upload URL to users while the email promises a
7-day expiry; replace use of uploadResult.url with a signed GET URL generated
via storage.getSignedUrl(..., ttl) and call it with a 7-day TTL of 604800
seconds. Update the storage.getSignedUrl implementation to accept a TTL
parameter (instead of hardcoding 600), use that TTL when creating the signed
URL, and ensure the signed URL is created for HTTP GET (not PUT). Finally,
update the route to request the signed GET URL with ttl=604800 and pass that
signed URL into the email payload.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
π Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
π Files selected for processing (2)
apps/web/app/(ee)/api/cron/partners/export/route.ts(1 hunks)packages/email/src/templates/partner-export-ready.tsx(1 hunks)
π§ Files skipped from review as they are similar to previous changes (1)
- apps/web/app/(ee)/api/cron/partners/export/route.ts
π§° Additional context used
𧬠Code graph analysis (1)
packages/email/src/templates/partner-export-ready.tsx (1)
packages/email/src/react-email.d.ts (11)
Html(4-4)Head(5-5)Preview(17-17)Tailwind(18-18)Body(6-6)Container(7-7)Section(8-8)Img(13-13)Heading(16-16)Text(15-15)Link(14-14)
β° Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
π Additional comments (3)
packages/email/src/templates/partner-export-ready.tsx (3)
1-15: LGTM! Clean import structure.All imports are properly organized and used throughout the template.
30-54: LGTM! Well-structured email template.The email structure follows React Email best practices with proper component usage, good accessibility (alt text for image), and appropriate personalization using
program.name. The download button styling and overall layout are clean and professional.Also applies to: 58-63
55-57: Verify whether the expiration notice should be enabled.The expiration notice is currently commented out. If exports expire after 7 days, this information would be valuable for users to know. Please confirm whether this should be enabled or if it's intentionally disabled because exports don't expire.
Summary by CodeRabbit
New Features
Improvements