Async commissions export #3015
Async commissions export #3015
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThis PR refactors the commissions data retrieval and export infrastructure by extracting shared utility functions (getCommissions, getCommissionsCount, formatCommissionsForExport) and implementing background processing via QStash for large exports. It simplifies existing API routes by delegating to these utilities, introduces batch processing for exports, and updates the export email template to be generic across export types. Changes
Sequence DiagramsequenceDiagram
participant User
participant Modal as Export Modal
participant API as /api/commissions/export
participant QStash
participant Cron as /api/cron/commissions/export
participant Storage as R2 Storage
participant Email as Email Service
User->>Modal: Click Export
Modal->>API: POST /api/commissions/export
rect rgb(200, 220, 255)
Note over API: Check export size
API->>API: count = getCommissionsCount()
end
alt Large Export (> threshold)
rect rgb(255, 200, 200)
Note over API: Queue background job
API->>QStash: Queue export task
API-->>Modal: 202 Accepted
Modal->>User: Show toast (will email results)
end
QStash->>Cron: Trigger export job
rect rgb(200, 255, 200)
Note over Cron: Background processing
Cron->>Cron: fetchCommissionsBatch() → stream
Cron->>Cron: formatCommissionsForExport()
Cron->>Cron: Convert to CSV
end
Cron->>Storage: Upload CSV to R2
Cron->>Email: Send download link to user
Email-->>User: Email with ready-export link
else Small Export (≤ threshold)
rect rgb(200, 255, 200)
Note over API: Inline processing
API->>API: getCommissions() + formatCommissionsForExport()
API->>API: Convert to CSV
end
API-->>Modal: 200 OK + CSV
Modal->>User: Download starts immediately
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…e customer transformation for exports
|
@coderabbitai review please |
|
@devkiran: I'll review the changes in this PR for you. ✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (9)
packages/email/src/templates/partner-export-ready.tsx (1)
17-31: LGTM!The generalization from
PartnerExportReadytoExportReadywith theexportTypeprop enables reuse across multiple export types while maintaining backward compatibility with the default value.Optional: Consider pluralizing the export type label for better grammar.
The phrasing "Your export of partners from..." reads naturally, but you could enhance it slightly:
const exportTypeLabel = exportType === "commissions" ? "commissions" : "partners";Then use it in the text. However, this is a minor stylistic point and the current implementation is acceptable.
Also applies to: 35-48
apps/web/app/(ee)/api/cron/commissions/export/route.ts (2)
68-84: Avoid loading entire export into memory; stream CSV to storage.Building an in-memory array and one big Blob risks high memory usage and long GC pauses for large exports. Prefer line-by-line/row-chunk streaming to R2 (multipart or streaming upload) while iterating the async generator.
Example adjustment outline:
- Convert batches to CSV rows incrementally.
- Initiate multipart upload; upload parts per N rows.
- Finalize upload and then email the link.
If streaming is not available in your storage SDK, buffer modest chunks (e.g., 5–10 MB) before flushing to a part. This keeps memory bounded.
Also applies to: 85-96
101-112: Defensive email payload: handle nullable program name.If program.name can be null, the template might render poorly. Fallback to a safe string.
- program: { - name: program.name, - }, + program: { + name: program.name ?? "Program", + },apps/web/lib/api/commissions/get-commissions-count.ts (1)
49-58: Nested partner.programs.some filter may be expensive; consider pre-resolving partnerIds or adding an index.Filtering via a nested relation in a groupBy can be slow at scale. If feasible, prefetch the partnerIds for (programId, groupId) and use partnerId: { in: [...] }. At minimum, ensure an index covers (programId, groupId) on the join table.
apps/web/lib/api/commissions/format-commissions-for-export.ts (3)
49-53: Don’t mutate the input columns array when sorting.Sorting in place can produce side effects for callers.
- const sortedColumns = columns.sort( + const sortedColumns = [...columns].sort( (a, b) => (COLUMN_LOOKUP.get(a)?.order || 999) - (COLUMN_LOOKUP.get(b)?.order || 999), );
38-46: Select partnerTenantId for the current program rather than the first enrollment.Using programs[0] is brittle for partners in multiple programs. Prefer the enrollment that matches the commission’s program.
If commission carries programId or partnerEnrollment, use that; otherwise, fallback:
- partnerTenantId: commission.partner?.programs[0]?.tenantId || "", + partnerTenantId: + commission.partnerEnrollment?.tenantId ?? + commission.partner?.programs?.find?.((p: any) => p.programId === commission.programId)?.tenantId ?? + "",If programId isn’t available here, consider passing it as a third parameter to formatCommissionsForExport.
55-66: Column schema construction is good; pair with ordered headers for CSV.Schema-driven shaping ensures consistent row objects. To guarantee header order for downstream CSV, return both data and the ordered column list from this function.
-export function formatCommissionsForExport( +export function formatCommissionsForExport( commissions: any[], columns: string[], -): Record<string, any>[] { +): { rows: Record<string, any>[]; columns: string[] } { ... - return z.array(z.object(columnSchemas)).parse(formattedCommissions); + const rows = z.array(z.object(columnSchemas)).parse(formattedCommissions); + return { rows, columns: sortedColumns }; }Then, update callers to use convertToCSV(rows, { columns: sortedColumns }).
Also applies to: 68-69
apps/web/app/(ee)/api/commissions/export/route.ts (2)
40-40: Provide tracking information in the 202 response.The 202 Accepted response returns an empty object. Consider including information to help users track their export, such as a message indicating the export is being processed and will be emailed, or an estimated completion time.
- return NextResponse.json({}, { status: 202 }); + return NextResponse.json( + { + message: "Export is being processed. You will receive an email when it's ready.", + estimatedTime: "5-10 minutes" + }, + { status: 202 } + );
57-62: Add filename to Content-Disposition header.The
Content-Dispositionheader is set toattachmentbut doesn't include a filename. Browsers will use a generic name. Consider adding a descriptive filename with timestamp:headers: { "Content-Type": "text/csv", - "Content-Disposition": "attachment", + "Content-Disposition": `attachment; filename="commissions-export-${new Date().toISOString().split('T')[0]}.csv"`, },
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (14)
apps/web/app/(ee)/api/commissions/count/route.ts(1 hunks)apps/web/app/(ee)/api/commissions/export/route.ts(1 hunks)apps/web/app/(ee)/api/commissions/route.ts(3 hunks)apps/web/app/(ee)/api/cron/commissions/export/fetch-commissions-batch.ts(1 hunks)apps/web/app/(ee)/api/cron/commissions/export/route.ts(1 hunks)apps/web/app/(ee)/api/cron/partners/export/fetch-partners-batch.ts(2 hunks)apps/web/app/(ee)/api/cron/partners/export/route.ts(4 hunks)apps/web/lib/api/commissions/format-commissions-for-export.ts(1 hunks)apps/web/lib/api/commissions/get-commissions-count.ts(1 hunks)apps/web/lib/api/commissions/get-commissions.ts(1 hunks)apps/web/lib/api/utils/generate-export-filename.ts(1 hunks)apps/web/ui/modals/export-commissions-modal.tsx(4 hunks)apps/web/ui/modals/export-partners-modal.tsx(2 hunks)packages/email/src/templates/partner-export-ready.tsx(1 hunks)
🔇 Additional comments (14)
apps/web/app/(ee)/api/cron/partners/export/fetch-partners-batch.ts (1)
12-34: LGTM!The parameter rename from
batchSizetopageSizeimproves clarity and accurately reflects its usage. All internal references are updated consistently.apps/web/ui/modals/export-partners-modal.tsx (1)
1-1: LGTM!The adoption of the
generateExportFilenameutility provides consistent, timestamp-based filename generation across export flows.Also applies to: 103-103
apps/web/ui/modals/export-commissions-modal.tsx (1)
1-1: LGTM!The adoption of
generateExportFilenameanduseSessionenables consistent filename generation and personalized user feedback for async export flows.Also applies to: 102-102
apps/web/app/(ee)/api/commissions/count/route.ts (1)
1-19: LGTM!The refactoring to delegate count logic to the
getCommissionsCountutility reduces duplication and centralizes data aggregation logic.apps/web/app/(ee)/api/cron/commissions/export/fetch-commissions-batch.ts (1)
12-35: LGTM!The async generator pattern for batch processing is well-suited for large exports. The implementation is consistent with the partners batch fetcher and correctly handles pagination.
apps/web/lib/api/commissions/get-commissions.ts (1)
11-90: LGTM!The
getCommissionsutility provides flexible filtering with proper pagination, conditional includes, and date-range handling. The dual where-clause logic (invoiceId vs. earnings-based) and conditional program enrollment inclusion are correctly implemented.apps/web/app/(ee)/api/cron/partners/export/route.ts (2)
4-4: Filename generation and unified ExportReady template look good.Using generateExportFilename and passing exportType: "partners" standardizes UX and download behavior.
Also applies to: 10-10, 91-91, 102-106
75-75: No action required—default batch size confirms no throughput regression.The
fetchPartnersBatchfunction infetch-partners-batch.ts(line 14) has a defaultpageSize: number = 1000. Removing the explicit parameter at the call site has no effect; the function will still use 1000 as the batch size. There is no throughput regression or runtime cost increase.Likely an incorrect or invalid review comment.
apps/web/app/(ee)/api/commissions/route.ts (1)
18-20: Verification found inconsistency: many API routes still use direct Prisma queries instead of delegating to getCommissions.The refactored commissions/route.ts correctly uses getCommissions, but verification revealed that several other API routes bypass it entirely:
- apps/web/app/(ee)/api/partners/analytics/route.ts
- apps/web/app/(ee)/api/partner-profile/.../earnings/route.ts
- apps/web/app/(ee)/api/embed/referrals/earnings/route.ts
- apps/web/app/(ee)/api/cron/payouts/aggregate-due-commissions/route.ts
- apps/web/app/(ee)/api/cron/partner-program-summary/route.ts
Confirm whether these should also delegate to getCommissions or if their operations (groupBy, aggregate) require different query patterns.
apps/web/lib/api/commissions/get-commissions-count.ts (1)
67-85: No issues found with status handling — code is correct.The CommissionStatus enum contains only active statuses (pending, processed, paid, refunded, duplicate, fraud, canceled) with no deprecated or legacy values. All seven enum values are:
- Mapped in CommissionStatusBadges with complete coverage
- Properly handled by the forEach() loop at line 87 (filling missing statuses)
- Safely used by downstream consumers
- Type-checked via the Record<CommissionStatus | "all", ...> annotation
The verification confirms the original concern about deprecated statuses is not applicable.
apps/web/app/(ee)/api/cron/commissions/export/route.ts (1)
83-84: The review comment's concern is valid, but the suggested fix code is incorrect and incompatible with the current API.Here's why:
convertToCSV only accepts a single
dataparameter—it does not accept a second options parameter. The suggested callconvertToCSV(allCommissions, { columns: ..., headers: ... })will not work.json2csv (the underlying library) does support column ordering via
options.keys, butconvertToCSVdoesn't expose that parameter to its callers.formatCommissionsForExport sorts the columns array and builds a Zod schema in that order, but there's no guarantee that the resulting parsed objects will preserve key order when passed to
convertToCSV, since json2csv will auto-detect header order from the first object's key iteration order.The legitimate fix requires modifying
convertToCSVitself to accept and pass through json2csv options (specifically thekeysparameter), or modifyingformatCommissionsForExportto return both sorted data and an explicit ordered keys array for use with convertToCSV.Likely an incorrect or invalid review comment.
apps/web/app/(ee)/api/commissions/export/route.ts (3)
23-26: No action required—code is type-safe.The function
getCommissionsCountexplicitly constructs and returns an object wherecounts.allis always present with a guaranteed structure{ count: number; amount: number; earnings: number }(lines 91–97 of get-commissions-count.ts). The usage ofcounts.all.countin the export route is type-safe and requires no validation or type guards.Likely an incorrect or invalid review comment.
16-16: Session type is already guaranteed as non-null by WithWorkspaceHandler interface.The
sessionparameter in theWithWorkspaceHandlerinterface is typed as required (session: Session) rather than optional. ThewithWorkspacefunction validates this guarantee through a guard clause that throws if session orsession.user.idis falsy, ensuring handlers always receive a valid session object. No additional null checks are needed.Likely an incorrect or invalid review comment.
9-9: Incorrect review comment – no production safety issue.The
APP_DOMAIN_WITH_NGROKconstant correctly resolves tohttps://app.${NEXT_PUBLIC_APP_DOMAIN}in production environments (whenNEXT_PUBLIC_VERCEL_ENV === "production") and to preview or local development URLs otherwise. The constant name is appropriate—it reflects that ngrok can optionally be used in local development via theNEXT_PUBLIC_NGROK_URLenvironment variable, but production deployments always use the correct production domain. The QStash callback URL at line 31 is safe and follows the established pattern used throughout the codebase.Likely an incorrect or invalid review comment.
| url: `${APP_DOMAIN_WITH_NGROK}/api/cron/commissions/export`, | ||
| body: { | ||
| ...parsedParams, | ||
| columns: columns.join(","), |
There was a problem hiding this comment.
Handle edge case where column names contain commas.
Converting the columns array to a CSV string with columns.join(",") could cause parsing issues if any column name contains a comma. Consider using JSON serialization instead, or verify that column names are sanitized.
- columns: columns.join(","),
+ columns: JSON.stringify(columns),Then ensure the cron job handler parses this with JSON.parse(columns).
Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In apps/web/app/(ee)/api/commissions/export/route.ts around line 34, the code
uses columns.join(",") which breaks when column names contain commas; replace
columns.join(",") with JSON.stringify(columns) when serializing the columns into
the request and update the cron job handler to parse the incoming value with
JSON.parse(columns) (or validate/sanitize after parse) so arrays with commas in
names are preserved correctly.
| const commissions = await getCommissions({ | ||
| ...filters, | ||
| programId, | ||
| page: 1, | ||
| pageSize: MAX_COMMISSIONS_TO_EXPORT, | ||
| includeProgramEnrollment: true, | ||
| }); |
There was a problem hiding this comment.
Consider race condition between count check and data fetch.
There's a potential race condition: if commissions are created between the getCommissionsCount call (line 23) and the getCommissions call (line 44), the synchronous path might miss records or fetch more than expected. For critical exports, consider using database transactions or document this behavior.
| const { startDate, endDate } = getStartEndDates({ | ||
| interval, | ||
| start, | ||
| end, | ||
| }); | ||
|
|
There was a problem hiding this comment.
Avoid toISOString on Date filters to prevent TZ drift; pass Date objects.
Converting to ISO strings can shift the window in non‑UTC timezones. Prisma accepts Date instances directly.
- createdAt: {
- gte: startDate.toISOString(),
- lte: endDate.toISOString(),
- },
+ createdAt: {
+ gte: startDate,
+ lte: endDate,
+ },If the intent is end‑exclusive, use lt and normalize endDate to the next tick/day.
Also applies to: 45-49
🤖 Prompt for AI Agents
In apps/web/lib/api/commissions/get-commissions-count.ts around lines 27-32 (and
similarly lines 45-49), the code converts startDate/endDate to ISO strings which
causes timezone drift; instead keep and pass Date objects directly into Prisma
filters, and if the intent is an end-exclusive window use the appropriate lt
operator and normalize endDate to the next tick/day (e.g., add one day or set
time to 23:59:59.999 then use lt) so the filter is accurate across timezones.
|
/bug0 run |
Summary by CodeRabbit
New Features
Improvements