-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Handle payouts in batches of 100 #3002
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
WalkthroughAdds QStash-based batching to the charge-succeeded cron route with a 100-item batch cap, changes payout selection to Changes
Sequence Diagram(s)sequenceDiagram
participant Route as Cron Route
participant DB as Prisma DB
participant Processor as Payout Processor
participant Qstash as QStash
participant Log as logAndRespond
Note over Route,DB: Fetch batch (status="processing", take=100)
Route->>DB: findMany payouts (status="processing", take=100)
DB-->>Route: returns N payouts (N ≤ 100)
Route->>Processor: process N payouts (PayPal or Stripe)
Processor-->>DB: update payouts -> status="sent", paidAt=now (PayPal case)
Processor-->>Route: processing result
alt More payouts remain (>100 total)
Route->>Qstash: publish { invoiceId } -> APP_DOMAIN_WITH_NGROK
Qstash-->>Route: messageId / error
Route->>Log: logAndRespond("Next batch scheduled", messageId)
Route-->>Client: logAndRespond response
else All payouts processed
Route->>Log: logAndRespond("All payouts completed")
Route-->>Client: logAndRespond response
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Possibly related PRs
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
38-46: Filter mismatch between count query and processing logic.The count query includes ALL payouts with
status != "completed", but the actual processing applies stricter filters:
- Stripe (send-stripe-payouts.ts line 34):
status = "processing"ANDstripeConnectId != null- PayPal (send-paypal-payouts.ts lines 10-20):
status != "completed"ANDpaypalEmail != nullThis discrepancy means the count could be 200 but only 50 payouts are actually processable. The batching logic at line 93 would then schedule unnecessary follow-up jobs that find no work.
Consider aligning the count query with the actual processing filters, or use a combined count of both pools:
const invoice = await prisma.invoice.findUnique({ where: { id: invoiceId, }, include: { _count: { select: { - payouts: { - where: { - status: { - not: "completed", - }, - }, - }, + stripePayouts: { + where: { + status: "processing", + partner: { + payoutsEnabledAt: { not: null }, + stripeConnectId: { not: null }, + }, + }, + }, + paypalPayouts: { + where: { + status: { not: "completed" }, + partner: { + payoutsEnabledAt: { not: null }, + paypalEmail: { not: null }, + }, + }, + }, }, }, }, });Then update line 93 to check:
if (invoice._count.stripePayouts + invoice._count.paypalPayouts > 100)
🧹 Nitpick comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-stripe-payouts.ts (1)
55-64: Consider adding a limit to prevent performance issues.The
previouslyProcessedPayoutsquery has notakelimit. While it's scoped to partners in the current batch (max 100 partners), each partner could accumulate many previously processed payouts if transfers have been failing. This could lead to performance degradation in edge cases.Consider adding a reasonable limit or an index to optimize this query:
const previouslyProcessedPayouts = await prisma.payout.findMany({ where: { status: "processed", stripeTransferId: null, partnerId: { in: currentInvoicePayouts.map((p) => p.partnerId), }, }, include: commonInclude, + take: 1000, // reasonable upper bound to prevent performance issues });
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts(2 hunks)apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-paypal-payouts.ts(1 hunks)apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-stripe-payouts.ts(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (2)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-paypal-payouts.ts (1)
36-36: LGTM!The
take: 100limit correctly restricts the batch size and aligns with the batching strategy.apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-stripe-payouts.ts (1)
45-45: LGTM!The
take: 100limit correctly restricts the batch size for current invoice payouts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (2)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (2)
22-24: Spec/docs mismatch: “batches of 100” but current run can process up to 200 (100 Stripe + 100 PayPal).Either enforce 100 total per run (split remaining between providers) or update wording to “up to 200 (100 per provider)”. This was raised earlier and still applies.
91-116: Don’t use the stale pre-batch count to schedule next batch; re-count after writes, add a short QStash delay and dedup.
- invoice._count.payouts was fetched before processing; using it can schedule unnecessary follow-ups or miss when >100 but processed fully this run.
- Re-count remaining “processing” payouts after Promise.allSettled and schedule only if remaining > 0.
- Add a small delay (e.g., "5s") to avoid reading from replicas before writes are visible and enable content-based deduplication to prevent duplicate follow-ups. QStash supports delay and contentBasedDeduplication in publishJSON. (upstash.com)
Apply:
- if (invoice._count.payouts > 100) { - console.log( - "More than 100 payouts found for invoice, scheduling next batch...", - ); - const qstashResponse = await qstash.publishJSON({ - url: `${APP_DOMAIN_WITH_NGROK}/api/cron/payouts/charge-succeeded`, - body: { - invoiceId: invoiceId, - }, - }); - if (qstashResponse.messageId) { - console.log( - `Message sent to Qstash with id ${qstashResponse.messageId}`, - ); - } else { - console.error("Error sending message to Qstash", qstashResponse); - } - - return logAndRespond( - `Completed processing current batch of payouts for invoice ${invoiceId}. Next batch scheduled.`, - ); - } - - return logAndRespond( - `Completed processing all payouts for invoice ${invoiceId}.`, - ); + // Re-count remaining after processing to avoid using stale counts + const fresh = await prisma.invoice.findUnique({ + where: { id: invoiceId }, + select: { + _count: { + select: { + payouts: { where: { status: "processing" } }, + }, + }, + }, + }); + const remaining = fresh?._count.payouts ?? 0; + + if (remaining > 0) { + await log({ + message: `Remaining payouts (${remaining}) for invoice ${invoiceId}; scheduling next batch...`, + type: "cron", + }); + const qstashResponse = await qstash.publishJSON({ + url: `${APP_DOMAIN_WITH_NGROK}/api/cron/payouts/charge-succeeded`, + body: { invoiceId }, + delay: "5s", + contentBasedDeduplication: true, + }); + await log({ + message: `QStash publish result: ${qstashResponse.messageId ?? "no messageId"}`, + type: "cron", + }); + return logAndRespond( + `Completed current batch for invoice ${invoiceId}. Next batch scheduled.`, + ); + } + + return logAndRespond(`Completed processing all payouts for invoice ${invoiceId}.`);Note: If you truly want “batches of 100 total”, expose a limit param on sendStripePayouts/sendPaypalPayouts and split the 100 between them; otherwise update PR/title/comments accordingly.
🧹 Nitpick comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
2-7: Prefer structured logging/response helpers over console logs; keep usage consistent.You import qstash, APP_DOMAIN_WITH_NGROK, log, and logAndRespond. Downstream you still use console.log/console.error; consider standardizing on log/logAndRespond for consistency and observability.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts(4 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
41-41: Ensure PayPal sender uses consistent status filtering and verify webhook-driven status updates complete before route retries.Status "processing" correctly indicates ready-to-send payouts. However, there is an idempotency gap:
- Stripe: Uses provider-level idempotency key (
${invoiceId}-${partnerId}) increate-stripe-transfer.ts, preventing double transfers. Status transitions "processing" → "sent" immediately after. ✓- PayPal:
send-paypal-payouts.tsqueries payouts with statusNOT "completed"(line 10), which includes "processing", "failed", and "canceled" states. Unlike Stripe which filters to exactly "processing", this broader filter risks re-sending failed/canceled payouts on route retries. PayPal status transitions happen asynchronously via webhooks (payouts-item-succeeded.ts,payouts-item-failed.ts), so if the route retries before webhook processing completes, the same payouts may be re-queued.Recommendation: Align PayPal status filter to explicitly query
status: "processing"(like Stripe) instead ofNOT "completed", so only genuinely pending payouts are re-sent on retries.
|
@coderabbitai full review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-paypal-payouts.ts(3 hunks)apps/web/app/(ee)/api/paypal/webhook/payouts-item-failed.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-paypal-payouts.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (2)
apps/web/app/(ee)/api/paypal/webhook/payouts-item-failed.ts (1)
12-15: HELD and UNCLAIMED payouts will trigger incorrect failure emails.Mapping HELD and UNCLAIMED to
"processed"causes these payouts to bypass the early return at line 61 and proceed to send failure emails (lines 71-91) with the subject "Your recent partner payout failed."However, these are temporary states, not actual failures:
- HELD: Payment on hold pending compliance review or risk assessment
- UNCLAIMED: Recipient hasn't claimed the payment yet (typically 30 days to claim)
Partners will receive misleading failure notifications for payouts that haven't actually failed.
Consider one of these solutions:
- Keep these as
"processing"to leverage batch retry logic- Add distinct
"held"and"unclaimed"statuses with appropriate handling- Add a conditional check at line 61 to exclude these statuses from failure email flow:
- if (payoutStatus === "processing") { + if (payoutStatus === "processing" || payoutStatus === "held" || payoutStatus === "unclaimed") { await log({ message: `Paypal payout is stuck in processing for invoice ${invoiceId} and partner ${paypalEmail}. PayPal webhook status: ${body.event_type}.${ failureReason ? ` Failure reason: ${failureReason}` : "" }`, type: "errors", }); return; // we only send emails for failed payouts }apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
91-112: Batch size inconsistency: Each batch can process up to 200 payouts.The condition
invoice._count.payouts > 100checks if more than 100 payouts exist, but each batch actually processes up to 200 payouts:
- Stripe: up to 100 payouts (send-stripe-payouts.ts line 45)
- PayPal: up to 100 payouts (send-paypal-payouts.ts line 34)
- Total: up to 200 payouts per batch
This is inconsistent with the PR title "Handle payouts in batches of 100" and could cause confusion about actual batch capacity.
Consider one of these approaches:
Option 1: Adjust the threshold to match actual capacity:
-if (invoice._count.payouts > 100) { +if (invoice._count.payouts > 200) { console.log( - "More than 100 payouts found for invoice, scheduling next batch...", + "More than 200 payouts found for invoice, scheduling next batch...", );Option 2: Coordinate limits so each batch truly processes 100 total:
- Fetch Stripe payouts first with
take: 100- Calculate remaining:
remaining = 100 - stripePayouts.length- Fetch PayPal payouts with
take: remainingOption 3: Update PR title and documentation to reflect "batches of up to 200 payouts (100 per payment method)."
🧹 Nitpick comments (2)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-stripe-payouts.ts (1)
45-45: LGTM: Batch limit correctly applied to current invoice payouts.The
take: 100limit aligns with the PR objective of processing payouts in batches of 100.Consider also limiting the
previouslyProcessedPayoutsquery (line 55) if a partner could accumulate a large number of unprocessed historical payouts. Currently, this query is unbounded and could fetch an arbitrarily large result set, potentially impacting performance:const previouslyProcessedPayouts = await prisma.payout.findMany({ where: { status: "processed", stripeTransferId: null, partnerId: { in: currentInvoicePayouts.map((p) => p.partnerId), }, }, include: commonInclude, + take: 500, // reasonable upper bound for historical payouts per batch });apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-paypal-payouts.ts (1)
49-59: Add try-catch error handling around batch creation and status update to improve reliability and observability.Verification confirms the "sent" status is properly handled throughout the payout lifecycle:
- ✅ Status is recognized across webhooks, UI, and type definitions (PayoutStatus enum from Prisma)
- ✅ Recovery mechanism exists: the Stripe webhook (
payout-paid.ts) transitions "sent" → "completed" when payment arrives- ✅ Implicit retry via cron: if the status update fails, payouts remain "processing" and are retried in the next batch
However, error handling remains missing. While idempotency is protected by PayPal's
sender_item_id, wrapping the batch creation and status update in try-catch would improve observability and prevent silent failures from clouding logs.const batchPayout = await createPayPalBatchPayout({ payouts, invoiceId, }); console.log("PayPal batch payout created", batchPayout); +try { // update the payouts to "sent" status const updatedPayouts = await prisma.payout.updateMany({ where: { id: { in: payouts.map((p) => p.id) }, }, data: { status: "sent", paidAt: new Date(), }, }); console.log(`Updated ${updatedPayouts.count} payouts to "sent" status`); +} catch (error) { + console.error("Failed to update payout statuses to 'sent'", error); + // Payouts will remain in "processing" and be retried in next batch + // PayPal's sender_item_id provides idempotency protection +}
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts(4 hunks)apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-paypal-payouts.ts(3 hunks)apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-stripe-payouts.ts(1 hunks)apps/web/app/(ee)/api/paypal/webhook/payouts-item-failed.ts(1 hunks)apps/web/app/(ee)/api/paypal/webhook/payouts-item-succeeded.ts(1 hunks)
🔇 Additional comments (5)
apps/web/app/(ee)/api/paypal/webhook/payouts-item-succeeded.ts (1)
48-48: LGTM: Correctly preserves existing paidAt timestamp.Using
payout.paidAt ?? new Date()ensures that the original payment timestamp is preserved when a payout is updated multiple times, which aligns with the batch processing flow.apps/web/app/(ee)/api/cron/payouts/charge-succeeded/send-paypal-payouts.ts (2)
10-10: LGTM: Status filter correctly aligned with batching logic.Filtering for
status: "processing"aligns with the route's payout counting logic (route.ts line 41) and ensures only pending payouts are processed in each batch.
34-34: LGTM: Batch limit correctly applied.The
take: 100limit implements the batching behavior described in the PR objective.apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (2)
41-41: LGTM: Status filter correctly aligned across the payout flow.Filtering for
status: "processing"ensures consistent payout selection between the count check and the actual processing functions (send-stripe-payouts.ts and send-paypal-payouts.ts).
80-89: Status inconsistency risk if sendPaypalPayouts fails mid-execution.The core concern is valid but the mechanism differs from the original description.
Promise.allSettled()results are not inspected, so if either function throws, the entire request fails—there's no partial success scenario.However, a real issue exists:
sendPaypalPayoutsupdates payouts to "sent" status (line 50–58 in send-paypal-payouts.ts) before sending emails (line 61–75). If email sending fails, the function throws and payouts remain in "sent" state. On retry, the cron job queries forstatus: "processing"payouts, so these already-"sent" payouts won't be retried.Result: Payouts can be stuck in "sent" state without webhook confirmation or successful email delivery, creating data inconsistency. The same pattern exists for Stripe transfers.
Idempotency via
sender_item_idis sufficient to prevent duplicate PayPal payouts on API retries, but it does not address the status synchronization issue.Recommended action: Consider wrapping the DB update and email send in a transaction or restructuring to send emails before transitioning status, ensuring state consistency if operations fail mid-execution.
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
120-123: Avoid accessing error.message on unknown; this can throw inside catch.Harden logging to handle non-Error throwables.
- await log({ - message: `Error sending payouts for invoice: ${error.message}`, - type: "cron", - }); + const message = + error instanceof Error ? error.message : JSON.stringify(error); + await log({ + message: `Error sending payouts for invoice: ${message}`, + type: "cron", + });
♻️ Duplicate comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
89-114: Re-count remaining payouts after updates; avoid stale threshold, align logs, and remove console.log.Current check uses the pre-send count and a 100 threshold while each provider can take 100, leading to extra no-op invocations and confusing “100 vs 200” semantics. Prefer re-counting “processing” after updates to decide whether to schedule. Also use the shared log utility instead of console.log. Optional: add a small QStash delay if supported to smooth replica lag.
- if (invoice._count.payouts > 100) { - console.log( - "More than 100 payouts found for invoice, scheduling next batch...", - ); - const qstashResponse = await qstash.publishJSON({ - url: `${APP_DOMAIN_WITH_NGROK}/api/cron/payouts/charge-succeeded`, - body: { - invoiceId: invoiceId, - }, - }); - if (qstashResponse.messageId) { - console.log( - `Message sent to Qstash with id ${qstashResponse.messageId}`, - ); - } else { - // should never happen but just in case - await log({ - message: `Error sending message to Qstash to schedule next batch of payouts for invoice ${invoiceId}: ${JSON.stringify(qstashResponse)}`, - type: "errors", - }); - } - - return logAndRespond( - `Completed processing current batch of payouts for invoice ${invoiceId}. Next batch scheduled.`, - ); - } - - return logAndRespond( - `Completed processing all payouts for invoice ${invoiceId}.`, - ); + // Re-check remaining "processing" payouts after sender updates to avoid stale counts + const remaining = await prisma.payout.count({ + where: { invoiceId, status: "processing" }, + }); + if (remaining > 0) { + await log({ + message: `Remaining payouts=${remaining} for invoice ${invoiceId}; scheduling next batch…`, + type: "cron", + }); + const qstashResponse = await qstash.publishJSON({ + url: `${APP_DOMAIN_WITH_NGROK}/api/cron/payouts/charge-succeeded`, + body: { invoiceId }, + // delay: 5, // optional: if QStash supports delay, consider a short delay to absorb replica lag + }); + if (qstashResponse.messageId) { + await log({ + message: `QStash messageId=${qstashResponse.messageId} scheduled for next batch (invoice ${invoiceId}).`, + type: "cron", + }); + } else { + await log({ + message: `Error scheduling next batch for invoice ${invoiceId}: ${JSON.stringify(qstashResponse)}`, + type: "errors", + }); + } + return logAndRespond( + `Completed processing current batch of payouts for invoice ${invoiceId}. Next batch scheduled.`, + ); + } + return logAndRespond( + `Completed processing all payouts for invoice ${invoiceId}.`, + );Also applies to: 116-118
🧹 Nitpick comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
78-87: Capture and log failures from Promise.allSettled.You await settlement but drop rejection reasons. Log them to aid ops without failing the whole job.
- await Promise.allSettled([ - sendStripePayouts({ - invoiceId, - chargeId, - }), - - sendPaypalPayouts({ - invoiceId, - }), - ]); + const results = await Promise.allSettled([ + sendStripePayouts({ invoiceId, chargeId }), + sendPaypalPayouts({ invoiceId }), + ]); + const failures = results.filter( + (r): r is PromiseRejectedResult => r.status === "rejected", + ); + if (failures.length) { + await log({ + message: `Payout tasks failed: ${failures + .map((f) => String(f.reason)) + .join(" | ") + .slice(0, 1000)}`, + type: "errors", + }); + }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts(5 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
🔇 Additional comments (1)
apps/web/app/(ee)/api/cron/payouts/charge-succeeded/route.ts (1)
41-42: Verified: PayPal sender correctly updates status to "sent" to prevent duplicates.Confirmed that
updateMany()insend-paypal-payouts.ts(lines 50–55) advances status from "processing" to "sent" before returning. This ensures the batching mechanism in charge-succeeded route will not re-pick the same payouts in subsequent runs.
Summary by CodeRabbit
New Features
Bug Fixes
Other