-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Improve bounty notification batching mechanism #3091
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Rate limit exceeded@steven-tey has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 4 minutes and 10 seconds before requesting another review. β How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. π¦ How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. π Files selected for processing (2)
WalkthroughThe bounty partner notification cron moved from page-based pagination to cursor-driven batch processing using Changes
Sequence Diagram(s)sequenceDiagram
participant Cron as Cron Trigger
participant API as notify-partners Route
participant DB as Database
participant Mail as Mailer/Resend
participant QStash as QStash (enqueue)
Cron->>API: POST { bountyId, startingAfter?, batchNumber? }
API->>DB: Query bounty (include program.emailDomains) and enrollments
DB-->>API: Return up to EMAIL_BATCH_SIZE enrollments (ordered by id)
rect rgb(220,245,220)
Note over API,Mail: For each enrollment: build email (From set by verified domain) and send via Mailer
API->>Mail: sendEmail(batch payload)
Mail-->>API: send result
end
API->>API: if results.length == EMAIL_BATCH_SIZE\n compute nextStartingAfter = lastEnrollment.id\n compute delay = (batchNumber % EXTENDED_DELAY_INTERVAL == 0) ? EXTENDED_DELAY_SECONDS : BATCH_DELAY_SECONDS
alt More batches needed
API->>QStash: Enqueue next batch { bountyId, startingAfter: nextStartingAfter, batchNumber+1 } with delay and idempotencyKey
QStash-->>API: ack
else Final/short batch
Note over API: Do not enqueue further batches
end
API-->>Cron: 200 OK (batch queued / processed)
Estimated code review effortπ― 3 (Moderate) | β±οΈ ~25 minutes
Possibly related PRs
Poem
Pre-merge checks and finishing touchesβ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai full review |
β Actions performedFull review triggered. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
π§Ή Nitpick comments (2)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts (2)
26-29: Consider making batch configuration environment-based.The hardcoded constants are reasonable defaults, but making them configurable via environment variables would provide flexibility for production tuning without code changes.
Example:
const EMAIL_BATCH_SIZE = parseInt(process.env.BOUNTY_BATCH_SIZE || "100", 10); const BATCH_DELAY_SECONDS = parseInt(process.env.BOUNTY_BATCH_DELAY || "2", 10); const EXTENDED_DELAY_SECONDS = parseInt(process.env.BOUNTY_EXTENDED_DELAY || "30", 10); const EXTENDED_DELAY_INTERVAL = parseInt(process.env.BOUNTY_EXTENDED_INTERVAL || "25", 10);
174-179: Remove redundant check and add explanatory comment.The
batchNumber > 0check is redundant sincebatchNumberdefaults to 1 and is always incremented. Additionally, the throttling strategy would benefit from a brief comment.Apply this diff:
- // Add BATCH_DELAY_SECONDS pause between each batch, and a longer EXTENDED_DELAY_SECONDS cooldown after every EXTENDED_DELAY_INTERVAL batches. let delay = 0; - if (batchNumber > 0 && batchNumber % EXTENDED_DELAY_INTERVAL === 0) { + // Apply extended delay every 25 batches (at batch 25, 50, 75, etc.) to prevent rate limiting + if (batchNumber % EXTENDED_DELAY_INTERVAL === 0) { delay = EXTENDED_DELAY_SECONDS; } else { delay = BATCH_DELAY_SECONDS; }
π Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
π Files selected for processing (1)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts(5 hunks)
β° Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: api-tests
π Additional comments (3)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts (3)
16-24: LGTM! Well-structured schema for cursor-based batching.The addition of
startingAfterfor cursor pagination andbatchNumberfor batch tracking is clean and appropriate. The inline description onbatchNumberis helpful documentation.
152-152: LGTM! Idempotency key correctly reflects cursor-based batching.Using
startingAfter || "initial"ensures each batch has a unique idempotency key, preventing duplicate email sends if the job is retried.
181-190: QStash delay parameter is correctly specified in seconds.The web search confirms that QStash's
publishJSONdelay parameter expects seconds. The code correctly passes delay values asBATCH_DELAY_SECONDS=2andEXTENDED_DELAY_SECONDS=30, which aligns with the expected unit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Caution
Some comments are outside the diff and canβt be posted inline due to platform limitations.
β οΈ Outside diff range comments (1)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts (1)
168-180: Add error handling for email sending failures.When
sendBatchEmailfails and returnsnulldata, the code silently continues without logging the failure or creating notification records. This could result in partners not being notified without any indication of failure in the logs or database.Add explicit error handling:
if (data) { await prisma.notificationEmail.createMany({ data: programEnrollments.map(({ partner }, idx) => ({ id: createId({ prefix: "em_" }), type: NotificationEmailType.Bounty, emailId: data.data[idx].id, bountyId: bounty.id, programId: bounty.programId, partnerId: partner.id, recipientUserId: partner.users[0].userId, })), }); + } else { + await log({ + message: `Failed to send batch emails for bounty ${bountyId}, batch ${batchNumber}. Skipping notification records.`, + type: "errors", + }); }
β»οΈ Duplicate comments (1)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts (1)
113-123: Critical: ChangeorderByfromidtocreatedAtfor reliable pagination.This issue was flagged in a previous review and remains unfixed. The
idfield uses CUID, which is cryptographically random and non-sequential. Ordering byidproduces unpredictable, non-chronological results that make cursor-based pagination unreliable. Records may be skipped or processed multiple times across batches.When changing to
createdAt, you'll also need to update:
- The
orderByclause (line 121)- The cursor field (line 117)
- The
startingAfterassignment (line 183)Apply this diff:
- take: EMAIL_BATCH_SIZE, - skip: startingAfter ? 1 : 0, - ...(startingAfter && { - cursor: { - id: startingAfter, - }, - }), orderBy: { - id: "asc", + createdAt: "asc", }, + take: EMAIL_BATCH_SIZE, + skip: startingAfter ? 1 : 0, + ...(startingAfter && { + cursor: { + createdAt: new Date(startingAfter), + }, + }),And update line 183:
- startingAfter = programEnrollments[programEnrollments.length - 1].id; + startingAfter = programEnrollments[programEnrollments.length - 1].createdAt.toISOString();
π§Ή Nitpick comments (1)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts (1)
182-207: Batch continuation logic is mostly correct.The delay calculation and batch enqueueing logic works properly. Extended delays are applied every 25 batches as intended.
For clarity, consider using a new variable name instead of reassigning the parameter:
if (programEnrollments.length === EMAIL_BATCH_SIZE) { - startingAfter = programEnrollments[programEnrollments.length - 1].id; + const nextCursor = programEnrollments[programEnrollments.length - 1].id; // Add BATCH_DELAY_SECONDS pause between each batch, and a longer EXTENDED_DELAY_SECONDS cooldown after every EXTENDED_DELAY_INTERVAL batches. let delay = 0; if (batchNumber > 0 && batchNumber % EXTENDED_DELAY_INTERVAL === 0) { delay = EXTENDED_DELAY_SECONDS; } else { delay = BATCH_DELAY_SECONDS; } await qstash.publishJSON({ url: `${APP_DOMAIN_WITH_NGROK}/api/cron/bounties/notify-partners`, method: "POST", delay, body: { bountyId, - startingAfter, + startingAfter: nextCursor, batchNumber: batchNumber + 1, }, }); return logAndRespond( - `Enqueued next batch (${startingAfter}) for bounty ${bountyId} to run after ${delay} seconds.`, + `Enqueued next batch (${nextCursor}) for bounty ${bountyId} to run after ${delay} seconds.`, ); }
π Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
π Files selected for processing (2)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts(7 hunks)apps/web/tests/partners/resource.ts(1 hunks)
π§° Additional context used
π§ Learnings (3)
π Learning: 2025-08-16T11:14:00.667Z
Learnt from: devkiran
Repo: dubinc/dub PR: 2754
File: apps/web/lib/partnerstack/schemas.ts:47-52
Timestamp: 2025-08-16T11:14:00.667Z
Learning: The PartnerStack API always includes the `group` field in partner responses, so the schema should use `.nullable()` rather than `.nullish()` since the field is never omitted/undefined.
Applied to files:
apps/web/tests/partners/resource.ts
π Learning: 2025-09-17T17:44:03.965Z
Learnt from: TWilson023
Repo: dubinc/dub PR: 2857
File: apps/web/lib/actions/partners/update-program.ts:96-0
Timestamp: 2025-09-17T17:44:03.965Z
Learning: In apps/web/lib/actions/partners/update-program.ts, the team prefers to keep the messagingEnabledAt update logic simple by allowing client-provided timestamps rather than implementing server-controlled timestamp logic to avoid added complexity.
Applied to files:
apps/web/tests/partners/resource.ts
π Learning: 2025-08-26T14:20:23.943Z
Learnt from: TWilson023
Repo: dubinc/dub PR: 2736
File: apps/web/app/api/workspaces/[idOrSlug]/notification-preferences/route.ts:13-14
Timestamp: 2025-08-26T14:20:23.943Z
Learning: The updateNotificationPreference action in apps/web/lib/actions/update-notification-preference.ts already handles all notification preference types dynamically, including newBountySubmitted, through its schema validation using the notificationTypes enum and Prisma's dynamic field update pattern.
Applied to files:
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts
𧬠Code graph analysis (1)
apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts (1)
packages/email/src/index.ts (1)
sendBatchEmail(31-70)
β° Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: build
π Additional comments (5)
apps/web/tests/partners/resource.ts (1)
32-32: LGTM: Consistent date field addition.The
trustedAtfield follows the same pattern as other date fields in the schema, using.nullish()to allow bothnullandundefinedvalues.apps/web/app/(ee)/api/cron/bounties/notify-partners/route.ts (4)
16-24: LGTM: Schema appropriately extended for cursor-based pagination.The addition of
startingAfterandbatchNumberfields properly supports the batch processing mechanism with clear typing and documentation.
26-29: LGTM: Well-chosen batch processing constants.The batch size and delay configuration provides good balance between throughput and rate limiting, with extended cooldowns to prevent service exhaustion.
53-58: LGTM: Appropriate query extension for email domain support.The nested include for
program.emailDomainsenables custom sender addresses from verified domains, which enhances email deliverability.
135-166: LGTM: Email domain verification and batch sending implementation.The verified email domain lookup and conditional
fromaddress configuration enhances email deliverability. The idempotency key properly includesstartingAfterto ensure each batch is uniquely identified.
Summary by CodeRabbit
New Features
Refactor
Bug Fixes
Tests