Codestin Search App

cursor · 2026-08-06T11:41:29Z

Summary

handle_end_date_change_for_challenge called delete_workers whenever end_date was moved into the past, with no check for pending submissions. That bypassed the pending-aware drain policy from the cleanup Lambda (#5179): a host/admin shortening end_date (challenge.yml / admin) could tear down the ECS stack while submissions were still queued/running/resuming, killing in-flight eval and leaving queued work with no consumers.

Trigger scenario

Fargate challenge has workers up and submissions still pending.
Host/admin sets end_date to a time in the past (config update or admin edit).
Old signal path unconditionally called delete_workers.
ECS service deleted mid-drain → running evaluations interrupted; remaining queue has no workers until a new submission triggers ensure_workers_for_submission.

Changes

Before deleting on a past end_date, check the same pending statuses as the cleanup Lambda / pending_submission_count API.
If pending work exists, skip delete and schedule a delayed cleanup re-check (now + 60m). EventBridge cannot be pointed at the already-past end_date.
Refactor create/update schedule helpers to share _put_challenge_cleanup_schedule so retry scheduling can pass an explicit run_at.
Regression tests for the pending skip path and retry schedule timing.

Test plan

pytest focused suites: TestHandleEndDateChange, TestScheduleChallengeCleanupRetry, TestUpdateChallengeCleanupSchedule, TestScheduleChallengeCleanup, TestDeleteChallengeCleanupSchedule — 21 passed
black --check --line-length=79 / isort --check --profile=black --line-length=79 on changed files

Summary by CodeRabbit

Bug Fixes
- Improved challenge cleanup scheduling when schedules are missing or need updating.
- Prevented worker cleanup while submissions are still pending.
- Added automatic retry scheduling so cleanup can complete after pending submissions are resolved.
- Ensured retries use a future execution time, even when a challenge has already ended.
Reliability
- Centralized scheduling validation and recovery for more consistent cleanup behavior.

handle_end_date_change previously called delete_workers whenever end_date was set to the past, even with queued/running submissions. That bypassed the pending-aware drain policy from the cleanup Lambda (#5179) and could kill in-flight evaluations after a host/admin shortened end_date. Skip delete when pending submissions exist and schedule a delayed cleanup re-check instead (cannot point EventBridge at the already-past end_date). Co-authored-by: Rishabh Jain <[email protected]>

Co-authored-by: Rishabh Jain <[email protected]>

codecov · 2026-08-06T11:52:39Z

Codecov Report

❌ Patch coverage is 96.55172% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.30%. Comparing base (b269a39) to head (db5839c).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
apps/challenges/aws_utils.py	96.15%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5189      +/-   ##
==========================================
- Coverage   91.30%   91.30%   -0.01%     
==========================================
  Files         115      115              
  Lines        8996     9001       +5     
==========================================
+ Hits         8214     8218       +4     
- Misses        782      783       +1

Flag	Coverage Δ
backend	`93.83% <96.55%> (-0.02%)`	⬇️
frontend	`87.54% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
Accounts & Authentication	`97.40% <ø> (ø)`
Challenges Management	`96.16% <96.55%> (-0.03%)`	⬇️
Job Processing	`90.19% <ø> (ø)`
Participants & Teams	`99.54% <ø> (ø)`
Challenge Hosts	`100.00% <ø> (ø)`
Analytics	`100.00% <ø> (ø)`
Web Interface	`100.00% <ø> (ø)`
Frontend (Gulp)	`87.54% <ø> (ø)`
All Models	`97.38% <100.00%> (+<0.01%)`	⬆️
All Views	`100.00% <ø> (ø)`
All Serializers	`98.20% <ø> (ø)`
Utility Functions	`97.25% <96.15%> (-0.05%)`	⬇️
Core Configuration	`78.94% <ø> (ø)`

Files with missing lines	Coverage Δ
apps/challenges/models.py	`96.14% <100.00%> (+0.01%)`	⬆️
apps/challenges/aws_utils.py	`98.67% <96.15%> (-0.10%)`	⬇️

Files with missing lines	Coverage Δ
apps/challenges/models.py	`96.14% <100.00%> (+0.01%)`	⬆️
apps/challenges/aws_utils.py	`98.67% <96.15%> (-0.10%)`	⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b269a39...db5839c. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

RishabhJain2018 · 2026-08-07T06:17:34Z

@coderabbitai Please review this!

coderabbitai · 2026-08-07T06:17:47Z

@RishabhJain2018 I will review pull request #5189, with focus on the pending-submission drain behavior and cleanup retry scheduling.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai · 2026-08-07T06:17:56Z

Walkthrough

EventBridge cleanup scheduling now uses shared create/update logic and centralized naming. Expired challenges check for pending submissions. Cleanup retries while submissions remain pending and deletes workers after submissions finish.

Changes

Challenge cleanup lifecycle

Layer / File(s)	Summary
Shared cleanup schedule management `apps/challenges/aws_utils.py`, `tests/unit/challenges/test_aws_utils.py`	EventBridge schedules now share validation, naming, target construction, creation, update, deletion, and missing-schedule recreation. Retry schedules use a future time based on the current time.
Pending-submission cleanup handling `apps/challenges/models.py`, `tests/unit/challenges/test_aws_utils.py`	Cleanup detects running, submitted, queued, and resuming submissions. It schedules a retry while submissions remain pending and deletes workers when no pending submissions exist. Tests cover both paths.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ChallengeModel
  participant AWSUtils
  participant EventBridge
  participant Workers
  ChallengeModel->>AWSUtils: Check pending submissions
  alt Submissions remain pending
    AWSUtils->>EventBridge: Create one-hour cleanup retry
  else No submissions remain pending
    ChallengeModel->>Workers: Delete challenge workers
  end

Possibly related PRs

Cloud-CV/EvalAI#5061: Both modify handle_end_date_change_for_challenge with different cleanup behavior.
Cloud-CV/EvalAI#5179: Both add pending-submission checks and EventBridge cleanup retries.
Cloud-CV/EvalAI#5183: Both handle pending submissions for expired challenges.

Suggested reviewers: rishabhjain2018

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: preserving workers when a past end date coincides with pending submissions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch cursor/critical-bug-management-571e

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/unit/challenges/test_aws_utils.py (1)
5707-5767: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add direct coverage for the pending-submission predicate.

These tests patch challenge_has_pending_submissions. They only test the model-handler branches. They do not test the new query in apps/challenges/aws_utils.py at Line 817. Add focused tests that verify each pending status returns True and a terminal status returns False. Otherwise, a status-list regression can delete active workers or defer cleanup after work completes.

As per path instructions, “Test code. Check that tests actually assert behavior, cover edge cases, and are not silently passing.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/challenges/test_aws_utils.py` around lines 5707 - 5767, Add
focused unit tests for challenge_has_pending_submissions, covering every pending
submission status as True and at least one terminal status as False. Exercise
the predicate’s actual query behavior without mocking it, and assert the
returned boolean so status-list regressions are detected independently of
handle_end_date_change_for_challenge.
Sources: Path instructions, Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@tests/unit/challenges/test_aws_utils.py`:
- Around line 5707-5767: Add focused unit tests for
challenge_has_pending_submissions, covering every pending submission status as
True and at least one terminal status as False. Exercise the predicate’s actual
query behavior without mocking it, and assert the returned boolean so
status-list regressions are detected independently of
handle_end_date_change_for_challenge.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 27d2f8e7-cd39-41dc-88ee-0580b8215004

📥 Commits

Reviewing files that changed from the base of the PR and between b269a39 and db5839c.

📒 Files selected for processing (3)

apps/challenges/aws_utils.py
apps/challenges/models.py
tests/unit/challenges/test_aws_utils.py

cursoragent and others added 2 commits August 6, 2026 11:13

style: black-format challenge cleanup schedule helpers

db5839c

Co-authored-by: Rishabh Jain <[email protected]>

coderabbitai Bot reviewed Aug 7, 2026

View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

fix: keep workers when end_date is moved past with pending submissions - #5189

fix: keep workers when end_date is moved past with pending submissions#5189
cursor[bot] wants to merge 2 commits into
masterfrom
cursor/critical-bug-management-571e

cursor Bot commented Aug 6, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

codecov Bot commented Aug 6, 2026 •

edited

Loading

Uh oh!

RishabhJain2018 commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

cursor Bot commented Aug 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Trigger scenario

Changes

Test plan

Summary by CodeRabbit

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

RishabhJain2018 commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cursor Bot commented Aug 6, 2026 •

edited by coderabbitai Bot

Loading

codecov Bot commented Aug 6, 2026 •

edited

Loading

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading

coderabbitai Bot commented Aug 7, 2026 •

edited

Loading