Fix: keep workers running after end_date while submissions are pending - #5185
Open
cursor[bot] wants to merge 3 commits into
Open
Fix: keep workers running after end_date while submissions are pending#5185cursor[bot] wants to merge 3 commits into
cursor[bot] wants to merge 3 commits into
Conversation
…d_date The auto_scale_* cron scripts treated end_date < now as an unconditional scale-down signal. That undid #5179/#5183: the cleanup Lambda keeps the worker stack up past end_date while queued/running/resuming submissions drain, but the next cron tick stopped those workers and left pending work without consumers. Only scale down when pending_submission_count is 0; if the challenge has ended but still has pending work, leave capacity running to drain. Co-authored-by: Rishabh Jain <[email protected]>
Co-authored-by: Rishabh Jain <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #5185 +/- ##
=======================================
Coverage 91.36% 91.36%
=======================================
Files 115 115
Lines 9054 9054
=======================================
Hits 8272 8272
Misses 782 782
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
2 tasks
RishabhJain2018
marked this pull request as ready for review
August 10, 2026 20:20
Member
|
@coderabbitai Please review this. |
|
✅ Action performedReview finished.
|
WalkthroughAutoscaling now keeps EC2 workers, EKS nodes, and shared workers running when ended challenges have pending submissions. Scale-down remains enabled when no submissions are pending. Unit tests cover ended and active challenge states. ChangesAutoscaling drain behavior
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/monitoring/auto_scale_workers.py`:
- Around line 160-168: Ensure ended challenges with pending submissions restart
capacity when it is zero: in scripts/monitoring/auto_scale_workers.py lines
160-168, call scale_up_workers when pending_submissions > 0 and num_workers ==
0; in scripts/monitoring/auto_scale_ec2_workers.py lines 113-121, call
start_instance when pending submissions exist and the instance is stopped; in
scripts/monitoring/auto_scale_eks_nodes.py lines 224-234, scale up when pending
submissions exist and original_desired_size == 0. In
tests/unit/monitoring/test_auto_scale_workers.py lines 58-76, add an
ended-challenge case with pending submissions and workers=0 asserting
scale_up_workers is called.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c9cd4a47-c92f-4607-a17e-24c1ab60667d
📒 Files selected for processing (4)
scripts/monitoring/auto_scale_ec2_workers.pyscripts/monitoring/auto_scale_eks_nodes.pyscripts/monitoring/auto_scale_workers.pytests/unit/monitoring/test_auto_scale_workers.py
Comment on lines
+160
to
+168
| if pending_submissions == 0: | ||
| scale_down_workers(challenge, num_workers) | ||
| elif challenge_ended: | ||
| print( | ||
| "Challenge ID: {}, Title: {} has ended but still has {} " | ||
| "pending submission(s); leaving workers running to drain.".format( | ||
| challenge["id"], challenge["title"], pending_submissions | ||
| ) | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Start capacity when an ended challenge has pending submissions and zero capacity.
If capacity is already zero, each ended-challenge branch only logs and performs no start or scale-up action. Pending submissions then have no consumer and cannot drain. Ensure each script starts capacity when pending submissions exist and capacity is stopped or zero.
scripts/monitoring/auto_scale_workers.py#L160-L168: callscale_up_workerswhenpending_submissions > 0andnum_workers == 0, including afterend_date.scripts/monitoring/auto_scale_ec2_workers.py#L113-L121: callstart_instancewhen pending submissions exist and the instance is stopped, including afterend_date.scripts/monitoring/auto_scale_eks_nodes.py#L224-L234: scale up when pending submissions exist andoriginal_desired_size == 0, including afterend_date.tests/unit/monitoring/test_auto_scale_workers.py#L58-L76: add an ended-challenge case with pending submissions andworkers=0that assertsscale_up_workersis called.
As per path instructions, **/test_*.py must “cover edge cases.”
📍 Affects 4 files
scripts/monitoring/auto_scale_workers.py#L160-L168(this comment)scripts/monitoring/auto_scale_ec2_workers.py#L113-L121scripts/monitoring/auto_scale_eks_nodes.py#L224-L234tests/unit/monitoring/test_auto_scale_workers.py#L58-L76
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/monitoring/auto_scale_workers.py` around lines 160 - 168, Ensure
ended challenges with pending submissions restart capacity when it is zero: in
scripts/monitoring/auto_scale_workers.py lines 160-168, call scale_up_workers
when pending_submissions > 0 and num_workers == 0; in
scripts/monitoring/auto_scale_ec2_workers.py lines 113-121, call start_instance
when pending submissions exist and the instance is stopped; in
scripts/monitoring/auto_scale_eks_nodes.py lines 224-234, scale up when pending
submissions exist and original_desired_size == 0. In
tests/unit/monitoring/test_auto_scale_workers.py lines 58-76, add an
ended-challenge case with pending submissions and workers=0 asserting
scale_up_workers is called.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/monitoring/auto_scale_workers.py(and the EC2/EKS siblings) treatedend_date < nowas an unconditional scale-down signal viapending_submissions == 0 or end_date < now. That undid the pending-aware cleanup work in #5179/#5183: the challenge-cleanup Lambda keeps the worker stack up pastend_dateso queued/running/resuming submissions can drain, but the next monitoring cron tick stopped those workers and left pending work without consumers (ensure_workers_for_submissiononly runs on new submits).Trigger scenario
end_datewith submissions stillqueued/running/resuming.auto_scale_workers.py.end_date < now, even thoughpending_submissions > 0.Changes
pending_submissions == 0.auto_scale_ec2_workers.pyandauto_scale_eks_nodes.py.Test plan
pytest tests/unit/monitoring/test_auto_scale_workers.py— 4 passedpytest tests/unit/monitoring/— 32 passedblack --check/isort --checkon changed filesNote
Medium Risk
Changes production cron behavior for worker lifecycle across ECS, EC2, and EKS; incorrect logic could strand submissions or leave capacity running longer, but the change is narrowly scoped to the scale decision and covered by new unit tests.
Overview
Monitoring auto-scale crons no longer tear down capacity just because a challenge passed
end_date. They now scale down only whenpending_submissions == 0(queued/running/submitted/resuming). If the challenge has ended but work is still pending, the scripts log and skip stop/scale-down so evaluations can drain—aligning with the cleanup Lambda’s pending-aware behavior (#5179).The same decision split is applied in
auto_scale_workers.py(ECS/Fargate),auto_scale_ec2_workers.py, andauto_scale_eks_nodes.py. Previously,pending == 0 or end_date < nowstopped workers/nodegroups/instances even with pending work, killing in-flight evals and leaving the queue without consumers.Adds
tests/unit/monitoring/test_auto_scale_workers.pywith four regression tests for ended-vs-active challenges and pending vs idle metrics.Reviewed by Cursor Bugbot for commit 0518656. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
Bug Fixes
Tests