Codestin Search App

imnasnainaec · 2026-07-06T19:54:42Z

Fixes

Problem

maintenance/scripts/combine_restore.py copies backend files into the backend pod with a per-project kubectl cp loop (and copies the DB dump the same way). kubectl cp streams a tar over the exec channel and can stall intermittently. Because maint_utils.run_cmd() called subprocess.run(...) with no timeout, a stalled copy hung forever with no error and no output. Since this is The Combine's disaster-recovery restore, a silent infinite hang is a serious reliability problem.

Changes

maint_utils.py: add an optional timeout parameter to run_cmd(), passed through to subprocess.run(...). When set, a stalled command is killed and subprocess.TimeoutExpired is raised for the caller to handle. Default None preserves existing behavior.
combine_app.py: thread timeout through CombineApp.exec() and CombineApp.kubectl() (also adding check_results to kubectl()).
combine_restore.py: wrap each kubectl cp (the database dump and every backend-file item) in a new kubectl_cp() helper that:
- bounds each attempt with a timeout so a stalled stream is killed;
- retries a few times so a transient stall is recovered instead of aborting the whole restore;
- on repeated failure, logs the failing item (project) and exits non-zero so it surfaces as an error rather than a silent hang.
The timeout and retry count are configurable via the kubectl_cp_timeout (default 300s) and kubectl_cp_retries (default 3) environment variables.

I kept the existing per-item kubectl cp loop rather than switching to a single tar-pipe (listed as optional in the issue) to keep the change focused; the timeout/retry guard applies either way.

Testing

tox equivalents pass locally on the changed files: isort --check, black --check, flake8, and mypy --platform linux maintenance/scripts (strict) — all clean.
Exercised the new logic directly:
- run_cmd(..., timeout=1) on a 30s sleep raises TimeoutExpired (no longer hangs) and still returns output for fast commands.
- kubectl_cp() succeeds without retry on a clean copy (and passes check_results=False + timeout), recovers from transient stalls (timeout → timeout → success), and exits non-zero after exhausting retries on repeated failure.

🤖 Generated with Claude Code

Devin review: https://app.devin.ai/review/sillsdev/TheCombine/pull/4323

This change is

Summary by CodeRabbit

Bug Fixes
- Improved backup and restore reliability by retrying file copy operations that can stall or fail temporarily.
- Added timeouts to long-running copy and command operations so they fail more predictably instead of hanging indefinitely.
- Backup and restore flows now use clearer copy-step labeling for better troubleshooting.
Chores
- Added configurable environment-based settings for copy timeouts and retry attempts.

kubectl cp streams a tar over the exec channel and can stall intermittently. run_cmd() called subprocess.run() with no timeout, so a stalled copy during a restore hung forever with no output or error (see issue #4322), undermining the disaster-recovery guarantee. - Add an optional timeout to run_cmd() and thread it (plus check_results) through CombineApp.kubectl() and CombineApp.exec(). - Wrap each restore kubectl cp (the database dump and every backend-file item) in kubectl_cp(): bound each attempt with a timeout, retry a few times so a transient stall is recovered, and on repeated failure log the failing item and exit non-zero instead of hanging silently. - Make the timeout and retry count configurable via the kubectl_cp_timeout and kubectl_cp_retries environment variables. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

coderabbitai · 2026-07-06T19:54:50Z

Warning

Review limit reached

@imnasnainaec, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 42 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 47f3efd7-50b8-4471-9fc3-0e9d01d03207

📥 Commits

Reviewing files that changed from the base of the PR and between adae68b and d4d3916.

📒 Files selected for processing (1)

maintenance/scripts/combine_app.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix-4322-cp-timeout

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-07-06T19:58:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.88%. Comparing base (e2b613d) to head (d4d3916).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4323      +/-   ##
==========================================
- Coverage   75.96%   66.88%   -9.08%     
==========================================
  Files         305      248      -57     
  Lines       11384     6303    -5081     
  Branches     1411      794     -617     
==========================================
- Hits         8648     4216    -4432     
+ Misses       2332     1826     -506     
+ Partials      404      261     -143

Flag	Coverage Δ
backend	`?`
frontend	`66.88% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The backup script's `kubectl cp` calls (database and backend files) can stall on the exec channel exactly like the restore ones. combine_backup.py runs unattended as a cron job, so a silent hang there quietly stops backups from being produced. Lift the restore-only kubectl_cp() into CombineApp.cp_with_retry() so both scripts share one timeout/retry implementation, and wrap combine_backup.py's two copies with it. The kubectl_cp_timeout and kubectl_cp_retries environment variables now configure both backup and restore. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The loop runs the copy 'retries' total times, so the conventional reading of retries (extra tries after the first) was misleading. Rename the parameter, constant, and env var (kubectl_cp_attempts) before the knob becomes part of the ops interface. Co-Authored-By: Claude Fable 5 <[email protected]>

coderabbitai

🧹 Nitpick comments (1)

maintenance/scripts/combine_app.py (1)
100-140: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Optional: add a short backoff between retries.

Attempts currently retry immediately after a non-zero return code. A brief sleep between attempts would avoid hammering a transient failure that isn't a timeout. Timeout attempts already have inherent delay, so this mainly helps the fast-fail path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@maintenance/scripts/combine_app.py` around lines 100 - 140, The retry loop in
cp_with_retry currently retries immediately after a non-zero kubectl cp failure,
which can rapidly hammer a transient fast-fail path. Add a short sleep between
failed attempts in cp_with_retry (for the returncode != 0 case, and optionally
after timeouts if desired) before the next iteration, using the existing
attempt/attempts logging to keep behavior clear. Keep the change localized to
cp_with_retry and preserve the final abort behavior after all attempts are
exhausted.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@maintenance/scripts/combine_app.py`:
- Around line 100-140: The retry loop in cp_with_retry currently retries
immediately after a non-zero kubectl cp failure, which can rapidly hammer a
transient fast-fail path. Add a short sleep between failed attempts in
cp_with_retry (for the returncode != 0 case, and optionally after timeouts if
desired) before the next iteration, using the existing attempt/attempts logging
to keep behavior clear. Keep the change localized to cp_with_retry and preserve
the final abort behavior after all attempts are exhausted.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 729281c5-f5f1-4e34-8f56-0b6bcceee7db

📥 Commits

Reviewing files that changed from the base of the PR and between e2b613d and adae68b.

📒 Files selected for processing (4)

maintenance/scripts/combine_app.py
maintenance/scripts/combine_backup.py
maintenance/scripts/combine_restore.py
maintenance/scripts/maint_utils.py

Without a delay, a fast-failing kubectl cp exhausts all attempts within the same second, giving a transient failure no time to clear. Co-Authored-By: Claude Fable 5 <[email protected]>

A misconfigured kubectl_cp_attempts of 0 or negative made range(1, attempts+1) empty, so cp_with_retry aborted without ever attempting the copy. Clamp both env-driven knobs so a bad value cannot skip the copy or set a nonsensical timeout. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

jasonleenaylor

@jasonleenaylor reviewed 4 files and all commit messages, and made 1 comment.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on imnasnainaec).

github-actions Bot added deployment python labels Jul 6, 2026

imnasnainaec and others added 2 commits July 6, 2026 17:37

coderabbitai Bot reviewed Jul 7, 2026

View reviewed changes

imnasnainaec and others added 3 commits July 7, 2026 17:35

Pause between failed cp attempts

2062d90

Without a delay, a fast-failing kubectl cp exhausts all attempts within the same second, giving a transient failure no time to clear. Co-Authored-By: Claude Fable 5 <[email protected]>

Tweak

25a7b50

imnasnainaec marked this pull request as ready for review July 8, 2026 18:09

Add _S to timeout const name

d4d3916

jasonleenaylor approved these changes Jul 8, 2026

View reviewed changes

imnasnainaec enabled auto-merge (squash) July 8, 2026 18:28

imnasnainaec merged commit 3e78f92 into master Jul 8, 2026
15 checks passed

imnasnainaec deleted the fix-4322-cp-timeout branch July 8, 2026 18:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add timeout and retry to combine_restore kubectl cp#4323

Add timeout and retry to combine_restore kubectl cp#4323
imnasnainaec merged 7 commits into
masterfrom
fix-4322-cp-timeout

imnasnainaec commented Jul 6, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading

Review limit reached

Uh oh!

codecov Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

jasonleenaylor left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

imnasnainaec commented Jul 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes

Problem

Changes

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

codecov Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

jasonleenaylor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

imnasnainaec commented Jul 6, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading

codecov Bot commented Jul 6, 2026 •

edited

Loading