Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix long retry delays #2020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 2, 2025
Merged

Fix long retry delays #2020

merged 5 commits into from
May 2, 2025

Conversation

nicktrn
Copy link
Collaborator

@nicktrn nicktrn commented May 2, 2025

v4 runs with delays >1m currently display very strange behaviour, with duplicate attempt spans and delays not being respected.

This was caused by two things, both fixed:

  • we didn't pass the retryWarmStartThresholdMs through, we also didn't set it anywhere
  • the run queue nackMessage method didn't respect retryAt

This PR also re-enables unit test requirements for our publish workflow after fixing the testcontainers issue.

Summary by CodeRabbit

  • New Features

    • Added a new environment variable to configure retry warm start thresholds for the run engine.
  • Bug Fixes

    • Improved message requeueing in the run queue to support scheduling retries at specific times.
  • Tests

    • Added a test to verify correct scheduling of message retries in the run queue.
  • Chores

    • Updated workflow dependencies to ensure unit tests pass before publishing.
    • Improved container stop logic to wait for full termination during test cleanup.

Copy link

changeset-bot bot commented May 2, 2025

⚠️ No Changeset found

Latest commit: 0d4064e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

coderabbitai bot commented May 2, 2025

Walkthrough

This update introduces a new environment variable, RUN_ENGINE_RETRY_WARM_START_THRESHOLD_MS, to the webapp's configuration schema and propagates its value through the RunEngine and RunAttemptSystem initialization. The RunQueue's nackMessage method is enhanced to accept an optional retryAt timestamp, allowing messages to be requeued at a specific time, with corresponding tests added to verify this behavior. Additionally, container stop logic in test utility functions is updated to wait up to 10 seconds for container termination. The GitHub Actions workflow is also modified to require unit tests to pass before publishing jobs can proceed.

Changes

File(s) Change Summary
.github/workflows/publish.yml Updated workflow dependencies: publish-webapp and publish-worker now depend on both typecheck and units jobs.
apps/webapp/app/env.server.ts Added RUN_ENGINE_RETRY_WARM_START_THRESHOLD_MS to environment schema with a default of 30,000 ms.
apps/webapp/app/v3/runEngine.server.ts Passed new retryWarmStartThresholdMs config property to RunEngine from environment.
internal-packages/run-engine/src/engine/index.ts Passed retryWarmStartThresholdMs option to RunAttemptSystem during RunEngine initialization.
internal-packages/run-engine/src/run-queue/index.ts Modified nackMessage to call private method with both message and optional retryAt timestamp.
internal-packages/run-engine/src/run-queue/tests/nack.test.ts Added test verifying nackMessage schedules message requeue with correct retryAt timestamp.
internal-packages/testcontainers/src/index.ts Updated container stop logic to use await container.stop({ timeout: 10 }) for proper wait on termination.

Sequence Diagram(s)

sequenceDiagram
    participant Env as Environment
    participant WebApp as WebApp
    participant RunEngine
    participant RunAttemptSystem

    Env->>WebApp: Provides RUN_ENGINE_RETRY_WARM_START_THRESHOLD_MS
    WebApp->>RunEngine: Passes retryWarmStartThresholdMs option
    RunEngine->>RunAttemptSystem: Initializes with retryWarmStartThresholdMs
Loading
sequenceDiagram
    participant Producer
    participant RunQueue

    Producer->>RunQueue: nackMessage({ message, retryAt })
    RunQueue->>RunQueue: #callNackMessage({ message, retryAt })
    Note right of RunQueue: Message requeued with scheduled retry time
Loading

Possibly related PRs

  • triggerdotdev/trigger.dev#1867: Reverses a similar workflow dependency change, directly modifying the same job prerequisites in the GitHub Actions workflow.

Poem

In the warren of code, a new path appears,
With retry times set, we hop past our fears.
Warm starts now measured, queues know when to act,
Containers wait gently, no more rushing the pack.
Tests keep us honest, workflows align—
A rabbit’s delight in a codebase so fine! 🐇✨


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a10e2a8 and 0d4064e.

📒 Files selected for processing (7)
  • .github/workflows/publish.yml (1 hunks)
  • apps/webapp/app/env.server.ts (2 hunks)
  • apps/webapp/app/v3/runEngine.server.ts (1 hunks)
  • internal-packages/run-engine/src/engine/index.ts (1 hunks)
  • internal-packages/run-engine/src/run-queue/index.ts (1 hunks)
  • internal-packages/run-engine/src/run-queue/tests/nack.test.ts (1 hunks)
  • internal-packages/testcontainers/src/index.ts (3 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
internal-packages/run-engine/src/run-queue/tests/nack.test.ts (3)
internal-packages/testcontainers/src/index.ts (1)
  • redisTest (136-136)
internal-packages/run-engine/src/run-queue/index.ts (1)
  • RunQueue (74-1390)
internal-packages/run-engine/src/run-queue/keyProducer.ts (1)
  • queueKey (51-77)
⏰ Context from checks skipped due to timeout of 90000ms (7)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
  • GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
  • GitHub Check: units / 🧪 Unit Tests
  • GitHub Check: typecheck / typecheck
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (10)
internal-packages/testcontainers/src/index.ts (3)

56-58: Good improvement to container management reliability

Adding a timeout to the container stop operation ensures that the function waits until the container has fully terminated before proceeding. This prevents potential test flakiness that could occur if subsequent tests try to reuse resources before they're fully released.


97-99: Consistent approach to container termination

The same timeout pattern is correctly applied to Redis containers, maintaining consistency across different container types and ensuring reliable cleanup during tests.


149-151: Clear documentation with implementation

The detailed comments explaining why the timeout is necessary are valuable. They help future developers understand the rationale behind these changes and prevent them from being accidentally removed.

apps/webapp/app/v3/runEngine.server.ts (1)

98-98: Correctly propagating retry threshold configuration

This change addresses one of the root causes mentioned in the PR objectives - ensuring that the retryWarmStartThresholdMs parameter is properly passed through the system. This value controls when retries should use warm starts, which is critical for preventing duplicate attempt spans.

apps/webapp/app/env.server.ts (1)

463-463: Good default value for retry warm start threshold

Setting a default of 30 seconds for the retry warm start threshold provides a reasonable balance. It's long enough to handle most typical processing delays while not being so long that it would significantly impact user experience for retriable operations.

internal-packages/run-engine/src/engine/index.ts (1)

307-307:

✅ Verification successful

Completing the configuration chain for warm start threshold

This change completes the propagation of the retryWarmStartThresholdMs configuration from environment variables through to the RunAttemptSystem where it will be used to determine whether to use warm starts for retries. This addresses one of the key issues mentioned in the PR objectives.

Consider verifying that the RunAttemptSystem constructor properly accepts and utilizes this parameter with:


🏁 Script executed:

#!/bin/bash
# Confirm that RunAttemptSystem accepts the retryWarmStartThresholdMs parameter
fd RunAttemptSystem.ts --type f | xargs cat | grep -A 20 "constructor"

Length of output: 70


🏁 Script executed:

#!/bin/bash
# Locate the TypeScript file defining RunAttemptSystem
file=$(rg --files-with-matches "class RunAttemptSystem" --glob "*.ts")
if [[ -z "$file" ]]; then
  echo "No file found containing RunAttemptSystem class"
  exit 1
fi

echo "Found in: $file"
echo
echo "=== class definition ==="
rg -n "class RunAttemptSystem" "$file"
echo
echo "=== constructor ==="
rg -n "constructor" "$file"

Length of output: 802


🏁 Script executed:

#!/bin/bash
# Locate and display the definition of RunAttemptSystemOptions
echo "=== RunAttemptSystemOptions Definition ==="
rg -n "interface RunAttemptSystemOptions" -A 20 --glob "*.ts"

echo
# Find all occurrences of retryWarmStartThresholdMs across the codebase
echo "=== Occurrences of retryWarmStartThresholdMs ==="
rg -n "retryWarmStartThresholdMs" --glob "*.ts"

Length of output: 1070


Propagated and Utilized: retryWarmStartThresholdMs Verified

– The RunAttemptSystemOptions interface (internal-packages/run-engine/src/engine/types.ts:43) declares retryWarmStartThresholdMs.
– The RunAttemptSystem constructor (internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts:54) accepts these options.
– The retry logic in RunAttemptSystem (lines 710–711) correctly checks this.options.retryWarmStartThresholdMs when deciding on a warm start.

No further changes needed; this parameter is fully wired through and used as intended.

.github/workflows/publish.yml (2)

59-59: Appropriate dependency update to ensure unit tests pass before publishing

Adding the units job as a dependency ensures that unit tests must pass before the webapp is published, improving the reliability of deployments.


66-66: Appropriate dependency update to ensure unit tests pass before publishing

Adding the units job as a dependency ensures that unit tests must pass before the worker is published, improving the reliability of deployments.

internal-packages/run-engine/src/run-queue/index.ts (1)

544-544: Fixed retry timing respect in nackMessage

This change correctly passes the retryAt parameter to the #callNackMessage method, ensuring that when a message is nacked with a specific retry time, that timing is respected when re-queueing the message.

This directly addresses one of the root causes mentioned in the PR objectives: "the run queue's nackMessage method did not honor the retryAt timing for retries".

internal-packages/run-engine/src/run-queue/tests/nack.test.ts (1)

218-277: Good test coverage for the retryAt functionality

This test appropriately verifies that when a message is nacked with a specific retryAt timestamp, the message is requeued with the correct scheduled retry time. The test is thorough in:

  1. Setting up the test environment with a Redis container
  2. Enqueuing and dequeuing a message
  3. Nacking it with a specific future timestamp
  4. Verifying the queue timestamp matches the expected retry time

This ensures the fix for the retryAt parameter works as expected.

✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

recurseml bot commented May 2, 2025

⚠️ Only 5 files will be analyzed due to processing limits.

1 similar comment
Copy link

recurseml bot commented May 2, 2025

⚠️ Only 5 files will be analyzed due to processing limits.

Copy link

recurseml bot commented May 2, 2025

😱 Found 2 issues. Time to roll up your sleeves! 😱

@nicktrn nicktrn merged commit 15816e9 into main May 2, 2025
12 of 13 checks passed
@nicktrn nicktrn deleted the fix/long-retry-delays branch May 2, 2025 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants