Fix controller waitpoint resolution, suspendable state, and snapshot race conditions #2006
Conversation
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (10)
apps/supervisor/src/workloadServer/index.ts (1)
345-369: Implementation follows consistent pattern with other endpointsThe new endpoint to fetch snapshots since a specific snapshot ID is well-structured and follows the established patterns in the codebase. It properly validates parameters, handles error cases, and returns appropriately typed responses.
One small suggestion to improve error logging:
if (!sinceSnapshotResponse.success) { console.error("Failed to get snapshots since", { runId: params.runFriendlyId, + snapshotId: params.snapshotFriendlyId, error: sinceSnapshotResponse.error, }); reply.empty(500); return; }apps/webapp/app/routes/engine.v1.worker-actions.runs.$runFriendlyId.snapshots.since.$snapshotId.ts (1)
1-30: Well-structured API route for snapshot retrievalThis new route handler for retrieving snapshots is cleanly implemented using the
createLoaderWorkerApiRoutehelper. It correctly validates parameters and uses the authenticated worker API client to fetch snapshots since a specified ID.The error handling is functional but could be more specific about the type of failure.
Consider enhancing the error message to be more specific:
if (!snapshots) { - throw new Error("Failed to retrieve snapshots since given snapshot"); + throw new Error(`Failed to retrieve snapshots since snapshot ${snapshotId} for run ${runFriendlyId}`); }internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts (1)
45-60: Unnecessaryindex === -1check + O(n²) scan – simplify and speed-up.
indexesis never populated with-1, so the ternary adds no value and slightly obscures intent.
Additionally, scanningcompletedWaitpointOrderfor every wait-point yields O(N·M) behaviour (N = waitpoints, M = order length). A singleMap<id,index>built once would turn this into O(N).- index: index === -1 ? undefined : index, + // `index` is already either a valid number or `undefined` + index,Consider pre-building a lookup:
const orderIndex = new Map<string, number>(); snapshot.completedWaitpointOrder.forEach((id, i) => orderIndex.set(id, i)); … index: orderIndex.get(w.id),packages/cli-v3/src/entryPoints/managed/controller.ts (2)
34-36:notificationCount&lastNotificationAtare never updatedThe counters are declared and surfaced via
metrics, but no code now mutates them after the removal of"run:notify"listeners.
Consider deleting these members or moving the increment logic intoRunNotifierto avoid stale or misleading metrics.
465-472: Environment diff logging shows identical objects
currentEnvandnewEnvare constructed from the same runtime values, making the diff in the debug log uninformative.
If the intention is to capture override effects, buildnewEnvafter callingprocessEnvOverridesand from the mutated source.packages/cli-v3/src/executions/taskRunProcess.ts (2)
101-108:unsafeDetachEvtHandlers()relies on caller disciplineDetaching every handler is powerful but dangerous: forgetting to call it when disposing the instance will leak listeners and promises.
Consider invoking it insidekill()/cleanup()once the child has exited, or wrapping the Evt instances in a disposable object to enforce cleanup.
404-408: Error path inkill()swallows important diagnostics
console.debuging the caught error is fine for dev, but production troubleshooting benefits from propagating or logging at warn/error level so it surfaces in monitoring.- logger.debug("kill: failed to kill child process", { error }); + logger.warn?.("kill: failed to kill child process", { error });packages/cli-v3/src/entryPoints/managed/snapshot.ts (1)
338-359:hasBeenRestoredperforms a remote call on every snapshot – cache result
hasBeenRestored()reaches out to theMetadataClientfor every deprecated snapshot detection, which (a) adds latency inside the queue loop, and (b) hammers the metadata service under high-frequency snapshot traffic.Since a run can only be “restored” once per execution, you can safely cache the first non-null
TRIGGER_RUNNER_IDcomparison and short-circuit subsequent calls.+private _restored?: boolean; ... if (this._restored !== undefined) { return this._restored; } const [error, overrides] = await this.metadataClient.getEnvOverrides(); ... this._restored = restored; return restored;This keeps the logic intact while eliminating N redundant network round-trips.
packages/cli-v3/src/entryPoints/managed/execution.ts (2)
683-690: Redundant self-check insidecase "RETRY_IMMEDIATELY"Inside the
"RETRY_IMMEDIATELY"branch you re-verify
if (attemptStatus !== "RETRY_IMMEDIATELY") { return; }, which can never be true because we are already in that case label. This is dead code that obscures the intent.Please remove the redundant guard.
920-931: Log helper may emit"snapshotId": undefinedbefore manager initialisation
sendDebugLogalways injectssnapshotId: this.currentSnapshotFriendlyId, but many early logs (constructor,cancel, etc.) are executed before theSnapshotManageris created, resulting in noisyundefinedfields.Consider suppressing the property when not yet available:
properties: { ...properties, runId: this.runFriendlyId, - snapshotId: this.currentSnapshotFriendlyId, + ...(this.currentSnapshotFriendlyId && { snapshotId: this.currentSnapshotFriendlyId }), ... }This keeps the log payload clean and avoids confusion during debugging.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (25)
apps/supervisor/src/workloadServer/index.ts(2 hunks)apps/webapp/app/routes/engine.v1.worker-actions.runs.$runFriendlyId.snapshots.since.$snapshotId.ts(1 hunks)apps/webapp/app/v3/services/worker/workerGroupTokenService.server.ts(1 hunks)internal-packages/run-engine/src/engine/index.ts(2 hunks)internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts(4 hunks)internal-packages/run-engine/src/engine/tests/waitpoints.test.ts(1 hunks)packages/cli-v3/package.json(2 hunks)packages/cli-v3/src/commands/deploy.ts(7 hunks)packages/cli-v3/src/entryPoints/dev-run-worker.ts(3 hunks)packages/cli-v3/src/entryPoints/managed-run-worker.ts(3 hunks)packages/cli-v3/src/entryPoints/managed/controller.ts(12 hunks)packages/cli-v3/src/entryPoints/managed/env.ts(3 hunks)packages/cli-v3/src/entryPoints/managed/execution.ts(31 hunks)packages/cli-v3/src/entryPoints/managed/notifier.ts(1 hunks)packages/cli-v3/src/entryPoints/managed/overrides.ts(2 hunks)packages/cli-v3/src/entryPoints/managed/poller.ts(5 hunks)packages/cli-v3/src/entryPoints/managed/snapshot.test.ts(1 hunks)packages/cli-v3/src/entryPoints/managed/snapshot.ts(1 hunks)packages/cli-v3/src/executions/taskRunProcess.ts(8 hunks)packages/core/src/v3/runEngineWorker/supervisor/http.ts(2 hunks)packages/core/src/v3/runEngineWorker/supervisor/schemas.ts(2 hunks)packages/core/src/v3/runEngineWorker/workload/http.ts(2 hunks)packages/core/src/v3/runEngineWorker/workload/schemas.ts(2 hunks)packages/core/src/v3/schemas/messages.ts(3 hunks)packages/core/src/v3/serverOnly/httpServer.ts(3 hunks)
✅ Files skipped from review due to trivial changes (1)
- packages/core/src/v3/runEngineWorker/workload/schemas.ts
🚧 Files skipped from review as they are similar to previous changes (7)
- packages/cli-v3/package.json
- packages/cli-v3/src/entryPoints/dev-run-worker.ts
- packages/cli-v3/src/commands/deploy.ts
- packages/cli-v3/src/entryPoints/managed-run-worker.ts
- packages/cli-v3/src/entryPoints/managed/env.ts
- packages/core/src/v3/schemas/messages.ts
- packages/core/src/v3/runEngineWorker/supervisor/schemas.ts
🧰 Additional context used
🧬 Code Graph Analysis (1)
packages/cli-v3/src/entryPoints/managed/notifier.ts (2)
packages/cli-v3/src/entryPoints/managed/controller.ts (1)
SupervisorSocket(21-21)packages/cli-v3/src/entryPoints/managed/logger.ts (2)
RunLogger(17-19)SendDebugLogOptions(9-15)
⏰ Context from checks skipped due to timeout of 90000ms (5)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
- GitHub Check: units / 🧪 Unit Tests
- GitHub Check: typecheck / typecheck
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (46)
apps/webapp/app/v3/services/worker/workerGroupTokenService.server.ts (1)
762-773: Clean implementation of snapshot retrieval.The new
getSnapshotsSincemethod follows the established pattern of other methods in this class, properly converting friendly IDs to internal IDs before calling the engine.packages/core/src/v3/runEngineWorker/supervisor/http.ts (2)
20-20: LGTM: Import addition for new snapshot feature.The import of
WorkerApiRunSnapshotsSinceResponseBodysupports the new snapshot retrieval functionality.
189-201: Well-structured HTTP client method.The implementation follows the established pattern in this class for HTTP client methods, with proper request construction and response validation.
packages/core/src/v3/runEngineWorker/workload/http.ts (2)
14-14: LGTM: Import addition for new snapshot feature.The import of
WorkloadRunSnapshotsSinceResponseBodysupports the new snapshot retrieval functionality.
146-157: Consistent implementation of snapshot retrieval.The
getSnapshotsSincemethod follows the same pattern as other HTTP methods in this class, with appropriate error handling via thewrapZodFetchutility.packages/core/src/v3/serverOnly/httpServer.ts (3)
204-204: Improved error formatting for better logging.Using the new
formatErrormethod provides more structured and consistent error logs.
213-213: Improved error formatting for better logging.Using the new
formatErrormethod provides more structured and consistent error logs.
368-378: Good error formatting utility.This helper method improves error logging by providing a consistent structure for different error types. It properly extracts name, message, and stack from Error objects while handling non-Error values gracefully.
apps/supervisor/src/workloadServer/index.ts (1)
20-20: Import addition looks goodClean addition of the
WorkloadRunSnapshotsSinceResponseBodyimport to support the new route.packages/cli-v3/src/entryPoints/managed/overrides.ts (2)
2-3: Added metadata properties for run and snapshot trackingThe addition of
TRIGGER_RUN_IDandTRIGGER_SNAPSHOT_IDproperties to theMetadatatype provides a clean way to track run and snapshot context across the managed runtime.
22-33: Improved error handling ingetEnvOverridesThis refactoring changes the return type to a discriminated union tuple, improving error handling and transparency. The implementation now explicitly checks response status and properly propagates errors rather than logging and returning null.
This makes error handling more predictable for consumers of this API.
internal-packages/run-engine/src/engine/tests/waitpoints.test.ts (1)
1348-1490: Comprehensive test coverage for the newgetSnapshotsSincefunctionalityThis test is well-structured and thoroughly tests the
getSnapshotsSincemethod with various inputs and edge cases. It properly:
- Creates multiple snapshots by triggering a run and using waitpoints
- Verifies that snapshots returned contain expected data
- Tests behavior with the first snapshot ID, latest snapshot ID, and invalid IDs
- Checks specific properties like
completedWaitpointsand error handlingThe test ensures proper cleanup by quitting the engine instance in a finally block, which is good practice.
packages/cli-v3/src/entryPoints/managed/notifier.ts (6)
1-12: Well-structured interface design for the RunNotifierThe notifier options interface and dependencies are clearly defined with appropriated types. This makes the notifier component easy to integrate and configure.
13-30: Good class structure with clean property initializationThe class properties are properly typed and initialized, clearly distinguishing between configuration properties and internal state tracking metrics.
32-64: Well-implemented start method with notification handlingThe notification mechanism generates a unique ID for tracking, performs validation, and includes comprehensive debug logging. The chained return of
thisenables fluent API usage.Consider enhancing the notification ID generation with a more robust unique ID approach if this needs to scale to high volumes of notifications:
- const notificationId = Math.random().toString(36).substring(2, 15); + const notificationId = `${Date.now()}-${Math.random().toString(36).substring(2, 8)}`;
66-69: Clean stop method implementationThe stop method appropriately removes all listeners to prevent memory leaks and includes debug logging.
71-78: Good observability with metrics getterThe metrics getter provides a clean interface for external components to access notification statistics without exposing internal class properties directly.
80-89: Well-structured debug logging methodThe private sendDebugLog method consistently formats logs and enriches them with metrics data, enhancing debuggability.
internal-packages/run-engine/src/engine/index.ts (3)
44-48: Enhanced imports for snapshot functionalityThe additional imports for snapshot processing support the new functionality for retrieving snapshots since a given point.
1102-1112: Improved error handling in getRunExecutionDataThe refactored method now uses the new
executionDataFromSnapshothelper for consistent data transformation and includes proper error handling with detailed logging.
1114-1134: Well-implemented new getSnapshotsSince methodThis method retrieves all execution snapshots since a specified ID, with appropriate error handling and consistent data transformation using the
executionDataFromSnapshothelper.packages/cli-v3/src/entryPoints/managed/poller.ts (10)
2-4: Good dependency management with clear type definitionThe poller now uses the IntervalService for timing functionality and defines a clean OnPoll callback type.
11-12: Simplified interface with callback approachThe poller options now use a callback approach instead of direct dependency on the HTTP client, improving modularity.
19-25: Enhanced property definitions with metrics trackingAdded metrics tracking for polls, which improves observability.
32-33: Improved constructor with callback assignmentThe constructor now assigns the onPoll callback instead of handling polling logic internally.
45-48: Good metrics tracking implementationThe poller now tracks metrics like lastPollAt and pollCount before invoking the callback.
64-67: Clear documentation on snapshotId usageThe comment clarifies that the snapshot ID is used as an indicator for stuck pollers, which helps with future maintenance.
73-85: Enhanced start method with fluent interfaceThe start method now returns
thisfor method chaining and includes improved logging.
87-102: Improved stop method with execution status detectionThe stop method now detects if the poller was executing during stop, which helps with debugging race conditions.
104-109: Added metrics getter for observabilityThe metrics getter provides a clean interface for accessing poll statistics.
111-122: Enhanced debug logging with metricsThe updated sendDebugLog method includes metrics in the log properties, which improves debugging capabilities.
packages/cli-v3/src/entryPoints/managed/snapshot.test.ts (9)
1-26: Well-structured test setup with mock dependenciesThe test suite uses proper mocking and setup for the SnapshotManager tests.
27-111: Comprehensive basic functionality testsThe tests for initialization, snapshot updates, and suspendable state transitions provide good coverage of the core functionality.
112-210: Thorough queue processing testsThe tests for queue order, duplicate handling, and snapshot validation logic are well-implemented.
212-277: Excellent concurrency handling testsThe tests for preventing concurrent handler execution use timing verification to ensure handlers don't overlap.
279-365: Comprehensive error handling and cleanup testsThese tests verify proper behavior during errors and cleanup scenarios.
367-478: Thorough edge case handlingThe tests for edge cases like empty IDs, long IDs, and high concurrency are well-designed to catch potential issues.
480-630: Well-structured queue processing testsThese tests verify proper queue processing behavior including rapid changes during processing.
632-699: Thorough runner restoration testingThe tests for detecting restored runners and handling deprecation markers are comprehensive.
702-734: Useful test helper functionThe createRunExecutionData helper provides a clean way to generate test data with sensible defaults.
internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts (1)
198-218: Possible snapshot omission when two snapshots share the samecreatedAtFiltering with
createdAt > sinceSnapshot.createdAtcan miss a snapshot generated in the same millisecond (ULID/Timestamp collisions on busy systems).
Safer options:
- Filter on
(createdAt > …) OR (createdAt = … AND id > sinceSnapshotId)if IDs are monotonic.- Simply request
gt: sinceSnapshot.idwhen your ID encodes time.Please verify whether duplicate timestamps are realistically possible with your DB adapter; adjust the query if necessary.
packages/cli-v3/src/entryPoints/managed/controller.ts (2)
196-198: Redundant removal of"run:notify"listeners
this.socket.removeAllListeners("run:notify")is executed, yet no listener is ever attached in this class any more (notification handling was delegated toRunNotifier).
Double-check that nothing else attaches that event to this socket; otherwise the call can be dropped.
397-399:kill()is asynchronous butprocess.exit()follows immediately
this.currentExecution?.kill()returns a promise that won’t settle onceprocess.exit()is invoked, possibly leaving child processes around for a brief period.
Consider awaiting the promise with a modest timeout (or calling a synchronous kill) before exiting.- this.currentExecution?.kill().catch(() => {}); - process.exit(code); + await Promise.race([ + this.currentExecution?.kill().catch(() => {}), + new Promise(r => setTimeout(r, 2_000)) // fallback timeout + ]); + process.exit(code);packages/cli-v3/src/executions/taskRunProcess.ts (2)
206-213: 👍 New event plumbing looks goodThe handlers for
SEND_DEBUG_LOGandSET_SUSPENDABLEare concise and forward messages viaEvt– no issues spotted.
297-298: Verify peer support for"RESOLVE_WAITPOINT"messageThe emitter now sends
"RESOLVE_WAITPOINT". Ensure both the worker side and the updatedSharedRuntimeManagerhandle this verb; otherwise waitpoint completion signalling will break silently.packages/cli-v3/src/entryPoints/managed/snapshot.ts (1)
103-106: Potentially unsafe ordering check –stringcomparison may drop newer snapshots
snapshotId < this.state.idrelies on lexicographical ordering of the friendlyId.
Unless you can guarantee that the ID scheme is strictly monotonic-sortable (e.g. ULIDs), the comparison may mis-classify a newer snapshot that happens to compare as “smaller” (common with CUID/cuid2). That would silently drop legitimate updates and stall the run.-if (snapshotId < this.state.id) { +// Prefer an explicit comparator that understands the ID format or, +// if the IDs embed a timestamp, compare the timestamps instead. +if (this.isOlderSnapshot(snapshotId, this.state.id)) {Consider introducing a helper (
isOlderSnapshot) that either decodes the timestamp portion or falls back tolocaleComparewith a formal guarantee.
Failing to do so could re-introduce the very race conditions this PR tries to eliminate.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
packages/core/src/v3/runtime/sharedRuntimeManager.ts (2)
261-265: Fix potential iteration issue when modifying the MapThe current implementation iterates over
waitpointsByResolverId.entries()while potentially modifying the Map insideresolveWaitpoint(), which could lead to unpredictable behavior in some JavaScript engines.
284-306: Add defensive JSON parsing to handle malformed outputThe
JSON.parsecall is not wrapped in a try-catch block, which could cause runtime errors if the output is not valid JSON.
🧹 Nitpick comments (5)
packages/core/src/v3/runtime/sharedRuntimeManager.ts (5)
39-41: Consider using a named constant instead of a magic numberThe interval duration
300_000(5 minutes in milliseconds) would be more maintainable if defined as a named constant at the top of the file or class.+ private static readonly STATUS_LOG_INTERVAL_MS = 5 * 60 * 1000; // 5 minutes constructor( private ipc: ExecutorToWorkerProcessConnection, private showLogs: boolean ) { // Log out the runtime status on a long interval to help debug stuck executions setInterval(() => { this.debugLog("SharedRuntimeManager status", this.status); - }, 300_000); + }, SharedRuntimeManager.STATUS_LOG_INTERVAL_MS); }
84-90: Unused reject parameter in Promise constructorThe Promise constructor receives a reject parameter that's never used. This can lead to confused developers and unhandled rejections.
const promises = Array.from({ length: params.runCount }, (_, index) => { const resolverId = `${params.id}_${index}` as ResolverId; - return new Promise<CompletedWaitpoint>((resolve, reject) => { + return new Promise<CompletedWaitpoint>((resolve) => { this.resolversById.set(resolverId, resolve); }); });
219-225: Document why BATCH waitpoints are ignoredWhile the code includes a comment about ignoring BATCH waitpoints, it would be helpful to document this design decision more clearly, especially if it might change in the future.
if (waitpoint.type === "BATCH") { - // We currently ignore these, they're not required to resume after a batch completes + // BATCH waitpoints are intentionally ignored because batches are resumed via their individual task waitpoints + // Each task in the batch has its own waitpoint that is processed separately + // TODO: If this behavior changes in the future, update this implementation this.debugLog("ignoring BATCH waitpoint", { waitpoint: this.waitpointForDebugLog(waitpoint), }); return; }
308-319: Consider optimizing debug log data structureThe method includes both ISO string representations and Date objects for timestamps, which is redundant and increases the log payload size.
private waitpointForDebugLog(waitpoint: CompletedWaitpoint): DebugLogPropertiesInput { const { completedAfter, completedAt, output, ...rest } = waitpoint; return { ...rest, output: output?.slice(0, 100), completedAfter: completedAfter?.toISOString(), completedAt: completedAt?.toISOString(), - completedAfterDate: completedAfter, - completedAtDate: completedAt, }; }
338-341: Consider unifying log methodsThere are two separate logging methods (
debugLogandlog) with different behaviors. Consider unifying them or more clearly documenting their distinct purposes.- private log(message: string, ...args: any[]) { - if (!this.showLogs) return; - console.log(`[${new Date().toISOString()}] ${message}`, args); - } + /** + * Basic logging method for high-volume logs that don't need to be sent via IPC + * Use debugLog for important diagnostic information that should be available remotely + */ + private log(message: string, ...args: any[]) { + if (!this.showLogs) return; + console.log(`[${new Date().toISOString()}] ${message}`, ...args); + }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
packages/core/src/v3/runtime/sharedRuntimeManager.ts(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
packages/core/src/v3/runtime/sharedRuntimeManager.ts (5)
packages/core/src/v3/workers/index.ts (2)
SharedRuntimeManager(24-24)RuntimeManager(2-2)packages/core/src/v3/runtime/preventMultipleWaits.ts (1)
preventMultipleWaits(7-32)packages/core/src/v3/lifecycle-hooks-api.ts (1)
lifecycleHooks(5-5)packages/core/src/utils.ts (1)
assertExhaustive(1-3)packages/core/src/v3/runEngineWorker/supervisor/schemas.ts (2)
DebugLogPropertiesInput(141-141)DebugLogPropertiesInput(142-142)
⏰ Context from checks skipped due to timeout of 90000ms (6)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
- GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
- GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
- GitHub Check: units / 🧪 Unit Tests
- GitHub Check: typecheck / typecheck
- GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (5)
packages/core/src/v3/runtime/sharedRuntimeManager.ts (5)
169-213: Approved method for resolver ID determinationThe resolver ID determination logic is comprehensive and handles all waitpoint types correctly with appropriate error handling and logging.
267-269: Approved simple suspendable state setter methodThe
setSuspendablemethod correctly sends an IPC message to communicate the suspendable state.
271-282: Approved suspendable wrapper with good error handlingThe
suspendablemethod implements a robust pattern for wrapping promises with suspendable state management and proper error handling.
321-336: Approved debug logging with runtime statusThe debug logging implementation includes helpful runtime status information and properly sends logs through IPC for monitoring.
343-348: Approved runtime status implementationThe status getter provides a clean, serializable representation of the internal state for debugging purposes.
Quite a few things in here:
We now require two things before suspending a run:
EXECUTING_WITH_WAITPOINTSorQUEUED_EXECUTINGSummary by CodeRabbit
New Features
SnapshotManagerandRunNotifierto centralize snapshot state and notification handling.ConsoleRunLoggerand enhanced debug logging capabilities.Bug Fixes
Refactor
ManagedRuntimeManagerwithSharedRuntimeManagerfor better waitpoint management.Tests
SnapshotManagerand snapshot retrieval since a given snapshot.getSnapshotsSincebehavior and error handling.Chores
Documentation