-
Notifications
You must be signed in to change notification settings - Fork 1k
Open
Description
Summary
Dynamic Parameters websocket connections are closed by AWS Application Load Balancer (ALB) after ~60 seconds of inactivity. The dynamic-parameters websocket handler accepts the connection but does not start the standard server-side heartbeat (15s ping), so the ALB idle timeout triggers and drops the TCP connection. Users see the UI error:
Websocket connection for dynamic parameters unexpectedly closed.
Refresh the page to reset the form.
Impact
- Users configuring templates may have the Dynamic Parameters form close unexpectedly after ~1 minute of inactivity behind ALB (default idle timeout = 60s), forcing them to refresh and losing context.
- Affects any deployment behind intermediaries with idle timeouts (ALB, reverse proxies, firewalls) when there is no traffic on the socket.
Environment
- Coder server behind AWS ALB (or similar proxy) with default Connection idle timeout of 60 seconds.
- Dynamic parameters websocket endpoint:
GET /api/v2/templateversions/{templateversion}/dynamic-parameters
. - Coder 2.24-2.25.2
Root Cause
- The dynamic parameters websocket handler
handleParameterWebsocket
does not start a server-side heartbeat loop that periodically pings the websocket. - Other websocket endpoints in the codebase call
go httpapi.Heartbeat(ctx, conn)
after accepting the connection (15s ping interval), which prevents ALB idle timeouts. - Without any application-level traffic or pings, ALB closes the idle connection after its configured idle timeout.
Evidence (code references)
- Dynamic parameters handler accepts the websocket but does not start a heartbeat:
coderd/parameters.go
inhandleParameterWebsocket
. - Standard heartbeat helper:
coderd/httpapi/websocket.go
(15 second ping loop). - Pattern used elsewhere:
coderd/workspaceagents.go
:go httpapi.Heartbeat(ctx, conn)
coderd/provisionerjobs.go
:go httpapi.Heartbeat(f.ctx, f.conn)
coderd/inboxnotifications.go
:go httpapi.Heartbeat(ctx, conn)
Proposed Fix
Add a heartbeat to the dynamic parameters websocket handler immediately after successful websocket.Accept
:
func (api *API) handleParameterWebsocket(rw http.ResponseWriter, r *http.Request, initial codersdk.DynamicParametersRequest, render dynamicparameters.Renderer) {
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Minute)
defer cancel()
conn, err := websocket.Accept(rw, r, nil)
if err != nil {
// ... existing error handling ...
return
}
+ // Keep the connection alive behind load balancers with idle timeouts
+ go httpapi.Heartbeat(ctx, conn)
stream := wsjson.NewStream[codersdk.DynamicParametersRequest, codersdk.DynamicParametersResponse](
conn,
websocket.MessageText,
websocket.MessageText,
api.Logger,
)
// ... rest of function ...
}
Alternatives / Mitigations
- Increase ALB "Connection idle timeout" (e.g., 3600-4000 seconds). This reduces frequency of drops but is less robust than an application heartbeat and still subject to other middleboxes.
- Client-side keepalive: periodically send a minimal DynamicParametersRequest (e.g., unchanged inputs with incremented id) every ~30-45s. Prefer server-side ping for consistency with other endpoints.
Not Related
- This is not related to removal of agent websocket pings in PR chore: remove pingWebSocket since yamux runs keepalives #11914; that change relies on yamux keepalives in the agent path. The dynamic parameters websocket does not use yamux and requires its own heartbeat.
Reproduction Steps
- Place Coder behind an ALB with default 60s idle timeout.
- Navigate to create/parameters UI that uses dynamic parameters.
- Open DevTools network tab to observe the websocket.
- Leave the UI idle (>60s) without any parameter changes.
- Observe the websocket close and the UI error message.
Acceptance Criteria
- Dynamic parameters websocket remains open indefinitely when idle (well beyond 60s) behind ALB when no user input occurs, due to heartbeat pings.
- No regressions in dynamic parameters functionality.
Risk
- Minimal: same heartbeat pattern is used broadly across the codebase. The heartbeat goroutine exits on context cancellation and closes cleanly if the client disconnects.
Owner / Area
- Area: Dynamic Parameters backend (coderd), UI Create Workspace / Template Parameters.
- Owner: Templates / Parameters subsystem maintainers.
References
- AWS ALB Connection idle timeout defaults and range (1-4000s).
- Existing heartbeat implementation in
coderd/httpapi/websocket.go
.
Metadata
Metadata
Assignees
Labels
No labels