Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Bug: Dynamic parameters websocket connections time out behind ALB due to missing heartbeat #19805

@bjornrobertsson

Description

@bjornrobertsson

Summary

Dynamic Parameters websocket connections are closed by AWS Application Load Balancer (ALB) after ~60 seconds of inactivity. The dynamic-parameters websocket handler accepts the connection but does not start the standard server-side heartbeat (15s ping), so the ALB idle timeout triggers and drops the TCP connection. Users see the UI error:

Websocket connection for dynamic parameters unexpectedly closed.
Refresh the page to reset the form.

Impact

  • Users configuring templates may have the Dynamic Parameters form close unexpectedly after ~1 minute of inactivity behind ALB (default idle timeout = 60s), forcing them to refresh and losing context.
  • Affects any deployment behind intermediaries with idle timeouts (ALB, reverse proxies, firewalls) when there is no traffic on the socket.

Environment

  • Coder server behind AWS ALB (or similar proxy) with default Connection idle timeout of 60 seconds.
  • Dynamic parameters websocket endpoint: GET /api/v2/templateversions/{templateversion}/dynamic-parameters.
  • Coder 2.24-2.25.2

Root Cause

  • The dynamic parameters websocket handler handleParameterWebsocket does not start a server-side heartbeat loop that periodically pings the websocket.
  • Other websocket endpoints in the codebase call go httpapi.Heartbeat(ctx, conn) after accepting the connection (15s ping interval), which prevents ALB idle timeouts.
  • Without any application-level traffic or pings, ALB closes the idle connection after its configured idle timeout.

Evidence (code references)

  • Dynamic parameters handler accepts the websocket but does not start a heartbeat: coderd/parameters.go in handleParameterWebsocket.
  • Standard heartbeat helper: coderd/httpapi/websocket.go (15 second ping loop).
  • Pattern used elsewhere:
    • coderd/workspaceagents.go: go httpapi.Heartbeat(ctx, conn)
    • coderd/provisionerjobs.go: go httpapi.Heartbeat(f.ctx, f.conn)
    • coderd/inboxnotifications.go: go httpapi.Heartbeat(ctx, conn)

Proposed Fix

Add a heartbeat to the dynamic parameters websocket handler immediately after successful websocket.Accept:

 func (api *API) handleParameterWebsocket(rw http.ResponseWriter, r *http.Request, initial codersdk.DynamicParametersRequest, render dynamicparameters.Renderer) {
     ctx, cancel := context.WithTimeout(r.Context(), 30*time.Minute)
     defer cancel()
 
     conn, err := websocket.Accept(rw, r, nil)
     if err != nil {
         // ... existing error handling ...
         return
     }
+    // Keep the connection alive behind load balancers with idle timeouts
+    go httpapi.Heartbeat(ctx, conn)
 
     stream := wsjson.NewStream[codersdk.DynamicParametersRequest, codersdk.DynamicParametersResponse](
         conn,
         websocket.MessageText,
         websocket.MessageText,
         api.Logger,
     )
     // ... rest of function ...
 }

Alternatives / Mitigations

  • Increase ALB "Connection idle timeout" (e.g., 3600-4000 seconds). This reduces frequency of drops but is less robust than an application heartbeat and still subject to other middleboxes.
  • Client-side keepalive: periodically send a minimal DynamicParametersRequest (e.g., unchanged inputs with incremented id) every ~30-45s. Prefer server-side ping for consistency with other endpoints.

Not Related

Reproduction Steps

  1. Place Coder behind an ALB with default 60s idle timeout.
  2. Navigate to create/parameters UI that uses dynamic parameters.
  3. Open DevTools network tab to observe the websocket.
  4. Leave the UI idle (>60s) without any parameter changes.
  5. Observe the websocket close and the UI error message.

Acceptance Criteria

  • Dynamic parameters websocket remains open indefinitely when idle (well beyond 60s) behind ALB when no user input occurs, due to heartbeat pings.
  • No regressions in dynamic parameters functionality.

Risk

  • Minimal: same heartbeat pattern is used broadly across the codebase. The heartbeat goroutine exits on context cancellation and closes cleanly if the client disconnects.

Owner / Area

  • Area: Dynamic Parameters backend (coderd), UI Create Workspace / Template Parameters.
  • Owner: Templates / Parameters subsystem maintainers.

References

  • AWS ALB Connection idle timeout defaults and range (1-4000s).
  • Existing heartbeat implementation in coderd/httpapi/websocket.go.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions