bug: workspace proxy health checks remain unhealthy after DNS switches from primary to proxy #29045

Current Behavior

Workspace proxy health checks continued reporting an HTML response from /healthz-report after the proxy hostname was switched from a wildcard record targeting the primary to an explicit record targeting the proxy.

Observed while bootstrapping a workspace proxy. Hostnames and addresses below are placeholders:

• Initially, *.example.com pointed to the primary load balancer. The proxy registered before its explicit DNS record existed.
• After the proxy load balancer was created, an explicit proxy.example.com record was added pointing to it.
• The proxy replicas were healthy and configured to run /opt/coder wsproxy server with the correct primary and proxy access URLs.
• A fresh external request to the proxy's /healthz-report returned HTTP 200 with application/json; charset=utf-8 and no redirect.
• A subsequent proxy-list response still showed unhealthy with an advancing proxy checked at timestamp, reporting HTML instead of JSON.
• Subsequent fresh DNS lookups and curl requests from every primary replica resolved the proxy hostname to the same IP and returned HTTP 200 JSON with no redirect.

This was observed on one deployment. Stale HTTP connection reuse is a hypothesis, not a confirmed root cause.

Relevant Log Output

The primary reported this on a fresh health check after the successful external request:

proxy url "https://proxy.example.com/healthz-report" is a coder instance, but unable to decode the response payload. Could this be a primary coderd and not a proxy?
attempted to query health at "https://proxy.example.com/healthz-report" but got back the incorrect content type: unexpected non-JSON response "text/html; charset=utf-8"

Fresh curl requests from each primary pod subsequently returned:

HTTP: 200
Content-Type: application/json; charset=utf-8
Remote IP: 203.0.113.10
Redirect:

The diagnostic request deliberately did not follow redirects:

getent hosts proxy.example.com
curl -sS --max-time 15 \
  -o /dev/null \
  -w 'HTTP: %{http_code}\nContent-Type: %{content_type}\nRemote IP: %{remote_ip}\nRedirect: %{redirect_url}\n' \
  https://proxy.example.com/healthz-report

Expected Behavior

After DNS points to the healthy proxy, health checks should recover automatically within a bounded interval without requiring a primary restart. If redirects are followed, diagnostics should distinguish the originally requested URL from the final URL that returned unexpected content.

Steps to Reproduce

The following describes the observed rollout sequence; it has not yet been reproduced independently:

1. Configure a parent wildcard DNS record that directs unmatched hostnames to the primary Coder endpoint.
2. Register and start a workspace proxy before creating its explicit DNS record. Allow primary health checks to run while the proxy hostname still reaches the primary.
3. Create an explicit DNS record pointing to the proxy's load balancer.
4. Verify that a new external request to the proxy's /healthz-report returns JSON.
5. Inspect coder workspace-proxy list --column 'name,url,proxy status,proxy checked at,proxy report'. In this rollout, the check timestamp advanced but the primary still reported non-JSON HTML.
6. Run fresh DNS lookups and curl requests inside each primary replica. In this rollout, every replica returned the correct proxy IP and JSON response.

Relevant Configuration

• Primary and proxy have separate load balancers.
• The deployment has multiple primary replicas.
• Primary access-URL redirection is enabled with CODER_REDIRECT_TO_ACCESS_URL=true.
• The reported failure comes from the primary's health checker, not the CLI's own HTTP probe.

Investigation Notes

Source inspection at acae170664 suggests a possible persistent-connection path:

• The proxy health checker constructor shallow-copies the supplied HTTP client and sets a five-second timeout, retaining its transport and redirect policy.
• The probe does not explicitly require a new connection. Its error identifies the original URL, not necessarily the final redirected destination.
• Health results are held per primary replica. The default background interval is one minute, and successful proxy registrations can trigger additional checks.
• The primary's canonical-URL redirect middleware can redirect /healthz-report; that path is not the exempt /healthz endpoint.

A connection established before the DNS change could therefore continue reaching the primary. Repeated checks can keep a pooled connection active, so a fresh check timestamp does not guarantee a fresh DNS lookup or connection. This still needs confirmation; no connection trace or transport instrumentation was collected.

The supported forced deployment-health refresh runs another probe but does not explicitly flush the HTTP transport. It was not tested during this investigation.

A controlled primary rolling restart was proposed and approved to test clearing the connection pools. Its execution and outcome have not yet been reported, so it is not a confirmed workaround.

Existing Issues

Searched open issues for workspace proxy health, DNS, and connection reuse; no matching issue was found in that search.