Docs: the on-demand TLS "ask" example for Caddy doesn't validate anything (and it took down our entire domain's HTTPS for 30+ hours) #29079

Summary

The Caddy reverse-proxy tutorial (https://coder.com/docs/tutorials/reverse-proxy-caddy) includes this global block:

{
    on_demand_tls {
        ask http://example.com
    }
}

with an explicit warning: "DO NOT CHANGE the ask http://example.com line! Doing so will result in your certs potentially not being generated."

This line looks like a security control but isn't one. example.com (and example.org, seen in some doc revisions) is a real, live, unrelated third-party domain that returns HTTP 200 for any GET request. Since Caddy's on-demand TLS treats any 2xx response from the ask endpoint as "approved," this configuration approves certificate issuance for literally any hostname an attacker points at the server. It provides zero validation while looking like it does.

What happened on our end

We followed this tutorial to expose Coder workspace apps via *.dv.example.com with on-demand TLS, exactly as documented. Within days, a scanner (fabricated hostnames included the literal string "notexists", suggesting an automated probe for this exact class of vulnerability — see caddyserver/caddy#6188 for the general pattern) sent TLS handshakes for many fake hostnames matching Coder's {port}--{agent}--{workspace}--{owner} wildcard naming convention. Because the ask check always returned 200, Caddy dutifully tried to obtain a real Let's Encrypt certificate for each fake hostname.

This exhausted Let's Encrypt's account-wide rate limit (50 certificates per registered domain per 168h) for our entire base domain — not just the Coder wildcard. Every other service on the same domain (unrelated apps, mail, other subdomains) lost the ability to renew or obtain certificates for roughly 30 hours across several retry cycles, since the rate limit is scoped to the registrable domain, not the specific site block.

Root cause

Caddy's own docs are explicit that on-demand TLS is "insecure unless you also configure the on_demand_tls global option to mitigate abuse" (https://caddyserver.com/docs/caddyfile/directives/tls). The Coder tutorial provides no real mitigation — the example.com placeholder is functionally equivalent to having no ask check at all, while the "DO NOT CHANGE" warning actively discourages users from fixing it.

What we built instead

A small standalone Node.js service that Caddy's ask directive points to, which:

1. Parses the incoming hostname against Coder's documented wildcard format ({app}--{agent}--{workspace}--{owner}.wildcard-domain, per https://coder.com/docs/admin/setup#wildcard-access-url).
2. Queries GET /api/v2/workspaces?q=owner:{owner} name:{workspace} with a Coder-Session-Token to confirm the workspace genuinely exists before approving certificate issuance.

This works, but required reverse-engineering the naming convention and building custom infrastructure that arguably should be a first-class, documented Coder feature.

Side note: we also hit CODER_MAX_TOKEN_LIFETIME being hard-capped at 168h regardless of the configured value (relevant to anyone building a similar long-lived service-account integration — see #17395), which forced us to add a token-rotation timer as well.

Ask

1. Fix the docs: replace the example.com/example.org placeholder with either a real working example, or an explicit warning that it must be replaced with a real validation endpoint before production use.
2. Consider shipping a first-class Coder endpoint purpose-built for this (e.g. something like GET /api/v2/ok-to-issue-cert?domain=...), so self-hosters can point Caddy's ask directly at Coder without building a custom validator and managing a separate API token.

Happy to share our validator script if useful as a reference/starting point.