Is it technically possible to run more than 1 coderd on each node? If yes does this benefit any of the use cases or customers? Why would someone run multiple coderd on a single node?

Is it technically possible to run more than 1 coderd on each node?

Yes, this can happen automatically during a rollout or during node unavailability.
Note that we do set a pod anti-affinity rule [1] in our Helm chart to prefer spreading out replicas across multiple nodes.

If yes does this benefit any of the use cases or customers?
Why would someone run multiple coderd on a single node?

As far as I'm aware, the main reason to do this would be more for redundancy in case one or more pods become unavilable for whatever reason.

The only other reason I could imagine for running multiple replicas on a single node is to spread out connections across more coderd replicas to minimize the user-facing impact of a single pod failing. However, this won't protect against a failure of the underlying node.

I'll defer to @spikecurtis to weigh in more on the pros and cons of running multiple replicas per node.

[1] https://github.com/coder/coder/blob/main/helm/coder/values.yaml#L223-L237

In any reference architectures we should always recommend having 1 coderd per node.

There are generally 2 reasons for multiple replicas: fault tolerance and scale.

For fault tolerance, you want the replicas spread out into different failure domains. Having all replicas on the same node means you aren't tolerant of node-level faults. There might still be some residual value in being tolerant to replica level faults: e.g. software crashes, OOM. But, most people would rather the higher fault tolerance.

For scale, coderd is written to take advantage of multiple CPU cores in one process, so there is no scale advantage of putting multiple coderd instances on a single node. In fact, it's likely bad for scale since you have multiple processes competing for resources, and extra overhead of coderd to coderd communication.