TIL, thanks!

If this is a low-hanging fruit, then most likely we should look into this.

We could also consider this for our dogfood template, removing provisioning load from the Coder server and relying on runners on each bare metal instance to provision containers with their respective local docker sockets.

👍

BTW well described issue, Ben.

This issue is becoming stale. In order to keep the tracker readable and actionable, I'm going close to this issue in 7 days if there isn't more activity.

Bump to remove the stale label, my team is still wishing for this feature. 🙏

It seems like this should both support targeting specific provisioners (e.g. single runner running on a VM) or tags/groups of provisioners (e.g. cluster 1 (100 daemons running)

One thing we could do fairly straightforwardly is to allow rich parameters to set tags on the build job, which then can restrict which provisionerds are able to build the workspace. For simplicity, we should require the parameter to be immutable.

Work items:

1. Enhance the Coder Terraform Provider to allow coder_parameter to accept an argument marking it as a tag.
2. Database migration that adds the tag information to the rich parameter in the database
3. Enhance the workspace Builder to set tags based on rich parameters so marked.

This gets us a basic working version, but falls short of

Instead of hardcoded values, Coder pulls the list of available runners and perhaps shows latency.

We'd have to depend on hardcoded values in the template, so there would have to be out-of-band coordination between the operators who set up the provisionerds with tags and the template authors who want to allow end users to tag their jobs. (This is basically the situation today, with tags set at the template version.)

The difficulty in pulling a list of available provisioners is that Coder doesn't presently track the lifecycle of provisionerds. They get added to the database when they connect, but then are never updated when they disconnect.

I also think latency isn't very sensible. Latency from what to what? Presumably users are interested in minimizing the latency from their computer to their workspace, but we can't measure that prior to building the workspace. Provisionerds don't respond to latency pings from end users, and adding that is difficult because there is no requirement for them to be externally reachable. While you might expect that provisionerds are often located near where the workspaces will be created, this is far from universal. Consider the case of public clouds like AWS: a provisioner job talking to the AWS APIs can create a workspace in any zone across the whole world.

Yeah latency was a bit of a silly idea, we don't need to do that :)

As this is part of this sprint, let's list changes in the terraform-provider.

Enhance the Coder Terraform Provider to allow coder_parameter to accept an argument marking it as a tag.

@spikecurtis I'd like to follow up on your vision. Did you mean something like this:

data "coder_parameter" "ci_provisioner" {
  name        = "ci_provisioner"
  type        = "string"
  description = "Select a runner"

  option {
    name  = "Ben's Macbook"
    value = "bens-macbook"
  }
  option {
    name  = "US Pittsburgh"
    value = "us-pittsburgh"
  }
  option {
    name  = "Sydney"
    value = "sydney"
  }
  option {
    name  = "Ben's Thinkpad"
    value = "bens-thinkpad"
  }

  role = "select_provisioner"
}

In this case, option values would be tags: bens-macbook, us-pittsburgh, sydney, and bens-thinkpad.
Coderd/UI could determine the role of coder_parameter to be a provisioner selector based on the role property.

1. Should we support default property for the provisioner selector?
2. Maybe the provisioner selector should be a new rich parameter type?
3. I presume that we don't allow to specify more than 1 selector per template?

Side note:

Maybe we could just introduce another TF resource like coder_provisioner, which is not a rich parameter,
but can provide a list of allowed provisioner tags.

Hey all, my team is one of the teams asking for this feature. We strictly need this aspect:

Instead of hardcoded values, Coder pulls the list of available runners

to support arbitrary developers spinning up workspaces on personal/local provisioners running on their laptops (see #909) that we can't pre-bake as hard-coded template parameter options.

We're trying to improve the experience today where:

1. If a developer has no local/personal provisioner daemons running, currently nothing prevents them from trying to create a workspace from a template that only targets their local/personal provisioner daemon(s), but that workspace build will be stuck as pending seemingly forever and the developer won't have much feedback to understand that the problem is their personal/local provisioner daemon isn't running. To avoid that, we're hoping we could show them the list of provisioner daemons that are available for them to target on the workspace creation webpage and require one to be selected for the creation process to proceed. This blocking requirement on the workspace creation page may also give us an opportunity to show some documentation (e.g. in the parameter help/description text) explaining what they may need to do to get their local/personal provisioner daemon running again.
2. In the rare-but-possible case that a developer has multiple laptops/computers that are each running personal/local provisioner daemons, it's not clear which of those machines would be targeted to host a workspace the developer starts up from a template that only targets the user's local/personal provisioner daemons, but the developer has a specific machine they are wanting to target. That's also why we're hoping the selected/targeted provisioner could become an immutable property of the workspace, so that when the workspace is restarted it always targets the same provisioner daemon. Since we use Docker containers on the same host where the provisioner daemon is running in our workspace template, it would be invalid for a workspace to unexpectedly move from one host to another if the provisioner daemon isn't constant because the workspace's persisted Docker resources would not be present on a different host.

I think this is a good opportunity for us to have "magic parameters" where the options and validation are not defined Terraform but instead by coderd: for example:

• which provisioners does the template (and user) have permissions have access to (a just-in-time API call right before the create workspace UI/CLI)
• which git repositories does the template (and user) have access to clone (uses the users' git access token + coderd to curate api responses for parameter values)?
• perhaps a bit crazier: which VPCs does this template (and user) have access to?

I suppose this could be a "pre-create-workspace-page plan" 😅 but these kind of these things could be better solved without Terraform (good ol' REST APIs).

I imagine this may cause some heavy refactors if we try to implement this in parameters and there is another risk in duplication if we introduce a different data source:

data "coder_magic_parameter" "provisioner" {
  options: "available provisioners"
}

data "coder_magic_parameter" "pick_a_repo" {
  options: "repos"
}

This kind of reminds me how we do gitauth in Coder. https://coder.com/docs/v2/latest/admin/git-providers#require-git-authentication-in-templates

I guess the parameter picker should be its own data source ❓

Nice! I like this

Would also like to suggest we add "provisioner metadata," which is very similar to agent metadata but can be used to display dynamic info about a provisioner. This does not have to be in the first version, but can easily inform users details about each provisioner (OS, number of GPUs on node, etc). Perhaps we provide some defaults but then the rest can be sent with an REST API request (no CLI definition or anything).

Edit: made an issue for this: #10412

From a chat with @mtojek - we need to agree on whether this is a special rich parameter or done via a UI form control. I am leaning towards a parameter.

I'll bring this up during our call with Kyle next week

FYI I'm working on the RFC right now, and will come up with the design by the end of this week.

Battle plan:

• coderd: implement coder_workspace_tags #13218
• coderd: select provisioners based on workspace tags #13219
• coderd: support workspace tags in "duplicate template" workflow #13220
• docs: add docs and examples with coder_workspace_tags  #13221

The RFC does not discuss dynamic provisioner tags, so the short answer is: no, not yet. We can focus on the dynamic list of active tags later.

@mtojek I agree, and I am writing down this idea for future reference.

What will be the behavior when the provided tags do not match any provisioner? What is the plan to evict these build jobs from the build queue? As currently, the behavior is that they always stay stuck in a pending state.

What will be the behavior when the provided tags do not match any provisioner? What is the plan to evict these build jobs from the build queue? As currently, the behavior is that they always #12331.

Stuck as it is now. We will need a separate RFC to describe a concept of "dead letter queue". It was mentioned in the RFC.

Stuck as it is now. We will need a separate RFC to describe a concept of "dead letter queue". It was mentioned in the RFC.

@mtojek Thanks for the update.

As all items have been implemented according to the RFC, I'm closing this issue now.