feat: track resource replacements when claiming a prebuilt workspace #17571

dannykopping · 2025-04-25T16:49:04Z

We can't know whether a replacement (i.e. drift of terraform state leading to a resource needing to be deleted/recreated) will take place apriori; we can only detect it at plan time, because the provider decides whether a resource must be replaced and it cannot be inferred through static analysis of the template.

This is likely to be the most common gotcha with using prebuilds, since it requires a slight template modification to use prebuilds effectively, so let's head this off before it's an issue for customers.

Drift details will now be logged in the workspace build logs:

Plus a notification will be sent to template admins when this situation arises:

A new metric - coderd_prebuilt_workspaces_resource_replacements_total - will also increment each time a workspace encounters replacements.

We only track that a resource replacement occurred, not how many. Just one is enough to ruin a prebuild, but we can't know apriori which replacement would cause this.
For example, say we have 2 replacements: a docker_container and a null_resource; we don't know which one might
cause an issue (or indeed if either would), so we just track the replacement.

enterprise/coderd/coderd.go

dannykopping · 2025-04-25T16:54:46Z

coderd/database/migrations/000318_resource_replacements_notification.up.sql

+If you’re using prebuilds to speed up provisioning, unexpected replacements will slow down
+workspace startup—even when claiming a prebuilt environment.
+
+For tips on preventing replacements and improving claim performance, see [this guide](https://coder.com/docs/TODO).


Relies on #17580

coderd/provisionerdserver/provisionerdserver.go

dannykopping · 2025-05-06T09:26:59Z

provisionerd/proto/provisionerd.proto

@@ -75,6 +75,7 @@ message CompletedJob {
        repeated provisioner.Resource resources = 2;
        repeated provisioner.Timing timings = 3;
        repeated provisioner.Module modules = 4;
+        repeated provisioner.ResourceReplacement resourceReplacements = 5;


I believe this is backwards-compatible since repeated fields are effectively optional if nil length.

dannykopping · 2025-05-08T08:42:12Z

enterprise/coderd/prebuilds/reconcile.go

+	// nolint:gocritic // Necessary to query all the required data.
+	ctx = dbauthz.AsSystemRestricted(ctx)
+	// Since this may be called in a fire-and-forget fashion, we need to give up at some point.
+	trackCtx, trackCancel := context.WithTimeout(ctx, time.Minute)


This is a best-effort attempt to warn operators of this situation; it's ok if it times out, we'll get a log to trace this with.

dannykopping · 2025-05-08T08:49:33Z

provisioner/terraform/executor.go

@@ -258,7 +258,7 @@ func getStateFilePath(workdir string) string {
 }

 // revive:disable-next-line:flag-parameter
-func (e *executor) plan(ctx, killCtx context.Context, env, vars []string, logr logSink, destroy bool) (*proto.PlanComplete, error) {
+func (e *executor) plan(ctx, killCtx context.Context, env, vars []string, logr logSink, metadata *proto.Metadata) (*proto.PlanComplete, error) {


This is the meat of the feature; everything else is just plumbing between system and user eyeball.

dannykopping · 2025-05-08T08:51:26Z

provisioner/terraform/executor.go

+			level := proto.LogLevel_INFO
+
+			// Terraform indicates that a resource will be deleted and recreated by showing the change along with this substring.
+			if bytes.Contains(line, []byte("# forces replacement")) {


A bit flimsy; open to other ideas.
In any case, this is just sugar. The fact that the plan, with all its drift details, are shown will be sufficient. Highlighting the lines is just a courtesy to the user.

dannykopping · 2025-05-08T12:30:02Z

coderd/prebuilds/api.go

+
+	// TrackResourceReplacement handles a pathological situation whereby a terraform resource is replaced due to drift,
+	// which can obviate the whole point of pre-provisioning a prebuilt workspace.
+	// See more detail at https://coder.com/docs/admin/templates/extending-templates/prebuilt-workspaces.md#preventing-resource-replacement.


Depends on #17580

dannykopping · 2025-05-08T12:30:19Z

coderd/database/migrations/000320_resource_replacements_notification.up.sql

+If you’re using prebuilds to speed up provisioning, unexpected replacements will slow down
+workspace startup—even when claiming a prebuilt environment.
+
+For tips on preventing replacements and improving claim performance, see [this guide](https://coder.com/docs/admin/templates/extending-templates/prebuilt-workspaces.md#preventing-resource-replacement).


Depends on #17580

dannykopping · 2025-05-08T12:34:26Z

enterprise/coderd/prebuilds/metricscollector.go

@@ -13,43 +16,65 @@ import (
 	"github.com/coder/coder/v2/coderd/prebuilds"
 )

+const (


Muddies the purpose of the PR a bit, but it was a worthwhile driveby refactoring given that we're adding a new metric (MetricResourceReplacementsCount) and we need to check for its value in a test.

Signed-off-by: Danny Kopping <[email protected]>

…replacement(s) Signed-off-by: Danny Kopping <[email protected]>

Signed-off-by: Danny Kopping <[email protected]>

also a test that was broken from an earlier fix Signed-off-by: Danny Kopping <[email protected]>

Signed-off-by: Danny Kopping <[email protected]>

evgeniy-scherbina · 2025-05-08T14:36:19Z

coderd/database/queries/prebuilds.sql

@@ -42,6 +42,11 @@ FROM templates t
 WHERE tvp.desired_instances IS NOT NULL -- Consider only presets that have a prebuild configuration.
 	AND (t.id = sqlc.narg('template_id')::uuid OR sqlc.narg('template_id') IS NULL);

+-- name: GetTemplatePresetsByID :one


should the name be singular instead of plural?

move it to presets.sql?

there are similar query: -- name: GetPresetByID :one, you may consider reuse

Thanks! I missed that.

spikecurtis · 2025-05-09T09:13:55Z

provisionerd/proto/provisionerd.proto

@@ -75,6 +75,7 @@ message CompletedJob {
        repeated provisioner.Resource resources = 2;
        repeated provisioner.Timing timings = 3;
        repeated provisioner.Module modules = 4;
+        repeated provisioner.ResourceReplacement resourceReplacements = 5;


We discussed on Slack, but these changes need to bump the minor version --- unless it was already bumped since the last release, in which case you need to update the comment describing the version bump, but don't need to bump it twice in a single release.

There's also prebuild_claim_for_user_id in provisionersdk/proto/provisioner.proto, but I'm trying to see if I can remove this in favour of passing whether the workspace-prebuilds experiment is used down, since it's effectively just being used as a control flag for that in provisioner/terraform/executor.go.

github-actions bot assigned dannykopping Apr 25, 2025

dannykopping commented Apr 25, 2025

View reviewed changes

SasSwart mentioned this pull request May 2, 2025

feat: reinitialize agents when a prebuilt workspace is claimed #17475

Open

github-actions bot added the stale This issue is like stale bread. label May 6, 2025

dannykopping force-pushed the dk/logreplacements branch 3 times, most recently from 4859410 to 634082b Compare May 7, 2025 21:06

github-actions bot removed the stale This issue is like stale bread. label May 8, 2025

dannykopping force-pushed the dk/logreplacements branch from 0322146 to 13168f4 Compare May 8, 2025 12:38

dannykopping commented May 8, 2025

View reviewed changes

dannykopping requested review from mafredri, SasSwart and evgeniy-scherbina May 8, 2025 12:39

dannykopping changed the title ~~WIP! feat: log resource replacements when claiming a prebuilt workspace~~ feat: track resource replacements when claiming a prebuilt workspace May 8, 2025

dannykopping added 14 commits May 8, 2025 14:48

feat: log resource replacements

b32923a

Signed-off-by: Danny Kopping <[email protected]>

feat: show terraform state drift diff in build logs

0b0830f

Signed-off-by: Danny Kopping <[email protected]>

feat: only highlight lines which mention replacement

256395a

Signed-off-by: Danny Kopping <[email protected]>

feat: notify template admins when prebuild claim results in resource …

61ef61a

…replacement(s) Signed-off-by: Danny Kopping <[email protected]>

chore: appease linter

a66559f

Signed-off-by: Danny Kopping <[email protected]>

chore: fix notifications test

222892b

also a test that was broken from an earlier fix Signed-off-by: Danny Kopping <[email protected]>

fix: don't panic

f34e011

Signed-off-by: Danny Kopping <[email protected]>

fix: renaming type

5168c01

Signed-off-by: Danny Kopping <[email protected]>

chore: updating migration numbers

41e5e0c

Signed-off-by: Danny Kopping <[email protected]>

chore: minor touch-ups

b29e8fa

Signed-off-by: Danny Kopping <[email protected]>

feat: add resource replacements metric

b31ed5e

Signed-off-by: Danny Kopping <[email protected]>

feat: add resource replacement notification

adf98d2

Signed-off-by: Danny Kopping <[email protected]>

make lint; make fmt

f24aef0

Signed-off-by: Danny Kopping <[email protected]>

chore: adding tests

70f9a53

Signed-off-by: Danny Kopping <[email protected]>

dannykopping force-pushed the dk/logreplacements branch from 13168f4 to 70f9a53 Compare May 8, 2025 12:48

dannykopping marked this pull request as ready for review May 8, 2025 12:59

dannykopping requested review from spikecurtis and johnstcn as code owners May 8, 2025 12:59

evgeniy-scherbina reviewed May 8, 2025

View reviewed changes

spikecurtis requested changes May 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: track resource replacements when claiming a prebuilt workspace #17571

feat: track resource replacements when claiming a prebuilt workspace #17571

dannykopping commented Apr 25, 2025 •

edited

Loading

dannykopping Apr 25, 2025 •

edited

Loading

dannykopping May 6, 2025

dannykopping May 8, 2025

dannykopping May 8, 2025

dannykopping May 8, 2025

dannykopping May 8, 2025

dannykopping May 8, 2025

dannykopping May 8, 2025

evgeniy-scherbina May 8, 2025

dannykopping May 8, 2025

spikecurtis May 9, 2025

dannykopping May 9, 2025 •

edited

Loading

feat: track resource replacements when claiming a prebuilt workspace #17571

Are you sure you want to change the base?

feat: track resource replacements when claiming a prebuilt workspace #17571

Conversation

dannykopping commented Apr 25, 2025 • edited Loading

dannykopping Apr 25, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannykopping May 9, 2025 • edited Loading

Choose a reason for hiding this comment

dannykopping commented Apr 25, 2025 •

edited

Loading

dannykopping Apr 25, 2025 •

edited

Loading

dannykopping May 9, 2025 •

edited

Loading