Silent parameter-value loss when a template version's module cache is absent #29099

🤖 This issue was filed by Coder Agents on behalf of @Emyrk.

Summary

When template_version_terraform_values.cached_module_files is NULL, the dynamic parameter renderer cannot resolve Terraform modules. coder/preview reports this as a warning, not an error, so the render succeeds with a reduced parameter set. ResolveParameters then deletes any previously stored value whose parameter is absent from that set, and the build proceeds with the module default.

For a Kubernetes template where the home volume size is a module-declared coder_parameter, this replaces the PVC. Observed in production: {"storage":"1000Gi"} -> {"storage":"50Gi"} (forces replacement) - ~1 TB of user data destroyed, with no error surfaced to the user or the operator.

This is a pre-existing latent issue. The 2.35.7 / 2.36.4 / 2.34.9 / 2.37.0 remediation for GHSA-vx42-ghc9-gw65 nulls cached_module_files for archives ingested during the incident window - correct and necessary - which surfaced the downstream behaviour at scale.

Mechanism

1. coderd/database/migrations/000589_delete_tf_modules.up.sql (and DeleteCachedModuleFilesCreatedBetween via dbpurge) sets cached_module_files = NULL and deletes the files row.
2. coderd/dynamicparameters/render.go mounts the .terraform/modules overlay FS only when r.terraformValues.CachedModuleFiles.Valid. With NULL, module sources are unavailable and no error is raised.
3. coder/preview warnings.go reports this at hcl.DiagWarning with code types.DiagnosticModuleNotLoaded: "Module not loaded. Did you run terraform init? ... This module will be ignored."
4. ResolveParameters gates on diags.HasErrors() at both render calls. A warning passes, so the reduced parameter set is treated as authoritative, and the final loop discards stored values:

for k := range values {
    if _, ok := parameterNames[k]; !ok {
        delete(values, k)
    }
}

5. wsbuilder.getParameters() runs on every transition with no stop/delete guard. coder update stops first (cli/update.go, bug: template upgrade fails on claimed prebuilt workspace #17840), so the stop build persists the truncated set. The subsequent start build reads it via getLastBuildParameters() - the original value is gone even when the target version's cache is healthy.

Net effect: a warning-level diagnostic silently and irreversibly discards user data, and the loss propagates from an unhealthy version to a healthy one via the intermediate stop build.

Customer mitigation (do this first)

Workspaces whose latest build still holds the full parameter set (i.e. not yet stopped) are not yet damaged. For those, enabling parameter compatibility mode on the template prevents the loss outright: getClassicParameters() reads template_version_parameters - untouched by the purge - and never consults CachedModuleFiles or preview.

• Template -> Settings -> Parameters -> "Use parameter compatibility mode for workspace builds", or PATCH /api/v2/templates/{template} with {"use_classic_parameter_flow": true}.
• Do this before anything triggers a stop build. Autostop schedules, dormancy, coder update, and require_active_version all go through the same wsbuilder path and will persist the truncation. Suspend autostop on affected workspaces if the toggle cannot be applied immediately.
• Back up current values first: GET /api/v2/workspacebuilds/{latest_build}/parameters.

For workspaces where a stop build already truncated the set, the values must be re-supplied explicitly from an earlier build (GET /api/v2/workspacebuilds/{older_build}/parameters) on the next start, e.g. coder start <ws> --parameter home_disk_size=1000Gi. Under compatibility mode this is reliable; under dynamic flow with a NULL cache the --parameter value is dropped by the same loop. If the parameter is mutable = false, ValidateResolve will reject the correction because the stop build recorded a differing previous value - those need a direct workspace_build_parameters fix.

Pair with storage class reclaimPolicy: Retain as an unconditional backstop; it converts data loss into a re-bindable PV, for PVs provisioned after the change.

Workarounds that do not work: disable_module_cache = true (gates only the provisionerd build path in provisionerdserver.go; the renderer reads CachedModuleFiles directly), and restoring the deleted files rows (those are the artifacts the advisory removed).

Proposed fix

Both changes belong in one PR - same failure path, and the tests only make sense as a pair.

1. Make the discard loop completeness-aware (coderd/dynamicparameters/resolver.go)

The delete loop is correct for a complete render and wrong for an incomplete one. Inspect the second render's diags for previewtypes.DiagnosticExtra{Code: types.DiagnosticModuleNotLoaded} (and any future warning indicating an incomplete graph) and gate on it:

incomplete := hasIncompleteGraph(diags)
var dropped []string
for k, v := range values {
    if _, ok := parameterNames[k]; ok {
        continue
    }
    if incomplete && v.Source != sourceDefault {
        dropped = append(dropped, k)
        continue
    }
    delete(values, k)
}
if len(dropped) > 0 {
    // actionable error naming the parameters, the cause, and the remedies
}

Properties: dropping from a complete render is unchanged; a value that only ever came from a default is still discardable. The condition only fires when real user input (sourcePrevious, sourceBuild, sourcePreset) would be destroyed. The distinction is the completeness of the render, not the diff itself.

2. Do not narrow the parameter set on non-start transitions (coderd/wsbuilder/wsbuilder.go)

A stop/delete build has no legitimate reason to shrink the stored set - the previous values are still the correct provisioner inputs for that build. This breaks the propagation chain where an intermediate stop build carries the loss into a healthy version, and independently fixes the coder update case.

Tests

• resolver_test.go: stub renderer emitting module_not_loaded plus a previous value absent from the output.
• wsbuilder_test.go: stop build preserves the full set.

Backports

2.35.x / 2.36.x / 2.37.x, matching where the remediation shipped.

Open decision

On an incomplete render that would drop a real value: fail the build, or preserve the value and proceed?

• Fail: no silent data loss, but every workspace on a purged version becomes unbuildable until the version is re-pushed or compatibility mode is enabled.
• Preserve: workspaces keep building and the apply is still correct (provisionerd re-inits modules itself, so only coderd's render is degraded); the risk is masking a legitimate parameter removal, and only while module warnings are present.

Follow-up (not in scope for the above PR)

Nothing in the UI or logs connects a NULL module cache to a pending destructive plan. Candidates: surface affected template versions in the template UI, warn on workspace update when the current or target version has a NULL cache, include it in health checks, and stop discarding preview's diagnostics in render.go (Logger: slog.New(slog.DiscardHandler)).

Environment

• Coder 2.35.7, provisionerd 1.18, PostgreSQL 14.20
• Kubernetes templates; home volume sized by a module-declared coder_parameter
• 4 template versions affected, all non-active at time of discovery
• 1 confirmed data-loss incident (1000Gi -> 50Gi)