CODAGT-463: Complete the existing V1 field codec

Direction for approval

Outcome

Gemini tool-use steps and other chat messages persist successfully when any codersdk.ChatMessagePart string contains NUL or any raw JSON field contains PostgreSQL-invalid Unicode escapes.

Recommended direction

Make the smallest extension of the existing V1 design:

1. Keep encodeNulInParts and decodeNulInParts as direct, handwritten traversal.
2. Add every current string-bearing field omitted from that traversal.
3. Keep the existing semantic json.RawMessage decode, walk, and re-marshal implementation, but activate it for NUL, escaped or literal U+E000, and unpaired surrogate escapes.
4. Add one reflection-driven behavioral test that automatically exercises every current and future string-bearing ChatMessagePart field through MarshalParts and ParseContent.

There is no production registry, reflection, code generation, whole-document transform, content-version change, or migration.

Reason

The production fix is ordinary field assignment and one loop for ParsedCommands. The test supplies the durability: a new direct string, raw JSON field, or nested string container is automatically populated with a probe value and fails by field name if the production traversal does not encode and decode it.

This retains the existing, understood V1 boundary and limits lexical JSON logic to detection. Actual raw JSON transformation continues to use encoding/json rather than a handwritten mutating parser.

Observable end state

• All current direct string fields round-trip NUL and natural sentinel values.
• Args, Result, and ProviderMetadata encode NUL in keys and values.
• ParsedCommands [][]string is encoded without mutating caller-owned slices.
• Lone surrogate escapes in raw JSON become U+FFFD before PostgreSQL.
• Valid surrogate pairs retain their semantic value.
• Adding a future string-bearing field causes a test failure naming the field and the required production functions to update.
• Adding an unsupported field shape causes an actionable test failure instead of being skipped.

Decisions and constraints

• ContentVersionV1 remains current.
• Historical V1 sentinel ambiguity is accepted because a new content version is explicitly out of scope.
• Existing REST and SSE serialization must not change.
• The encoder must not mutate the caller's parts or nested slices.
• Raw JSON may be reformatted only when it contains data requiring transformation. Clean raw JSON returns unchanged.
• The unavailable customer payload prevents proving its exact offending field. The regression covers the demonstrated Gemini metadata path and the separate raw-surrogate path.

Ruled-out directions

• Content V2: explicitly rejected.
• Whole-document lexical transform: rejected because it widens V1 decoding and adds a larger custom JSON parser surface.
• Production handler registry: rejected as more machinery than direct assignments.
• Generics-based registry: rejected as harder to read without removing explicit field maintenance.
• Code generation: rejected as unnecessary once behavior is mechanically tested.
• Runtime reflection or persistence tags: rejected as production complexity for a small traversal.

Implementation detail

1. Complete the handwritten traversal

Update encodeNulInParts and decodeNulInParts in coderd/x/chatd/chatprompt/chatprompt.go.

Apply the string codec to every direct or named string field on codersdk.ChatMessagePart, including the discriminant and all tool, source, file, context-file, and skill strings.

Apply the raw JSON codec to:

• Args
• Result
• ProviderMetadata

Handle ParsedCommands [][]string with typed nested loops. Encoding must clone the outer and inner slices before changing strings. Decoding may operate in place after json.Unmarshal.

Leave scalar booleans, integers, UUID values, timestamps, and []byte unchanged.

2. Extend raw JSON candidate detection

Keep encodeNulInJSON based on json.Unmarshal, recursive semantic transformation, and json.Marshal.

Replace its current incomplete fast-path check with a read-only detector that recognizes actual JSON string content requiring the slow path:

• \u0000
• escaped or literal U+E000
• lone high or low surrogate escapes

The detector must account for JSON string boundaries and escaped backslashes. It must not mutate JSON. Valid adjacent surrogate pairs alone do not require transformation.

When the detector reports a candidate:

• json.Unmarshal supplies the semantic value and normalizes lone surrogates to U+FFFD;
• the existing recursive value walker encodes NUL and U+E000 in keys and values;
• json.Marshal produces PostgreSQL-safe JSON.

Keep decodeNulInJSON unchanged except for applying it to ProviderMetadata through the completed part traversal.

3. Add automatic field coverage

Add TestChatMessagePartNULCoverage in the internal chatprompt test package.

For each reflected ChatMessagePart field, run a subtest named after that field:

• Direct or named strings receive a value containing NUL, U+E000, and U+E001.
• json.RawMessage receives valid nested JSON with NUL and natural sentinel values in keys and values.
• Slices and arrays recursively containing strings receive populated nested values, covering ParsedCommands.
• Known non-string representations such as booleans, numbers, UUIDs, timestamps, and []byte are recognized as safe and skipped.
• Any unknown shape fails with the full field name and type.

Each transforming subtest must:

1. Call MarshalParts on a value with only that field populated.
2. Assert the persisted JSON contains no \u0000.
3. Call ParseContent and compare the named field with its original value.
4. Assert the input value and nested containers were not mutated.

A missing traversal assignment must fail in the subtest for that field. An unsupported future shape must fail with guidance equivalent to:

ChatMessagePart.<Field> has unsupported persistence coverage type <Type>.
If it can marshal arbitrary strings, extend the test probe and
encodeNulInParts/decodeNulInParts. Otherwise classify its concrete type as safe.

This test has no production field-name list and no parallel test registry to maintain.

4. Add focused regressions

Extend chatprompt tests with:

• actual fantasygoogle.ReasoningMetadata whose signature contains binary NUL, serialized through ProviderMetadata;
• NUL in raw JSON object keys and values;
• lone high and low surrogate escapes becoming U+FFFD;
• valid surrogate pairs preserving their semantic character;
• escaped literal text such as \\u0000 and \\uD800 remaining literal text;
• clean raw JSON returning unchanged;
• PostgreSQL jsonb insertion and read-back for Gemini metadata and raw tool arguments.

5. Verification

Run:

go test ./coderd/x/chatd/chatprompt -count=1
go test ./coderd/x/chatd/chatstate ./coderd/database -count=1
make fmt
make lint
make pre-commit

Verify the original failure path: a Gemini-shaped tool-use step containing a binary thought signature marshals, inserts through InsertChatMessages, and parses successfully.

Invariants

• V1 remains the only current write format.
• Production traversal stays explicit and local.
• Every present and future string-bearing field is behaviorally covered by reflection-driven tests.
• Unknown future field shapes fail rather than being silently skipped.
• NUL and unpaired surrogate escapes cannot reach PostgreSQL JSONB through ChatMessagePart.
• Valid data round-trips once through the existing V1 codec.
• Caller-owned data is not mutated.
• No unrelated code, schema, API, or generated artifact changes.

Post-review revision (2026-08-26)

Applied after a fresh critique and a second independent review:

• The field policy now lives in one production table, partNulFields, mapping every string-bearing ChatMessagePart field to nulEncode or nulReject with pointer accessors shared by encode, decode, and validate. Removing an entry fails TestChatMessagePartNULCoverage (proven by mutation).
• MarshalParts wraps rejection as "validate chat message parts" and marshal failure as "marshal chat message parts".
• The coverage test uses testify and one shared probe walker; probe strings are named constants.
• A reviewer claim that the RawMessage natural-sentinel subtests were vacuous was rejected with evidence: the probes contain literal, invisible U+E000 U+E001 bytes (od-verified), and a no-op-encoder mutation fails those subtests.

Residual limitations for the PR body

• The fix stops the pq error, but Gemini thought signatures with invalid UTF-8 (nearly all binary blobs) are corrupted to U+FFFD by json.Marshal upstream of this codec. NUL is the only byte class preserved. Filed as CODAGT-986; root fix is fantasy's Signature string becoming []byte.
• NUL now survives decode into memory; chat debug step inserts fail non-fatally when prompt text contains NUL, silently dropping debug data. Filed as CODAGT-987.
• Lone surrogate escapes in raw JSON normalize to U+FFFD (PostgreSQL cannot store them); this is lossy by design and covered by tests.
• Historical rows containing natural U+E000 U+E001 pairs in newly covered fields decode as NUL (documented at encodeNulInParts; probability negligible).
• Alternative rejected with evidence: switching the column to the permissive json type would delete the codec but break jsonb_array_elements, jsonb_typeof, @>, and content search in chats.sql (verified against Postgres 13).