Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 7e0b879

Browse files
garrytanclaude
garrytan
and
claude
authored
feat: test coverage gate + plan completion audit + auto-verification (v0.11.13.0) (garrytan#428)
* feat: test coverage gate + plan completion audit + auto-verification Three new gates in /ship and /review: 1. Test coverage gate: configurable thresholds (60%/80% default), hard stop below minimum with user override 2. Plan completion audit: discovers plan file, extracts actionable items, cross-references against diff, gates on NOT DONE items 3. Auto-verification: invokes /qa-only inline with plan's verification section, conditional on localhost reachability Also: coverage warning in /review, plan completion data in /retro, shared plan file discovery helper (DRY), ship metrics logging. * chore: regenerate SKILL.md files * chore: bump version and changelog (v0.11.13.0) Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
1 parent 8500136 commit 7e0b879

13 files changed

Lines changed: 946 additions & 9 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,20 @@
11
# Changelog
22

3+
## [0.11.18.0] - 2026-03-24 — Ship With Teeth
4+
5+
`/ship` and `/review` now actually enforce the quality gates they've been talking about. Coverage audit becomes a real gate (not just a diagram), plan completion gets verified against the diff, and verification steps from your plan run automatically.
6+
7+
### Added
8+
9+
- **Test coverage gate in /ship.** AI-assessed coverage below 60% is a hard stop. 60-79% gets a prompt. 80%+ passes. Thresholds are configurable per-project via `## Test Coverage` in CLAUDE.md.
10+
- **Coverage warning in /review.** Low coverage is now flagged prominently before you reach the /ship gate, so you can write tests early.
11+
- **Plan completion audit.** /ship reads your plan file, extracts every actionable item, cross-references against the diff, and shows you a DONE/NOT DONE/PARTIAL/CHANGED checklist. Missing items are a shipping blocker (with override).
12+
- **Plan-aware scope drift detection.** /review's scope drift check now reads the plan file too — not just TODOS.md and PR description.
13+
- **Auto-verification via /qa-only.** /ship reads your plan's verification section and runs /qa-only inline to test it — if a dev server is running on localhost. No server, no problem — it skips gracefully.
14+
- **Shared plan file discovery.** Conversation context first, content-based grep fallback second. Used by plan completion, plan review reports, and verification.
15+
- **Ship metrics logging.** Coverage %, plan completion ratio, and verification results are logged to review JSONL for /retro to track trends.
16+
- **Plan completion in /retro.** Weekly retros now show plan completion rates across shipped branches.
17+
318
## [0.11.17.0] - 2026-03-24 — Cleaner Skill Descriptions + Proactive Opt-Out
419

520
### Changed

‎VERSION‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
0.11.17.0
1+
0.11.18.0

‎retro/SKILL.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -701,6 +701,28 @@ Narrative covering:
701701
- If prior retro exists and has `test_health`: show delta "Test count: {last} → {now} (+{delta})"
702702
- If test ratio < 20%: flag as growth area — "100% test coverage is the goal. Tests make vibe coding safe."
703703

704+
### Plan Completion
705+
Check review JSONL logs for plan completion data from /ship runs this period:
706+
707+
```bash
708+
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)"
709+
cat ~/.gstack/projects/$SLUG/*-reviews.jsonl 2>/dev/null | grep '"skill":"ship"' | grep '"plan_items_total"' || echo "NO_PLAN_DATA"
710+
```
711+
712+
If plan completion data exists within the retro time window:
713+
- Count branches shipped with plans (entries that have `plan_items_total` > 0)
714+
- Compute average completion: sum of `plan_items_done` / sum of `plan_items_total`
715+
- Identify most-skipped item category if data supports it
716+
717+
Output:
718+
```
719+
Plan Completion This Period:
720+
{N} branches shipped with plans
721+
Average completion: {X}% ({done}/{total} items)
722+
```
723+
724+
If no plan data exists, skip this section silently.
725+
704726
### Focus & Highlights
705727
(from Step 8)
706728
- Focus score with interpretation

‎retro/SKILL.md.tmpl‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -460,6 +460,28 @@ Narrative covering:
460460
- If prior retro exists and has `test_health`: show delta "Test count: {last} → {now} (+{delta})"
461461
- If test ratio < 20%: flag as growth area — "100% test coverage is the goal. Tests make vibe coding safe."
462462

463+
### Plan Completion
464+
Check review JSONL logs for plan completion data from /ship runs this period:
465+
466+
```bash
467+
eval "$(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)"
468+
cat ~/.gstack/projects/$SLUG/*-reviews.jsonl 2>/dev/null | grep '"skill":"ship"' | grep '"plan_items_total"' || echo "NO_PLAN_DATA"
469+
```
470+
471+
If plan completion data exists within the retro time window:
472+
- Count branches shipped with plans (entries that have `plan_items_total` > 0)
473+
- Compute average completion: sum of `plan_items_done` / sum of `plan_items_total`
474+
- Identify most-skipped item category if data supports it
475+
476+
Output:
477+
```
478+
Plan Completion This Period:
479+
{N} branches shipped with plans
480+
Average completion: {X}% ({done}/{total} items)
481+
```
482+
483+
If no plan data exists, skip this section silently.
484+
463485
### Focus & Highlights
464486
(from Step 8)
465487
- Focus score with interpretation

‎review/SKILL.md‎

Lines changed: 129 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -322,7 +322,120 @@ Before reviewing code quality, check: **did they build what was requested — no
322322
**If no PR exists:** rely on commit messages and TODOS.md for stated intent — this is the common case since /review runs before /ship creates the PR.
323323
2. Identify the **stated intent** — what was this branch supposed to accomplish?
324324
3. Run `git diff origin/<base>...HEAD --stat` and compare the files changed against the stated intent.
325-
4. Evaluate with skepticism:
325+
326+
### Plan File Discovery
327+
328+
1. **Conversation context (primary):** Check if there is an active plan file in this conversation — Claude Code system messages include plan file paths when in plan mode. Look for references like `~/.claude/plans/*.md` in system messages. If found, use it directly — this is the most reliable signal.
329+
330+
2. **Content-based search (fallback):** If no plan file is referenced in conversation context, search by content:
331+
332+
```bash
333+
BRANCH=$(git branch --show-current 2>/dev/null | tr '/' '-')
334+
REPO=$(basename "$(git rev-parse --show-toplevel 2>/dev/null)")
335+
# Try branch name match first (most specific)
336+
PLAN=$(ls -t ~/.claude/plans/*.md 2>/dev/null | xargs grep -l "$BRANCH" 2>/dev/null | head -1)
337+
# Fall back to repo name match
338+
[ -z "$PLAN" ] && PLAN=$(ls -t ~/.claude/plans/*.md 2>/dev/null | xargs grep -l "$REPO" 2>/dev/null | head -1)
339+
# Last resort: most recent plan modified in the last 24 hours
340+
[ -z "$PLAN" ] && PLAN=$(find ~/.claude/plans -name '*.md' -mmin -1440 -maxdepth 1 2>/dev/null | xargs ls -t 2>/dev/null | head -1)
341+
[ -n "$PLAN" ] && echo "PLAN_FILE: $PLAN" || echo "NO_PLAN_FILE"
342+
```
343+
344+
3. **Validation:** If a plan file was found via content-based search (not conversation context), read the first 20 lines and verify it is relevant to the current branch's work. If it appears to be from a different project or feature, treat as "no plan file found."
345+
346+
**Error handling:**
347+
- No plan file found → skip with "No plan file detected — skipping."
348+
- Plan file found but unreadable (permissions, encoding) → skip with "Plan file found but unreadable — skipping."
349+
350+
### Actionable Item Extraction
351+
352+
Read the plan file. Extract every actionable item — anything that describes work to be done. Look for:
353+
354+
- **Checkbox items:** `- [ ] ...` or `- [x] ...`
355+
- **Numbered steps** under implementation headings: "1. Create ...", "2. Add ...", "3. Modify ..."
356+
- **Imperative statements:** "Add X to Y", "Create a Z service", "Modify the W controller"
357+
- **File-level specifications:** "New file: path/to/file.ts", "Modify path/to/existing.rb"
358+
- **Test requirements:** "Test that X", "Add test for Y", "Verify Z"
359+
- **Data model changes:** "Add column X to table Y", "Create migration for Z"
360+
361+
**Ignore:**
362+
- Context/Background sections (`## Context`, `## Background`, `## Problem`)
363+
- Questions and open items (marked with ?, "TBD", "TODO: decide")
364+
- Review report sections (`## GSTACK REVIEW REPORT`)
365+
- Explicitly deferred items ("Future:", "Out of scope:", "NOT in scope:", "P2:", "P3:", "P4:")
366+
- CEO Review Decisions sections (these record choices, not work items)
367+
368+
**Cap:** Extract at most 50 items. If the plan has more, note: "Showing top 50 of N plan items — full list in plan file."
369+
370+
**No items found:** If the plan contains no extractable actionable items, skip with: "Plan file contains no actionable items — skipping completion audit."
371+
372+
For each item, note:
373+
- The item text (verbatim or concise summary)
374+
- Its category: CODE | TEST | MIGRATION | CONFIG | DOCS
375+
376+
### Cross-Reference Against Diff
377+
378+
Run `git diff origin/<base>...HEAD` and `git log origin/<base>..HEAD --oneline` to understand what was implemented.
379+
380+
For each extracted plan item, check the diff and classify:
381+
382+
- **DONE** — Clear evidence in the diff that this item was implemented. Cite the specific file(s) changed.
383+
- **PARTIAL** — Some work toward this item exists in the diff but it's incomplete (e.g., model created but controller missing, function exists but edge cases not handled).
384+
- **NOT DONE** — No evidence in the diff that this item was addressed.
385+
- **CHANGED** — The item was implemented using a different approach than the plan described, but the same goal is achieved. Note the difference.
386+
387+
**Be conservative with DONE** — require clear evidence in the diff. A file being touched is not enough; the specific functionality described must be present.
388+
**Be generous with CHANGED** — if the goal is met by different means, that counts as addressed.
389+
390+
### Output Format
391+
392+
```
393+
PLAN COMPLETION AUDIT
394+
═══════════════════════════════
395+
Plan: {plan file path}
396+
397+
## Implementation Items
398+
[DONE] Create UserService — src/services/user_service.rb (+142 lines)
399+
[PARTIAL] Add validation — model validates but missing controller checks
400+
[NOT DONE] Add caching layer — no cache-related changes in diff
401+
[CHANGED] "Redis queue" → implemented with Sidekiq instead
402+
403+
## Test Items
404+
[DONE] Unit tests for UserService — test/services/user_service_test.rb
405+
[NOT DONE] E2E test for signup flow
406+
407+
## Migration Items
408+
[DONE] Create users table — db/migrate/20240315_create_users.rb
409+
410+
─────────────────────────────────
411+
COMPLETION: 4/7 DONE, 1 PARTIAL, 1 NOT DONE, 1 CHANGED
412+
─────────────────────────────────
413+
```
414+
415+
### Integration with Scope Drift Detection
416+
417+
The plan completion results augment the existing Scope Drift Detection. If a plan file is found:
418+
419+
- **NOT DONE items** become additional evidence for **MISSING REQUIREMENTS** in the scope drift report.
420+
- **Items in the diff that don't match any plan item** become evidence for **SCOPE CREEP** detection.
421+
422+
This is **INFORMATIONAL** — does not block the review (consistent with existing scope drift behavior).
423+
424+
Update the scope drift output to include plan file context:
425+
426+
```
427+
Scope Check: [CLEAN / DRIFT DETECTED / REQUIREMENTS MISSING]
428+
Intent: <from plan file — 1-line summary>
429+
Plan: <plan file path>
430+
Delivered: <1-line summary of what the diff actually does>
431+
Plan items: N DONE, M PARTIAL, K NOT DONE
432+
[If NOT DONE: list each missing item]
433+
[If scope creep: list each out-of-scope change not in the plan]
434+
```
435+
436+
**No plan file found:** Fall back to existing scope drift behavior (check TODOS.md and PR description only).
437+
438+
4. Evaluate with skepticism (incorporating plan completion results if available):
326439

327440
**SCOPE CREEP detection:**
328441
- Files changed that are unrelated to the stated intent
@@ -631,6 +744,21 @@ If no test framework detected → include gaps as INFORMATIONAL findings only, n
631744

632745
**Diff is test-only changes:** Skip Step 4.75 entirely: "No new application code paths to audit."
633746

747+
### Coverage Warning
748+
749+
After producing the coverage diagram, check the coverage percentage. Read CLAUDE.md for a `## Test Coverage` section with a `Minimum:` field. If not found, use default: 60%.
750+
751+
If coverage is below the minimum threshold, output a prominent warning **before** the regular review findings:
752+
753+
```
754+
⚠️ COVERAGE WARNING: AI-assessed coverage is {X}%. {N} code paths untested.
755+
Consider writing tests before running /ship.
756+
```
757+
758+
This is INFORMATIONAL — does not block /review. But it makes low coverage visible early so the developer can address it before reaching the /ship coverage gate.
759+
760+
If coverage percentage cannot be determined, skip the warning silently.
761+
634762
This step subsumes the "Test Gaps" category from Pass 2 — do not duplicate findings between the checklist Test Gaps item and this coverage diagram. Include any coverage gaps alongside the findings from Step 4 and Step 4.5. They follow the same Fix-First flow — gaps are INFORMATIONAL findings.
635763

636764
---

‎review/SKILL.md.tmpl‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,10 @@ Before reviewing code quality, check: **did they build what was requested — no
4646
**If no PR exists:** rely on commit messages and TODOS.md for stated intent — this is the common case since /review runs before /ship creates the PR.
4747
2. Identify the **stated intent** — what was this branch supposed to accomplish?
4848
3. Run `git diff origin/<base>...HEAD --stat` and compare the files changed against the stated intent.
49-
4. Evaluate with skepticism:
49+
50+
{{PLAN_COMPLETION_AUDIT_REVIEW}}
51+
52+
4. Evaluate with skepticism (incorporating plan completion results if available):
5053

5154
**SCOPE CREEP detection:**
5255
- Files changed that are unrelated to the stated intent

‎scripts/resolvers/index.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ import { generateTestFailureTriage } from './preamble';
1111
import { generateCommandReference, generateSnapshotFlags, generateBrowseSetup } from './browse';
1212
import { generateDesignMethodology, generateDesignHardRules, generateDesignOutsideVoices, generateDesignReviewLite, generateDesignSketch } from './design';
1313
import { generateTestBootstrap, generateTestCoverageAuditPlan, generateTestCoverageAuditShip, generateTestCoverageAuditReview } from './testing';
14-
import { generateReviewDashboard, generatePlanFileReviewReport, generateSpecReviewLoop, generateBenefitsFrom, generateCodexSecondOpinion, generateAdversarialStep, generateCodexPlanReview } from './review';
14+
import { generateReviewDashboard, generatePlanFileReviewReport, generateSpecReviewLoop, generateBenefitsFrom, generateCodexSecondOpinion, generateAdversarialStep, generateCodexPlanReview, generatePlanCompletionAuditShip, generatePlanCompletionAuditReview, generatePlanVerificationExec } from './review';
1515
import { generateSlugEval, generateSlugSetup, generateBaseBranchDetect, generateDeployBootstrap, generateQAMethodology } from './utility';
1616

1717
export const RESOLVERS: Record<string, (ctx: TemplateContext) => string> = {
@@ -41,4 +41,7 @@ export const RESOLVERS: Record<string, (ctx: TemplateContext) => string> = {
4141
ADVERSARIAL_STEP: generateAdversarialStep,
4242
DEPLOY_BOOTSTRAP: generateDeployBootstrap,
4343
CODEX_PLAN_REVIEW: generateCodexPlanReview,
44+
PLAN_COMPLETION_AUDIT_SHIP: generatePlanCompletionAuditShip,
45+
PLAN_COMPLETION_AUDIT_REVIEW: generatePlanCompletionAuditReview,
46+
PLAN_VERIFICATION_EXEC: generatePlanVerificationExec,
4447
};

0 commit comments

Comments
 (0)