feat(analyze): detect closed vocabulary drift - #4160
AnkitPorwal04 wants to merge 1 commit into
Conversation
Teach the analysis command to inventory and compare repeated named finite sets while distinguishing explicit subsets. Assisted-by: OpenAI Codex (GPT-5, autonomous)
There was a problem hiding this comment.
Pull request overview
Adds closed-vocabulary drift detection to /speckit.analyze.
Changes:
- Inventories and compares repeated finite sets.
- Reports mismatches and permits declared subsets.
- Adds a template contract test.
Show a summary per file
| File | Description |
|---|---|
templates/commands/analyze.md |
Adds detection guidance and metrics. |
tests/test_analyze_closed_vocabulary.py |
Verifies required guidance exists. |
Review details
馃挕 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 2/2 changed files
- Comments generated: 4
- Review effort level: Balanced
| - Task ordering contradictions (e.g., integration tasks before foundational setup tasks without dependency note) | ||
| - Conflicting requirements (e.g., one requires Next.js while other specifies Vue) | ||
|
|
||
| #### G. Closed Vocabulary Consistency |
| - **User story/action inventory**: Discrete user actions with acceptance criteria | ||
| - **Task coverage mapping**: Map each task to one or more requirements or stories (inference by keyword / explicit reference patterns like IDs or key phrases) | ||
| - **Constitution rule set**: Extract principle names and MUST/SHOULD normative statements | ||
| - **Closed-vocabulary inventory**: Record explicitly named finite sets (for example statuses, roles, categories, or enum-like value lists), their members, and each location where they are declared |
| - Coverage % (requirements with >=1 task) | ||
| - Ambiguity Count | ||
| - Duplication Count | ||
| - Closed Vocabulary Mismatch Count |
| - Find explicitly named finite sets that are enumerated in more than one artifact or section | ||
| - Compare repeated declarations after normalizing case, quoting, and presentation order while preserving the actual member names | ||
| - Flag declarations of the same named set when members are missing, added, or renamed, and report both locations plus the symmetric difference | ||
| - Treat an explicitly identified subset as intentional when the artifact states the subset relationship; otherwise report the divergence as a warning rather than assuming it is an error |
mnriem
left a comment
There was a problem hiding this comment.
Please address Copilot feedback
|
Author of #4106 and of the closed-vocabulary preset. The summary here covers the divergence side well. Two things from building and mechanically testing the same check, both about the honesty of an empty result rather than the divergence metric itself. Coverage, not just the mismatch count. Report declarations inspected against declarations recognised on every run, including zero-divergence ones. A clean result and a run that recognised nothing produce identical output otherwise. Building the preset, my first extractor reported zero over a corpus that held a real divergence in plain sight, because it parsed 4 of 112 declarations. The zero was only honest once the recognised count sat beside it. Pin the unit of declarations inspected. Lines, sentences and candidate fragments are three defensible readings, and since this pass is prose a model executes, two runs on the same spec report different coverage and both are right. One defined unit, or the number is not comparable across runs. |
Summary
Fixes #4106
Testing
AI assistance disclosure
This pull request was implemented, tested, and drafted autonomously by OpenAI Codex (GPT-5) on behalf of @AnkitPorwal04. The human operator selected the overall goal but did not line-by-line author or review the patch before submission.