Problem
The vibe_check_mentor has NEVER worked correctly despite ~30 PRs attempting to fix it. I've identified the root cause through systematic analysis.
Test Evidence:
query = "I'm thinking about building a custom HTTP client wrapper for the GitHub API"
result = vibe_check_mentor(query)
# EXPECTED: High-confidence detection of infrastructure_without_implementation
# ACTUAL: Zero detections, corrupted persona responses, generic advice
Actual Output:
{
"immediate_feedback": {
"summary": "No concerning patterns detected - looking good!",
"confidence": 0.0,
"detected_patterns": []
},
"perspectives": {
"senior_engineer": {
"message": "For the exact, http client, a custom:\n\n\n\n",
"confidence": 0.68
}
}
}Root Cause Analysis
Issue #1: Regex Patterns Don't Match Verb Conjugations
The patterns in data/anti_patterns.json use exact word matching with word boundaries \b, which fails to match common English grammar:
Current Pattern (BROKEN):
\b(?:custom|build|implement|create)\s+(?:our\s+own|new|custom)\s+(?:http|client|server|api|wrapper)
Test Proof:
import re
pattern = r'\b(?:custom|build|implement|create)\s+(?:our\s+own|new|custom)\s+(?:http|client|server|api|wrapper)'
text = 'building a custom HTTP client wrapper'
match = re.search(pattern, text, re.IGNORECASE)
print(f'Match found: {match is not None}')
# OUTPUT: Match found: FalseWhy it fails:
- "building" != "build" (word boundary issue)
- "implementing" != "implement"
- "creating" != "create"
- Pattern doesn't account for past tense, gerunds, or other conjugations
Fixed Pattern (WORKS):
\b(?:custom|build(?:ing)?|implement(?:ing)?|creat(?:e|ing))\s+(?:a\s+)?(?:our\s+own|new|custom)\s+(?:http|client|server|api|wrapper)
Verification:
pattern_fixed = r'\b(?:custom|build(?:ing)?|implement(?:ing)?|creat(?:e|ing))\s+(?:a\s+)?(?:our\s+own|new|custom)\s+(?:http|client|server|api|wrapper)'
text = 'building a custom HTTP client wrapper'
match = re.search(pattern_fixed, text, re.IGNORECASE)
print(f'Match found: {match is not None}')
# OUTPUT: Match found: True
print(f'Matched text: {match.group()}')
# OUTPUT: Matched text: building a custom HTTPIssue #2: Corrupted Persona Responses
Persona messages are malformed:
"For the exact, http client, a custom:"
This suggests LLM prompt truncation or template issues in vibe_mentor_enhanced.py's generate_contribution() method.
Issue #3: Low Confidence Scores
When patterns don't match:
- System returns confidence: 0.0 (should be 0.9+ for obvious anti-pattern)
- Falls back to generic advice
- Personas generate gibberish responses
Historical Context
Failed Attempts (~30 PRs):
Why All PRs Failed:
They assumed pattern detection worked and tried to improve response generation. But if patterns never match in the first place, no amount of response engineering helps.
Impact
- Critical functionality broken: vibe_check_mentor can't detect obvious anti-patterns
- False negatives: System says "looking good" for textbook violations
- User trust destroyed: Generic/corrupted advice instead of specific guidance
- Wasted engineering effort: ~30 PRs fixing symptoms, not root cause
Proposed Fix
Phase 1: Fix Regex Patterns (Critical)
Update ALL patterns in data/anti_patterns.json to handle verb conjugations:
Pattern categories needing fixes:
infrastructure_without_implementation - all verb indicatorssymptom_driven_development - verb patternscomplexity_escalation - action verb patternsdocumentation_neglect - planning verb patterns
Regex changes needed:
build → build(?:ing|s)?implement → implement(?:ing|s|ed)?create → creat(?:e|ing|es|ed)planning → plan(?:ning|s|ned)?- etc.
Phase 2: Fix Persona Response Generation
Investigate vibe_mentor_enhanced.py:
generate_contribution() method- MCP sampling prompt construction
- Response validation and truncation handling
Phase 3: Add Regression Tests
Create test suite with real-world examples:
test_cases = [
("building a custom HTTP client", should_detect=True, confidence_min=0.7),
("implementing our own auth system", should_detect=True, confidence_min=0.7),
("creating a custom API wrapper", should_detect=True, confidence_min=0.7),
]Acceptance Criteria
Files Requiring Changes
data/anti_patterns.json - fix ALL regex patternssrc/vibe_check/tools/vibe_mentor_enhanced.py - investigate persona generationtests/ - add comprehensive regression tests
Related Issues/PRs
This is a P0 critical bug - the core functionality of vibe_check_mentor is fundamentally broken and has been since inception. All previous fixes addressed symptoms while the root cause (broken regex patterns) went undetected.
Problem
The vibe_check_mentor has NEVER worked correctly despite ~30 PRs attempting to fix it. I've identified the root cause through systematic analysis.
Test Evidence:
Actual Output:
{ "immediate_feedback": { "summary": "No concerning patterns detected - looking good!", "confidence": 0.0, "detected_patterns": [] }, "perspectives": { "senior_engineer": { "message": "For the exact, http client, a custom:\n\n\n\n", "confidence": 0.68 } } }Root Cause Analysis
Issue #1: Regex Patterns Don't Match Verb Conjugations
The patterns in
data/anti_patterns.jsonuse exact word matching with word boundaries\b, which fails to match common English grammar:Current Pattern (BROKEN):
Test Proof:
Why it fails:
Fixed Pattern (WORKS):
Verification:
Issue #2: Corrupted Persona Responses
Persona messages are malformed:
This suggests LLM prompt truncation or template issues in
vibe_mentor_enhanced.py'sgenerate_contribution()method.Issue #3: Low Confidence Scores
When patterns don't match:
Historical Context
Failed Attempts (~30 PRs):
Why All PRs Failed:
They assumed pattern detection worked and tried to improve response generation. But if patterns never match in the first place, no amount of response engineering helps.
Impact
Proposed Fix
Phase 1: Fix Regex Patterns (Critical)
Update ALL patterns in
data/anti_patterns.jsonto handle verb conjugations:Pattern categories needing fixes:
infrastructure_without_implementation- all verb indicatorssymptom_driven_development- verb patternscomplexity_escalation- action verb patternsdocumentation_neglect- planning verb patternsRegex changes needed:
build→build(?:ing|s)?implement→implement(?:ing|s|ed)?create→creat(?:e|ing|es|ed)planning→plan(?:ning|s|ned)?Phase 2: Fix Persona Response Generation
Investigate
vibe_mentor_enhanced.py:generate_contribution()methodPhase 3: Add Regression Tests
Create test suite with real-world examples:
Acceptance Criteria
Files Requiring Changes
data/anti_patterns.json- fix ALL regex patternssrc/vibe_check/tools/vibe_mentor_enhanced.py- investigate persona generationtests/- add comprehensive regression testsRelated Issues/PRs
This is a P0 critical bug - the core functionality of vibe_check_mentor is fundamentally broken and has been since inception. All previous fixes addressed symptoms while the root cause (broken regex patterns) went undetected.