RFC: Structured Machine Summary for Evaluation Reports

References

• Issue feat: add structured insight summary for evaluations and downstream pattern analysis #442 — Structured insight summary for evaluations and downstream pattern analysis
• PR feat: add structured machine summaries to evaluations #444 — Add structured machine summaries to evaluations

1. Problem

Career-Ops currently produces detailed markdown evaluation reports, but downstream tools need to extract structured insights from those reports.

The current approach relies on markdown structure, headings, regex patterns, and natural-language text. That works for human reading, but it is fragile for automation. Small wording or formatting changes can break pattern analysis, calibration, dashboards, or future reporting tools.

This affects features such as:

• analyze-patterns.mjs
• batch evaluation analysis
• future dashboard insights
• outcome calibration
• comparison of hard stops, soft gaps, risk levels, and decisions across many reports

The user-facing report should remain readable, but the system also needs a stable machine-readable contract inside each saved evaluation report.

2. Proposed Solution

Add a standardized fenced YAML ## Machine Summary block to every evaluation report generated by interactive and batch evaluation modes.

The block should summarize the final evaluation in a strict, machine-readable shape while preserving the existing human-readable markdown report.

Example:

company: "Example Corp"
role: "Software Engineer Intern"
score: 4.2
legitimacy_tier: "strong"
archetype: "backend/platform"
final_decision: "apply"
hard_stops: []
soft_gaps:
  - "Limited production experience with Kubernetes"
top_strengths:
  - "Strong backend API experience"
  - "Relevant full-stack project work"
risk_level: "medium"
confidence: "high"
next_action: "Tailor CV toward backend and cloud deployment experience"

The parser should:

• Prefer the structured Machine Summary block when available.
• Preserve legacy markdown parsing as fallback for older reports.
• Normalize list fields such as hard_stops, soft_gaps, and top_strengths.
• Validate/coerce scalar fields defensively so malformed YAML does not break downstream logic.
• Avoid changing the existing A-G scoring formula.
• Avoid changing user data, application tracking data, or report ownership semantics.

3. Files Affected

Expected system-layer files:

• modes/oferta.md

  • Add the required Machine Summary section to interactive evaluation output.
  • Document expected schema and placement.

• modes/batch.md

  • Ensure batch evaluation reports also emit the same structured block.

• batch/batch-prompt.md

  • Align worker prompt instructions with the same schema.
  • Keep examples consistent across interactive and batch output.

• analyze-patterns.mjs

  • Parse the fenced YAML block.
  • Prefer structured fields over markdown fallback.
  • Keep backward compatibility with existing reports.
  • Normalize list/scalar values safely.
  • Add self-test coverage for parser behavior.

• package.json

  • Add or expose a script for pattern analysis if needed.
  • Add parser dependency only if required.

• test-all.mjs

  • Include parser self-test in the quick validation suite.

• docs/SCRIPTS.md

  • Document usage of the pattern analysis command and relevant flags.

• CLAUDE.md

  • Document that reports may contain a structured Machine Summary section.

Potentially affected but not required:

• DATA_CONTRACT.md
  • Only update if maintainers want the generated reports/* format explicitly documented as containing both human-readable markdown and a structured machine-readable summary.

4. Data Contract Impact

This change does not introduce a new user-owned file.

It modifies the generated content format of reports/*, which are already part of the user layer. The new Machine Summary block mirrors information already present in the human-readable evaluation report.

No update process should modify existing user reports. Existing reports without the block should remain valid through legacy fallback parsing.

System-layer files are updated to produce and parse the new block going forward.

Data Contract summary:

• New user file: no
• New system file: no
• Existing user-layer file format affected: yes, reports/*
• Existing user data modified automatically: no
• Existing scoring behavior changed: no
• Existing report parsing compatibility preserved: yes

5. Phases

Phase 1 — Define the Schema

Create the canonical Machine Summary schema and document:

• required fields
• accepted scalar/list types
• enum guidance where useful
• placement inside reports
• empty list behavior
• confidence/risk wording rules

Acceptance criteria:

• Interactive and batch docs use the same schema.
• Examples are consistent across all touched mode/prompt files.
• No scoring formula changes.

Phase 2 — Emit Machine Summary in Evaluation Reports

Update evaluation prompts so generated reports include the structured block.

Acceptance criteria:

• modes/oferta.md emits the block.
• modes/batch.md emits the block.
• batch/batch-prompt.md emits the same block.
• Human-readable report structure remains intact.

Phase 3 — Parse Structured Summaries

Update analyze-patterns.mjs to prefer the structured block.

Acceptance criteria:

• Parser extracts fenced YAML safely.
• Parser allowlists expected fields.
• Parser normalizes list values.
• Parser preserves valid 0 scores instead of treating them as falsy.
• Parser handles malformed objects/arrays defensively.
• Legacy markdown fallback remains available.

Phase 4 — Tests and Documentation

Add parser self-tests and script documentation.

Acceptance criteria:

• node --check analyze-patterns.mjs passes.
• node analyze-patterns.mjs --self-test passes.
• node test-all.mjs --quick includes the parser self-test.
• docs/SCRIPTS.md documents the pattern analysis command and flags.
• CLAUDE.md references the structured report block where appropriate.

Phase 5 — Follow-Up Improvements

After the base schema is stable, consider follow-up PRs for:

• dashboard consumption
• outcome calibration
• report comparison
• schema versioning
• stricter enum validation
• migration helper for old reports

These should be separate PRs unless maintainers prefer bundling them under this RFC.

6. Non-Goals

This RFC does not propose:

• changing the A-G scoring formula
• changing final decision logic
• adding a database
• adding paid APIs
• auto-editing user profile files
• rewriting existing user reports
• removing legacy markdown parsing
• changing ownership of reports/*
• building a full dashboard in the same PR

7. Backward Compatibility

Existing reports should continue to work.

The parser should follow this priority:

1. Use Machine Summary if present and valid.
2. Fall back to legacy markdown parsing if the block is absent.
3. Avoid throwing on malformed optional fields.
4. Keep generated analysis useful even when some fields are missing.

This allows users with older reports to keep using pattern analysis while newer reports become more reliable for automation.

8. Risks and Mitigations

Risk: LLM-generated YAML may be inconsistent

Mitigation:

• Provide exact examples.
• Keep field names stable.
• Use simple scalar/list values.
• Normalize fields in the parser.
• Add parser self-tests.

Risk: Parser accepts malformed data

Mitigation:

• Allowlist fields.
• Validate scalar fields before assigning.
• Reject object mappings where list strings are expected.
• Preserve legacy fallback behavior.

Risk: Machine Summary duplicates human-readable content

Mitigation:

• Treat the block as a structured extraction layer, not a second explanation.
• Keep the block compact.
• Place it consistently after the score section.

Risk: Data Contract confusion

Mitigation:

• Clarify that reports/* remain user-layer files.
• System files only define how future reports are generated and parsed.
• Existing user reports are not modified automatically.

9. Open Questions

1. Should Machine Summary include a schema_version field now, or wait until the schema changes?
2. Should enum values be strictly enforced for fields like risk_level, confidence, and final_decision?
3. Should malformed Machine Summary blocks fail loudly, warn, or silently fall back to legacy markdown parsing?
4. Should DATA_CONTRACT.md explicitly mention that reports/* may contain machine-readable metadata?
5. Should dashboard and calibration features be opened as separate follow-up issues after this lands?
6. Should this schema also support comparison mode outputs in a later RFC?

10. Suggested PR Breakdown

• PR 1: Add schema to interactive/batch evaluation prompts.
• PR 2: Update parser to consume Machine Summary with fallback.
• PR 3: Add tests and docs.
• PR 4: Optional follow-up for dashboard/calibration consumers.

PR #444 already combines much of PR 1-3 into one implementation. If maintainers prefer smaller review units, the same work can be split according to the phases above.

Links

• feat: add structured insight summary for evaluations and downstream pattern analysis #442
• feat: add structured machine summaries to evaluations #444