Daily Workflow Audit — 2026-05-22

Audited 78 runs from the last 24 hours (6.1h total duration, $21.75 spend, 42.8M tokens). Success rate 78.2% — a slight rebound from yesterday's 75.6% but still well below the 85–92% baseline from earlier in the week. Two issues dominate today's findings: the Codex engine has now failed 100% for three consecutive days with no fix landed, and smoke test errors jumped 9x (63 vs 7 yesterday) in a single 5-minute window.

Summary

Engine mix: copilot 42, claude 11, codex 3, null 22 (smoke-test harness runs without engine attribution).

Critical Issues

🔴 Codex engine: 100% failure for 3rd consecutive day

• All 3 codex runs failed: AI Moderator (×2, runs §26312070679, §26312080745) and Daily Cache Strategy Analyzer (§26306461300).
• Root cause: codex-cli POSTs model=gpt-5.5 to api-proxy:10000/responses, the proxy returns HTTP 400 invalid_request_error, codex-harness exhausts all 3 retries (exitCode=1).
• The api-proxy /v1/models advertises gpt-5.5 as available — so the failure is likely a request-body / wire-format mismatch on the /responses endpoint, not a missing model.
• Yesterday's recommendation to pin to a known-good model has not yet shipped.

🔴 Smoke test error surge — 9x increase

• 63 errors today vs 7 yesterday. Concentrated in 14 smoke runs all started in a 5-minute window (~17:59 UTC) — strongly suggests a single environment change or regression at the smoke-batch trigger.
• Worst offenders: Smoke Claude 6 errors (§26303796120), Smoke Gemini 5 (§26303796069), Smoke Copilot ARM64 5 (§26303796192), Smoke Agent variants 3 each, Smoke Project 3, Smoke CI 3.

Trend Charts

Success rate has now declined four audit cycles in a row (92.8% → 88.9% → 85.9% → 75.6% → 78.2%), with raw failure counts climbing from 6 to 14. Today's mild rebound is driven by sheer run volume rather than fewer failures — absolute failure count is at a weekly high.

Cost dipped from yesterday's peak driven by recovery of the Daily Safe Output Tool Optimizer ($8.54 → $2.67), but Daily Code Metrics jumped from $3.94 to $6.46 (+64%) and the self-triggered Failure Investigator burned $4.88. Token usage rose 19% while cost held flat.

Recommendations

1. Urgent — Fix codex engine: Pin AI Moderator / Daily Cache Strategy Analyzer / Smoke Codex to a model that actually works against the /responses endpoint (try gpt-5-codex or gpt-5). Verify request body shape matches what the proxy expects. This is now a 3-day-old regression with no progress.
2. Urgent — Investigate smoke surge: All 14 erroring smoke runs started in the same 5-minute window. Check what shipped to main / the test image between 17:00–18:00 UTC. Start with Smoke Claude run §26303796120 (highest at 6 errors).
3. Investigate Daily Code Metrics cost growth: $6.46 (+64% day-over-day) at 30.8M tokens / 19.6 minutes / 53 turns. Inspect tool-call sequence for loops or expanded scope.
4. Stabilize Linter Miner: 0-89 turn variance with 2/3 errors. Add a turn budget or guard against runaway loops.
5. Audit Copilot PR Prompt Pattern Analysis allowed-domains: 36% block rate (19/53) — either tighten the workflow or extend the allowlist for legitimate calls.
6. Carry-over: Smoke CI continues intermittent errors (3/11 today); Chaos PR Fuzzer 54 blocks (likely by design — verify).

Resolved / Improved

• ✅ Daily Safe Output Tool Optimizer cost recovered ($8.54 → $2.67)
• ✅ Auto-Triage Issues fully recovered (2/2 success today)
• ✅ PR Sous Chef drift no longer top signal (eclipsed by Linter Miner)

References:

• §26312070679 — AI Moderator codex failure
• §26303796120 — Smoke Claude 6 errors
• §26305648141 — Linter Miner 89-turn run

Generated by 🔍 Agentic Workflow Audit Agent · ● 16M · ◷

• expires on May 23, 2026, 9:49 PM UTC