ControlLoop — AI-native RCSA & controls testing

ControlLoop runs the Risk & Control Self-Assessment (RCSA) lifecycle for banks — interview, evidence, testing, workpapers, committee pack, remediation — with a human holding every pen that matters.

Live demo: https://controlloop.onrender.com — no signup, fully synthetic data. Start with RB-CTRL-01, or click Run division sweep. (Free-tier hosting: the first load after idle takes about a minute.)

The problem

Controls testing at large banks is a coordination business. Testing is typically outsourced to Big-4-style delivery teams: the bank pays once for the testing and again for the coordination around it — walkthrough scheduling, evidence chasing, workpaper formatting, review cycles, and re-work queues. A quarterly RCSA cycle burns weeks of analyst time on process rather than judgement, and the output is stale before a committee sees it.

ControlLoop automates the mechanical majority of that lifecycle — evidence retrieval, interview structuring, attribute testing, workpaper drafting, aggregation — and keeps humans on exactly the decisions that carry regulatory weight, at explicit approval gates the workflow cannot structurally bypass.

What is automated vs human-gated

Automated (agents + rules)	Human-gated (explicit interrupt() gates)
Control identification + pre-interview profile card (extracted verbatim from the control's own documents)	—
Evidence retrieval, hard-scoped to the control under assessment	—
Adaptive interview questions that react to the owner's answers	—
Draft control rating with a self-explaining rationale + review pack	Gate 1 — approve / override / reject the draft rating
Test planning, attribute-based design testing, D1–D12 / C1–C4 design assessment, parallel OE sample evaluation, C5–C11 monitoring review, OET scripts, workpaper drafting	Gate 2 — review the workpaper; every questionnaire cell editable; PII attestation required
Final rating by transparent rule; remediation actions auto-registered	Reviewer decisions and edits are recorded with provenance
Division/bank-wide sweep, heat map, committee pack	Batch gates A & B — per-control approve / override / reject
Bounded self-tuning from tester feedback (whitelisted parameters only, versioned, revertible)	Owner approval for anything beyond the bounded whitelist

Every rule-based verdict is traceable to the evidence row it came from; every AI-drafted narrative is labelled as such. When the documentation simply doesn't answer a question, the recorded result is Not Determinable — never a fudged Pass.

The lifecycle, end to end

1 · Intake & profile card

Mention a control id (e.g. RB-CTRL-09) and the agent identifies it, maps its division, and posts a pre-interview profile card — objective, owner, frequency, control type, evidence on file — every field extracted verbatim from that control's own documents (app/profile.py), with an invitation to correct anything stale.

2 · Control-scoped retrieval

Retrieval (app/retriever.py) is hard-scoped to the control under assessment: a lexical hit on a similarly-worded sibling control can never surface as evidence. Three backends behind one interface — BM25 (offline demo), Chroma + OpenAI/Azure embeddings, or Azure AI Search — selected by environment variable, never a code change.

3 · Adaptive interview

The interviewer opens by confirming documented facts instead of re-asking them, then reacts to what the owner actually says (app/llm.py): a mentioned issue drills into root cause and recurrence; a stated cause drills into impact and remediation; a system/team change triggers a re-validation question; a frequency that contradicts the documentation is surfaced as a discrepancy. Each follow-up fires once per session.

4 · Draft assessment + Gate 1

The draft rating (Strong / Adequate / Weak) explains itself: the risk and strength signals matched in the evidence, the disclosed threshold rule, and citations. The approval screen carries a review pack — interview answers, key findings, gap/strength signals, evidence list — so the reviewer judges the basis, not just the conclusion. Rejection sends the agent back to re-interview with the reviewer's feedback in the transcript.

5 · Controls-testing engine (app/testing.py)

A multi-agent subgraph runs testing the way a second-line team does:

• Test planner builds the plan from the control's parsed Design_Testing workbook (synonym-tolerant headers and verdict vocabularies; a missing workbook produces an explicit, labelled stub — never an invented result).
• Design testing evaluates each workbook test case as a design attribute (expected vs actual, evidence-completeness score, a coverage self-critique that states which standard test dimensions the attribute set does not address).
• Design assessment (app/design_assessment.py) — the industry-standard design workpaper, agent-populated: C1–C4 control-description review, the D1–D12 design questionnaire, an agent-drafted walkthrough narration, control overview header and fieldwork block. Every rationale quotes the document it is based on; absence of evidence is recorded as Not Determinable with an observation telling the reviewer what to confirm.
• Operating effectiveness fans out one parallel branch per sample unit (LangGraph Send), applies documented deviation rules (a detected-and-cleared exception is the control working, not a deviation), and concludes by transparent thresholds.
• OE assessment (app/oe_assessment.py) wraps the executed tests in the standard OE workpaper anatomy: C5–C11 continuous-monitoring record review, OET test-script rows derived from the tests that actually ran, an agent-drafted basis-for-conclusion, and analytics/automation disclosures.
• Workpaper agent assembles the document: rule-based sections (plan, attribute table, per-attribute test procedures with source citations, sampling stats, thematic analysis) plus an LLM-drafted conclusion narrative explicitly labelled AI-drafted — verify before reliance.

6 · Gate 2 — testing review

The reviewer sees both workpapers with every questionnaire cell editable. Edits are provenance-tagged (agent / edited by reviewer / added by reviewer); reviewer-added questions and OE tests are logged to the improvement loop so recurring ones can be promoted into the standard bank. A no-PII attestation must be confirmed before the workpaper can be approved. Rejection triggers a round-tagged re-test with the feedback applied — prior rounds are retained, never overwritten.

7 · Reassess + remediation

The final rating follows a transparent rule (an OE fail forces Weak; OE exceptions cap an optimistic draft). Findings become tracked remediation actions (app/remediation.py): High/Critical testing exceptions and Weak ratings register P1s; design-assessment and monitoring-review exceptions register P2s — idempotently, so re-runs never duplicate obligations. A follow-up chaser produces per-owner chase reports and escalates overdue P1s; an unremediated overdue P1 caps the control's next rating at Weak until closed with evidence. The loop actually closes.

8 · Portfolio sweep (app/portfolio.py)

One click runs every control in a division — or the whole bank — in parallel. Assessments draw on the documented owner-call narratives in the corpus; two batch gates replace dozens of individual approvals (approve all, or decide per control with rating overrides; rejected controls selectively re-run, round-tagged). Output: rating distribution, exception profile, 5×5 heat map, residual-risk score, and a Risk-Committee pack with a by-division roll-up on bank-wide sweeps.

9 · Self-improvement loop (app/improvement.py)

Expert testers submit feedback in-product (public endpoint, rate-limited and capacity-capped). Feedback becomes proposals: a whitelisted, bounded parameter change (testing thresholds, signal keyword lists, a workpaper prompt addendum) auto-applies — versioned, with one-call revert — while everything else queues for explicit owner approval, and the runtime never rewrites its own code. The owner console is gated by OWNER_KEY and fails closed: key not configured → owner endpoints are localhost-only.

Exports

• Excel test sheet (app/testsheet.py) — the classic controls-testing sheet layout, carrying both workpapers with provenance columns and the PII-attestation status.
• PDF export (app/pdfexport.py) — workpapers, committee packs and chase reports as printable PDFs (offline reportlab).
• Tester kit — guide + scenario templates downloadable in-app.
• Scenario upload (app/scenarios.py) — testers add their own controls; gated by TESTER_KEY (fail closed: unset = uploads disabled), strict filename allowlist, size caps, and a full parse + engine dry-run before a file is accepted.

Quick start (60 seconds, no API keys)

pip install -r requirements.txt
python run.py
# open http://localhost:8000  (redirects to /ui)

Demo mode runs fully offline: retrieval over the 208 synthetic control documents is real (BM25); generation is deterministic extraction and templating — honest about what it is — and it exercises every part of the graph including both human-approval interrupts.

On Replit: import this repo and hit Run (.replit config included; set keys in Replit Secrets).

Tests (offline, also run in GitHub Actions CI on every push):

pip install -r requirements-dev.txt
python -m pytest -q     # 106 tests

The suite covers ingestion of all 42 workbooks, the single-control lifecycle (both gates, re-test loop, no-workbook fallback), the design and OE assessments (including reviewer edits and Not-Determinable honesty), control-scoped retrieval, the adaptive demo interviewer, portfolio sweeps (every division + bank-wide, batch gates, selective re-runs, determinism), the remediation loop, the improvement loop's autonomy boundary, scenario upload security, exports, and the API surface.

The three modes

Mode	Set	LLM	Retrieval	Use for
demo (default)	nothing	Deterministic templates (extraction, not generation)	BM25 (real, offline)	Demos and CI; runs anywhere, zero keys
openai	MODE=openai, OPENAI_API_KEY	GPT-4o	Chroma + OpenAI embeddings	Full generative behaviour
azure	MODE=azure, AZURE_* vars	Azure OpenAI (tenant-resident)	Chroma + Azure embeddings, or Azure AI Search	Enterprise pilot, zero data egress

The graph, nodes, API and UI are identical in all three modes — and in every mode, verdicts with regulatory weight (testing conclusions, rating caps, final ratings, aggregation) are rule-based code, not model output. The LLM contributes narrative drafting only.

Environment variables

Copy .env.example to .env (all optional in local demo mode):

Variable	Default	What it does
MODE	demo	demo | openai | azure
DATABASE_URL	(unset)	Unset → SQLite files on local disk. Set to a managed Postgres connection string (e.g. Neon/Supabase free tier) for production durability — checkpoints, feedback/tuning policy, remediation actions and upload manifests survive deploys. Pooled and reconnect-safe against idle-suspending servers.
OWNER_KEY	(unset)	Gates the improvement console (proposal decisions, reverts, feedback triage). Fail-safe: set → X-Owner-Key header required; unset → owner endpoints are localhost-only, so a forgotten key on a public deploy denies rather than opens.
TESTER_KEY	(unset)	Gates in-app scenario uploads. Fail closed: unset = uploads disabled entirely. Share only with invited testers.
OPENAI_API_KEY, OPENAI_CHAT_MODEL, OPENAI_EMBED_MODEL	—	openai mode
AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, AZURE_OPENAI_API_VERSION, AZURE_CHAT_DEPLOYMENT, AZURE_EMBED_DEPLOYMENT	—	azure mode
RETRIEVER_BACKEND	auto	auto | bm25 | chroma | azure_search (auto picks per mode)
AZURE_SEARCH_SERVICE, AZURE_SEARCH_INDEX, AZURE_SEARCH_KEY	—	Azure AI Search backend (azure mode)
CHUNK_SIZE / CHUNK_OVERLAP / TOP_K	800 / 120 / 4	Retrieval tuning
MIN_INTERVIEW_TURNS / MAX_INTERVIEW_TURNS	3 / 8	Interview pacing
TESTING_MAX_PARALLEL_SAMPLES	25	Cap on parallel OE sample branches
TESTING_OE_FAIL_RATE / TESTING_DESIGN_FAIL_RATE	0.10 / 0.30	Testing verdict thresholds (also runtime-tunable, bounded, via the improvement loop)

API

Endpoint	What
POST /session/start	new assessment thread
POST /session/{id}/message	one chat turn through the graph
POST /session/{id}/approve	resume a gate: {approved, feedback} + gate-2 questionnaire edits/additions, walkthrough, OE basis, pii_attested
GET /session/{id}/state	full state (audit view)
POST /session/{id}/rcsa_output	Risk-Committee JSON (scores, 3LoD, actions, evidence trace)
GET /session/{id}/testsheet.xlsx	Excel test sheet (both workpapers, provenance, attestation)
POST /portfolio/start	division or bank-wide sweep {division}
POST /portfolio/{id}/review	resume a batch gate {approve_all | decisions}
GET /portfolio/{id}/state	sweep state: controls, aggregates, committee pack
GET /actions · POST /actions/{id} · POST /actions/chase	remediation registry + follow-up chaser
POST /feedback · GET /feedback/agents	public expert feedback (rate-limited, capped)
GET /improvement · POST /improvement/proposals · …/{pid}/decide · …/policies/{key}/revert	owner console (OWNER_KEY-gated, fail closed)
POST /export/pdf	markdown artefact → PDF
GET /scenarios · POST /scenarios/upload	tester scenario uploads (TESTER_KEY-gated, fail closed)
GET /downloads/{key}	tester kit (guide + templates)
GET /status · GET /health · GET /knowledge_base	service status, health, corpus stats
GET /ui	the product UI (/ redirects here)

Documents

Drop .docx / .xlsx control documents into control_docs/ — filename convention {CTRL-ID}_{Doc_Type}.docx (e.g. RB-CTRL-09_Control_Narrative.docx). The corpus ships with 208 synthetic documents for 42 controls (166 docx + 42 Design_Testing.xlsx workbooks, which are both flattened into the retrieval corpus and parsed into structured test plans — fully offline via openpyxl). Divisions: RB Retail Banking · CL Corporate Lending · MT Markets & Trading · WM Wealth Management. All data is synthetic.

Bringing your own controls? Use the templates in templates/, then pre-flight: python -m tools.validate_scenario your/XX-CTRL-01_Design_Testing.xlsx — it parses the file (headers and verdict vocabularies are synonym-tolerant) and dry-runs the actual design/OE rules so you see the conclusions before the file ever enters the corpus. Or upload in-app (TESTER_KEY required), which runs the same validation.

Project layout

app/
  config.py             env-driven settings — MODE / retriever / keys / tuning
  ingestion.py          docx+xlsx loader, 800/120 chunker, workbook parser
  profile.py            pre-interview control profile card (verbatim extraction)
  prompts.py            interview + assessment + committee-JSON prompts
  llm.py                provider factory: DemoLLM (adaptive, deterministic) | OpenAI | Azure
  retriever.py          backend factory: BM25 | Chroma | Azure AI Search — control-scoped
  state.py              RCSAState — the shared graph state
  nodes.py              intake → retrieve → interview → assess → approve ⏸ → reassess
  testing.py            controls-testing subgraph (planner → design → ∥ Send OE → workpaper → review ⏸)
  design_assessment.py  C1–C4 description review + D1–D12 questionnaire + walkthrough
  oe_assessment.py      C5–C11 monitoring review + OET scripts + basis + attestation
  portfolio.py          division sweep (∥ Send per control → batch gates → heat map + pack)
  remediation.py        action registry + chaser + rating-cap feedback
  improvement.py        feedback → proposals → bounded auto-tuning (versioned, revertible)
  scenarios.py          tester scenario uploads (fail-closed, validated, audited)
  testsheet.py          Excel test-sheet export (openpyxl)
  pdfexport.py          markdown → PDF (reportlab)
  graph.py              StateGraph wiring
  _checkpointer.py      SQLite (default) | Postgres (DATABASE_URL) checkpointer
  _dbkit.py             SQLite/Postgres connection shim for the registries
  server.py             FastAPI API + static UI
static/index.html       the product UI (chat, gates, dashboard, owner console)
control_docs/           the document corpus (drop your own here)
templates/              scenario templates for expert testers (docx + xlsx)
tools/                  make_templates · validate_scenario
tests/                  106 offline tests (also run in CI)

See ARCHITECTURE.md for the full graph diagrams (main lifecycle, testing subgraph, portfolio fan-out, improvement loop, deployment topology) and the honesty/security design rules.

Deployment

The live demo runs on Render (free tier) from render.yaml — connect the repo as a Blueprint and every push to main auto-deploys. Notes:

• Ephemeral disk: Render's free-tier filesystem is wiped on every deploy/restart. Set DATABASE_URL to a managed Postgres (Neon/Supabase free tier works) so graph checkpoints, feedback, tuning policy, remediation actions and upload manifests survive. Unset, everything falls back to local SQLite — correct for local dev and CI.
• Set OWNER_KEY (and TESTER_KEY if you invite testers) in the Render dashboard; both are fail-closed if forgotten.
• healthCheckPath: /health is preconfigured.

Security & Responsible AI

See RESPONSIBLE_AI.md — the RAI principles (human oversight, transparency, auditability, determinism, data minimisation), the concrete guardrails with code references (including G13, the self-improvement autonomy boundary), the AI threat model (prompt injection, hallucination, automation bias), the honest list of prototype security gaps with their pilot requirements, and the regulatory orientation (EU AI Act, FCA/PRA SS1/23, NIST AI RMF).

Honest demo-mode disclosure

In demo mode the retrieval pipeline is real (chunking, indexing, ranked, control-scoped retrieval with source attribution) and all verdicts are real rules over real evidence — but narrative generation is templated extraction: the DemoLLM quotes and assembles from retrieved documents deterministically. This is disclosed in-product. Flip MODE=openai with an API key for generative narratives; nothing else changes — the rules, gates and audit trail are identical.

---

All demo data is synthetic. Built independently; not affiliated with any employer or client. No real bank, firm or personal data appears in this repository.