Thanks to visit codestin.com
Credit goes to github.com

Skip to content
View vishalhabib99's full-sized avatar
🎯
AI PM @ Vanguard — the same eval discipline, proven on 40+ real MCP servers
🎯
AI PM @ Vanguard — the same eval discipline, proven on 40+ real MCP servers
2 followers · 0 following

Block or report vishalhabib99

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
vishalhabib99/README.md

Hi, I'm Vishal 👋

Product Leader & Builder — Agentic AI, AI/ML-Powered Customer Experiences & Platform Products

AI-native product leader with 10+ years defining strategy, vision, roadmap, and launching 0-to-1 products and 1-to-100 platforms across fintech, wealth management, telecom, retail, ecommerce, and healthcare — spanning B2B, B2C, and B2B2C.

💼 Currently

Leading product strategy for Digital Advisor and Personal Advisor at Vanguard ($6B+ LOB, 4M+ MAU) — architecting the next-generation Agentic AI Digital Advisor from 0-to-1, including model evaluation frameworks (correctness, groundedness, safety, latency) and FINRA/SEC-compliant responsible AI design.

🚀 Selected work

  • Vanguard — Agentic AI Digital Advisor (0→1): LLM orchestration, autonomous agent workflows, personalized financial guidance at scale, with regulatory guardrails and model governance.
  • T-Mobile — Launched one of the first autonomous enterprise Agentic AI platforms in US telecom: 75% adoption, 46% automation, 60% containment, 80% CSAT, 30% fewer support calls. Built the IntentCX AI governance & model evaluation framework (aligned to NIST AI RMF), adopted org-wide by 3 additional teams.
  • eBay — Drove $300M+ in savings via marketplace platform modernization and API standardization across hundreds of engineering teams.

🎓 Background

Stanford University Graduate School of Business · Google AI certifications (AI Essentials, Responsible AI, Prompting, AI Tools for Productivity)

🔗 Why this portfolio exists

The "model evaluation frameworks" line above isn't abstract — it's the same discipline, one layer down. At Vanguard and T-Mobile that means correctness/groundedness/safety/latency checks on an agent's outputs; below is the same four-axis taxonomy applied to the MCP tools an agent actually calls: is the tool documented well enough to use correctly (mcp-doctor), does it fail safely on bad input and respond fast enough to matter (mcp-fuzz), does its response actually reflect reality (mcp-reality-check). Same eval mindset, different substrate — built and dogfooded against 40+ real, in-the-wild MCP servers up to 50k★, not synthetic fixtures.

What this doesn't cover, by design: semantic hallucination judgment ("is this answer actually true" needs an LLM judge, which breaks the fully-deterministic, no-API-cost property all three tools share) and dynamic agent-side red-teaming (mcp-doctor's security score audits a tool's own code and description for injection risk — tool poisoning, dangerous exec — not whether a live agent can be manipulated by a malicious tool response at runtime). Runtime authorization enforcement (does a tool actually refuse an out-of-scope action, not just claim to) is a related, permanent no for a different reason: everything else here works because it's a generic pattern meaning the same thing across any server; "out of scope" is different for every server (a project ID, a directory, a schema), and testing it for real needs per-target configuration — that's manual security review, not something a generic auditor can do honestly. Two more, handled differently: observability/structured-logging completeness was evaluated and explicitly declined — there's no spec-mandated logging format to check against, and the only proxy I could think of (flagging a tool with a Context param that never logs anything) would misfire on any simple tool that legitimately has nothing to log; not worth shipping a weak heuristic just to claim coverage. Drift/regression tracking and supply-chain (unpinned dependency) checking — both closed since: the leaderboard diffs every re-scan against whatever was live before it, and mcp-doctor now flags a dependency with no version floor at all. Real, named gaps, not swept under "the trilogy covers everything."

🧱 Portfolio

  • 📄 ai-pm-portfolio — PRDs, working prototypes, and honestly-reported evals (including failures, not just wins)
  • 🩺 mcp-doctor — static-analysis CLI that audits MCP server implementations for the things that break an agent calling them (pip install mcp-server-lint, or a GitHub Action). Dogfooded against 40+ real MCP servers in the wild across Python, TypeScript, and Go — up to 50k★, including official servers from GitHub, HashiCorp, Red Hat, Brave, and MathWorks — found and fixed 35 genuine bugs, verified against each repo before/after, including two real security-heuristic false positives (a plain RegExp.exec() call and Playwright's $eval() DOM helper both misflagged as dangerous code execution) caught auditing arabold/docs-mcp-server, and a false "no description" on two real tools caught auditing oraios/serena (29K★) — the tool's description was being read from the wrong docstring, verified directly against serena's own registration code. One fix led to a merged PR into ha-mcp (4.5K★, active); a separate clean pass surfaced a real doc-coverage gap filed upstream in excel-mcp-server (4.1K★, active). Added a read-only annotation-mismatch check — flags a tool declaring readOnlyHint: true whose own code contains a real write signature (raw SQL mutation, a file opened for writing, a mutating HTTP call) — a real, recurring bug class, not hypothetical: mcp-fuzz's own codebase-memory-mcp#2118 finding below found 13 of 15 tools mislabeled this exact way. Most recently, added --diff-against, a schema/contract diff mode: saves a run's tool signatures as a baseline and catches a breaking change later — a tool that's gone, a parameter that's gone, or one that's newly required — the exact gap a real commenter on the trilogy LinkedIn article (Tyler Robinson) named directly ("contract consistency"). Verified against real git history, not just synthetic fixtures: diffed jingcheng-chen/rhinomcp 20 commits behind its own main, honestly clean (tools added, none removed). Live results: a public leaderboard ranking every audited repo by quality + security grade, each with a real per-repo badge maintainers can embed in their own README — now with drift tracking, comparing every re-scan against whatever was live before it so a regression shows a real ▲/▼ delta instead of silently overwriting the old score. Caught a real cross-environment gotcha while verifying it (a Python 3.11-vs-3.12 parsing difference reading as a false "regression" locally, confirmed harmless since production scans are always 3.12-vs-3.12) — traced to root cause and documented rather than shipped blind. Then ran the full toolkit — every check across all three tools — against one fresh, real target at once for the first time: sooperset/mcp-atlassian (5.9K★, real Jira/Confluence integration), 98 tools, 100%/A quality, 99%/A security, --diff-against a commit 30 revisions back honestly clean. Also added an unpinned-dependency (supply-chain) check — flags a requirements.txt/package.json dependency with no version floor at all, excluding >=/caret/tilde ranges (standard practice, not a real signal on their own) to keep it low-noise — verified clean against six real repos.
  • 🧪 mcp-fuzz — mcp-doctor's companion (pip install mcp-runtime-check): instead of reading source, it launches a real MCP server and calls its tools with inputs derived from their own JSON schema, checking whether the server fails cleanly (a structured error) or crashes/hangs on missing or wrong-typed input. 22 real-world passes so far, up to 61K★ (upstash/context7) — including Ant Design's official mcp-server-chart (4.3K★) — all 27 tools crashed instead of erroring cleanly on realistic bad input, filed upstream and confirmed fixed on main after a live rescan — a real crash bug filed on shadcn-ui-mcp-server (3K★), and a tool-annotation mislabeling filed on codebase-memory-mcp (42.7K★, official-scale). An external maintainer (agent-inspect, a trajectory-debugging tool for TypeScript agents) asked to feed a real mcp-fuzz session into their evidence model — shipped --full-trace (v0.1.5) for full-fidelity JSONL export in response. The finding it surfaced (a healthy, 100%/A crash-resilience session reads as almost entirely failed once isError:true rejections collapse into a generic error status) was substantial enough that the maintainer asked for a reproducible write-up — filed as agent-inspect#362, with a verified fix demonstration built against AgentInspect's own outcome_observed schema. AgentInspect shipped a fix in response (v6.26.0, --preset behavioral-session) that scores outcome-level failures without collapsing a healthy session into a failed run, verified against the same sanitized gist and merged into their own repo's recipe fixtures. Also hardened the schema generator against adversarial tools/list metadata — found and fixed three real self-bugs (non-dict schema fields, a 5000-level-deep schema blowing the recursion limit). Also ran end-to-end against oraios/serena (29K★) with a real Python language server actually active (not a stub) — 28/28 tools tested with --include-destructive, 0 crashes, 100%/A, the deepest real-server coverage of any repo audited so far. Also added a real, scored latency check — the fourth axis (correctness/safety/groundedness/latency) alongside the other two tools — flagging a tool's valid call as slow via an absolute threshold or as a >3x outlier against its own server's median; re-running against oraios/serena found two real relative outliers (create_text_file at 9.4x median, get_current_config at 4.6x), 92.9%/B. Most recently, added a matching response-size check — same design, applied to payload size instead of response time, catching a tool that burns an agent's context window for no reason its description would predict; re-running against oraios/serena found four real relative outliers (onboarding at 17.2x median, get_current_config at 13.4x, plus two smaller ones), 85.7%/C. Most recently, added an opt-in concurrency check (--concurrency N): launches N independent connections (each its own subprocess) and calls a tool with identical arguments on all of them at once — a real test of concurrent access to a shared backend (a file, database, lock), not synthetic fan-out over one connection. Verified two ways: a new fixture tool with a genuine, reproducible race (a naive create-exclusive lock file) gets caught reliably; the official modelcontextprotocol/server-memory reference server comes back clean — an honest result, since each of its subprocesses keeps its own independent state, nothing to race on. Same full-checklist pass on sooperset/mcp-atlassian (5.9K★): 58/98 tools tested, 100%/A across crash resilience, latency, response size, and concurrency — the largest real target run against the complete toolkit, genuinely clean. Most recently, added an opt-in resource-lifecycle check (--sequential): every check above tests one tool call in isolation with synthetic arguments — this one detects create_X/get_X/delete_X-shaped tool groups by name, chains the real id a create call actually returns into the read/delete calls, then re-reads that id after deletion to catch a stale read (a resource that still reads as present after being deleted). Verified against a deliberately-planted bug in the test fixture (a fake delete that reports success but never removes the record) — caught correctly, with a clean healthy trio alongside it producing no false positive. Shipped as v0.5.0.
  • 🩻 mcp-reality-check — completes the trilogy (pip install mcp-reality-check): mcp-doctor asks is it documented, mcp-fuzz asks does it fail safely, this asks does it actually work. Calls each tool once with a realistic (not placeholder) input and checks the response for a refusal disguised as success ("I don't have access to..." wrapped in isError: false), empty content on a claimed success, and violations of a tool's own declared output schema — all fully deterministic, no LLM judge, no API key, no per-call cost. 18+ real-world passes so far, most recently corroborating a clean mcp-fuzz pass on arabold/docs-mcp-server — found and fixed a real bug in its own input generator (a missing "timezone" hint broke every call to mcp-server-time), then hardened the same generator against the malformed-metadata bug class found in mcp-fuzz. Everything else clean. Also corroborated the same oraios/serena LSP-backed run as mcp-fuzz above: 10/10 checkable tools, 0 flagged, 100%/A. Most recently, found a real bug in itself while dogfooding sooperset/mcp-atlassian: no --env flag existed at all, so any config-gated server (this one registers zero tools without Jira/Confluence credentials) was silently untestable — mcp-fuzz had already solved the identical problem months earlier for brave-search-mcp-server. Added --env, re-verified: 58 tested/40 skipped, matching mcp-fuzz's count exactly, 100%/A.

✍️ Writing

📫 Reach me

Pinned Loading

  1. ai-pm-portfolio ai-pm-portfolio Public

    AI Product Manager portfolio: PRDs, working prototypes, and honestly-reported evals for agentic AI product ideas

    Python

  2. mcp-doctor mcp-doctor Public

    CLI that audits MCP (Model Context Protocol) server implementations for spec conformance and quality — missing descriptions, undocumented params, unhandled errors, README coverage.

    Python 1

  3. mcp-fuzz mcp-fuzz Public

    Runtime behavioral testing for MCP servers: calls every tool with schema-derived inputs and checks whether it actually behaves the way its description and schema claim.

    Python 1

  4. mcp-reality-check mcp-reality-check Public

    Checks whether an MCP server's successful tool responses actually reflect reality: no disguised refusals, no empty content, no output that ignores the schema it declared.

    Python