docs: replace MkDocs with Zensical#3073
Conversation
📚 Documentation preview
|
There was a problem hiding this comment.
1 issue found across 15 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Note for reviewers: the red "Docs Preview" check is expected on this PRThe
The authoritative signal is One follow-up called out in the PR description: the combined build serves v1 (MkDocs) + v2 (Zensical), and this branch's Generated by Claude Code |
Migrate the v2 docs build from MkDocs Material to Zensical, the Material team's successor (as Starlette and uvicorn have done). Zensical natively re-implements search, glightbox and mkdocstrings, but — unlike MkDocs — runs no arbitrary plugins or hooks. This SDK's docs relied on three of those, so they are reimplemented as standalone build steps around `zensical build`: - API reference: `mkdocs-gen-files` + `mkdocs-literate-nav` become `scripts/docs/gen_ref_pages.py` (writes the `docs/api/` mkdocstrings stubs) and `scripts/docs/build_config.py` (splices the nested API nav into a generated `mkdocs.gen.yml`, since Zensical has no literate-nav). - llms.txt: the former MkDocs hook becomes `scripts/docs/llms_txt.py`, a post-build step over the source tree, with no dependency on build internals. `mkdocs.yml` stays the source config, with the MkDocs-only YAML tags (`!relative`, `!ENV`, `!!python/name:`) translated to the string forms Zensical understands and the emoji functions pointed at `zensical.extensions.emoji`. The generators moved out of `docs/` (Zensical publishes everything under docs_dir) into `scripts/docs/`, and `scripts/build-docs.sh` now picks the toolchain per worktree so the combined v1 (MkDocs) + v2 (Zensical) build stays correct. Adds `scripts/serve-docs.sh` for local preview. The docs dependency group drops mkdocs, mkdocs-material[imaging] and the gen-files/glightbox/literate-nav plugins for `zensical`. mkdocstrings is floored at 1.0.4 / mkdocstrings-python 2.0.5: Zensical's compatibility layer targets that API, and the older line renders API pages with an unregistered-autorefs error. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01CTkPwxyKkLTnqZdeiCMCzK
The wrapper forwards `"$@"` straight to `zensical serve`, so extra args are passed directly; the `--` in the usage example would be forwarded literally. Show args as direct forwarding instead. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01CTkPwxyKkLTnqZdeiCMCzK
The three-step build (materialise config, zensical build --strict, generate llms.txt) was copy-pasted in shared.yml, docs-preview.yml, and build-docs.sh. Move it into scripts/docs/build.sh and call that everywhere. docs-preview.yml now picks the toolchain from the checkout: pull_request_target runs the workflow file from the base branch, so hardcoding the Zensical steps would break previews for older heads and v1.x-targeted PRs (and previews of this branch before it merges).
Zensical ships a nav entry for a nonexistent page as a broken link with no diagnostic, even under --strict; MkDocs strict aborted the build. Validate the materialised nav in build_config.py so the guarantee survives the migration, and reject an empty generated API nav while at it.
The social plugin is not handled by the scripts under scripts/docs/ — it was
dropped (Zensical has no social-card support; the cards were gated on an env
var no workflow set). The NavItem alias excluded the {title: url} page shape
it mostly holds.
mkdocstrings renders a member re-exported across a package boundary as a full duplicate of its canonical documentation whenever the target package happens to be loaded already, and silently omits it when it isn't - which of the two the api/mcp/ index got depended on page rendering order (255 headings under Zensical vs 123 under MkDocs, from the same stub). Pin the semantics in the generated stub instead: every export with a canonical page elsewhere under the API reference becomes a compact link to it, and only exports documented nowhere else (re-exports from private modules) keep their full body via an explicit members list. Packages whose re-exports never cross a package boundary (mcp_types) are deterministic already and keep the plain stub. griffelib moves from a transitive to a declared docs dependency since the generator now imports it directly.
Zensical natively supports the watch key; dropping it broke serve live-reload for docs_src/ snippet edits. The deploy-docs banner env only applied to mkdocs-material builds, and the v1.x fallback arm already sets it inline.
Unresolvable cross-references render as literal bracket text without a diagnostic, even under --strict, and link validation skips non-markdown targets entirely (a missing image ships silently; MkDocs failed both). Grep the built site for the literal-bracket signature after the build, and validate relative asset targets while llms_txt walks the prose. Also fail when a generated package index is missing from llms.txt's Optional section, so adding a package to gen_ref_pages cannot silently under-publish.
Docstring summaries took the first line, not the first paragraph, so wrapped one-sentence docstrings shipped cut off mid-sentence on the package index. The private-module filter only looked at the path leaf, so a private subpackage would have published its whole subtree; and an export resolving outside the documented packages now fails with a named error instead of a raw griffe traceback.
build_config's --site-dir/--output flags had no callers and contradicted the fixed site/ output contract; nav validation now skips external URLs, which are valid nav entries. The griffelib comment preempts the reasonable suspicion that the name is a typo for griffe.
Rendering the stub resolves its cross-package aliases in mkdocstrings' own collection. On a warm incremental rebuild the target package's pages can all be cache hits, so nothing loads it and rendering the changed index crashes with AliasResolutionError — breaking the serve/edit loop for any src change that alters the generated index. Cold builds were unaffected. preload_modules pins the targets into the collection.
7c7b6f6 to
d629160
Compare
…e canvas Zensical's UI keeps Material's md-* class names but ships different stock nav styling (8px-radius pill rows with 7px 16px padding), which broke the sidebar design in two ways: section labels sat indented past their guide line (the label's inner anchor carries the pill padding), and the vertical rhythm doubled. Re-express the same design against the new stock rules, keeping Zensical's pill highlight for the active page. The dark scheme also swaps Zensical's near-black canvas back for the Material slate blue-grey this site shipped with; code blocks keep Zensical's own surfaces.
The stubs have no H1 of their own, so their pages fell back to the title Index — which is what navigation.prune shows on collapsed nav rows (every collapsed API subtree rendered as 'Index'), and what browser tabs and search results display.
The section-label overrides tied the density rule on specificity and lost on source order, re-indenting labels; and the density padding also hit the md-nav__container wrapper (stock 0), double-indenting group rows that link to an index page. Order the density rules first and densify the container's inner anchor instead of the wrapper. The slate canvas override moves under @media screen like the stock rule it replaces, so dark-mode printing keeps a light canvas; stub titles gain YAML quoting.
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Top-level pages outside any section kept the pill's left padding plus a stock margin, sitting 14px right of the section labels; pull their box left by the pill inset so text lines up and the active pill overhangs the gutter. The API Reference label (the one section without an index page, rendered as a label element rather than a container div) was still caught by the density rule via :not() specificity — exclude section items from the density rules outright instead of relying on source order.
The previous rules chased text alignment with per-element compensations (zeroed paddings, a negative margin), which clipped the active pill flat against the scroll container's paint edge, left label pills hugging their text, and fought the list's stock flex gap with per-link margins. Derive everything from the pill inset instead: every row is a direct .md-nav__link child with the same 10px horizontal padding (inner anchors inside container wrappers zeroed), so all text shares one column and pills always paint with breathing room inside the clip edge; section rules carry typography only, eliminating the specificity coordination; and the flex gap becomes the single density knob (2px, restoring the ~30px row pitch the site shipped with on Material).
Zensical paints nav hover/active pills on .md-nav__link[href] anchors and [for] labels, never on the .md-nav__container wrapper — which is where the previous commit put the padding, leaving container-type section labels with a pill hugging their text. Move the inset onto the paint elements (leaf links, bare labels, and a wrapper's inner anchor) and keep wrappers geometry-neutral like stock. The 10px inset also keeps the pill's corner radius (0.4rem) from crowding the glyphs.
The inline-code chip is an absolute dark surface designed for the page canvas; on an admonition's tinted panel it sits as an opaque slab (Zensical's modern theme has the same defect on its own docs — classic Material mitigated it with a hairline ring on title code, which modern dropped). Scope a tone-on-tone translucent chip to code inside admonitions, the pattern Starlight and Docusaurus ship for code inside callouts; prose chips keep the block-matching dark surface.
Every markdown link target is now classified in one place instead of pattern-matched by shape, so no link can dodge validation by its spelling: scheme-prefixed targets (including mailto:) are external, targets inside fenced blocks or code spans are illustrative, and everything else must resolve to a file under docs/ (containment enforced, directory-style and absolute targets rejected, assets rewritten to absolute URLs). Fences are scanned line-based per CommonMark (closer may be longer than the opener, unclosed fences run to EOF) and code spans cannot cross a blank line, so a stray backtick cannot exempt later paragraphs from validation. Frontmatter is stripped the way the old hook's meta extraction did, restoring title and description frontmatter support in llms.txt. Output is byte-identical for the current docs corpus.
Zensical stays green even under --strict when a cross-reference fails to resolve or an objects.inv inventory fails to download. The old guards had holes in both directions: grepping HTML for '[<code>' missed plain-text unresolved refs and false-positived on subscripted types in backticked prose, and nothing at all caught a failed inventory download, which silently degrades thousands of standard-library links to plain text on exactly the cold-build path deploys run on. check_crossrefs.py validates the built site instead, so no log-wording change can disarm it: it extracts prose text outside pre/code (marking skipped code so bracket shapes stay distinguishable) and fails on the literal reconstruction an unresolved reference leaves behind, and it requires every inventory declared in mkdocs.yml to contribute at least one autorefs-external anchor - hand-authored prose links to the same host don't count, and an inventory contributing nothing is dead config. Offline contributors can set DOCS_ALLOW_INVENTORY_FAILURE=1; CI never skips it. build.sh now also builds cold: zensical's incremental cache drops cross-references to cache-hit pages and leaves stale HTML for deleted pages, which made warm local runs fail on phantom breakage.
…btree The compact-index treatment was gated on top-level packages, but mcp.server.mcpserver (Icon from mcp_types) and mcp.client.auth (AuthorizationCodeResult from mcp.shared.auth) re-export across their subtree boundary too, so their index pages rendered full duplicates of canonical documentation. Register every package index for the pass; _compact_index still returns None for packages whose exports all live in their own subtree, so exactly those two indexes change. Compact stubs are titled with the last path component like every other stub (the full dotted path leaked into sibling nav rows), and the package gate keys on the __init__ module rather than the output filename so a real module named index.py cannot masquerade as a package index.
Replacing pymdownx.extra with bare superfences silently dropped abbr, def_list, footnotes, and pymdownx.betterem. The missing betterem was already visibly wrong: the 'application/*' and 'text/*' literals in the streamable_http docstring paired their asterisks into spurious emphasis. A full-site rendering diff shows that fix is the only change the four extensions cause.
MkDocs (--strict with validation.omitted_files: warn) failed the build for a page on disk that the nav omitted; Zensical builds it as unreachable orphan HTML. Validate both directions of nav/page drift in build_config with one nav walker, using the same scheme-based external-link test as llms_txt so a scheme-only nav entry (mailto:) cannot be misread as a page path. The generated api/ tree is exempt from the orphan check: its nav is spliced in from the same generator that writes the files.
Zensical is pre-1.0 and the build guards key on its rendering behavior (autorefs classes, compact-index handling), so toolchain bumps should be deliberate lockstep changes rather than incidental resolver upgrades. The pinned versions are what the lockfile already resolved.
zensical 0.0.48 shipped a regression that left the search UI hidden (fixed upstream in 0.0.50): the built pages carried the hidden attribute on the search components and the runtime never mounted them, so the site had no search bar. Verified against a full 0.0.50 build: the search markup renders without hidden, search.json is generated, and the crossref/inventory guards and llms artifacts are unchanged.
The compact treatment was gated on package indexes, but plain modules re-export across boundaries too: mcp.server.caching re-exports CACHEABLE_METHODS and CacheableMethod from mcp_types.methods and rendered full duplicates of their canonical documentation. Run every documented module through the pass; six module pages flip. Three mechanisms tightened while widening: compact pages keep the module's own public docstringed members that are not in __all__ (a plain stub renders those, and flipping must drop nothing); links target the deepest alias hop whose module is documented, so they point at the defining page by construction instead of relying on mkdocstrings registering alias identifiers; and the redundant self-root preload is omitted (rendering a stub loads its own package). Statically invisible __all__ names and shadowed module lookups fail with clean errors instead of tracebacks.
check_crossrefs: only block-level tags separate extracted text - inline tags flow, so a subscript like **tools**[`0`] keeps the word character that exempts it - and the block set covers the def-list/details/table tags the enabled extensions emit. A br separates text without resetting the skip state (it is legal inside code and implies no closed element). llms_txt: link titles accept all three CommonMark quoting forms (parenthesized titles previously made the whole link invisible to the classifier); reference-style definitions are rejected in every valid spelling (no space after colon, next-line destination, indented inside admonition bodies) while footnote definitions stay supported; and one _prose_h1 helper now owns both the title fallback and the H1 strip for llms-full.txt, skipping code blocks (snippet pointer lines) and matching only real ATX headings, so the two can no longer disagree.
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Nav validation rejects entries that are absolute or escape docs/ before
checking existence (a ../ value resolved against the wrong root and an
escaping entry would write its llms rendition outside the built site);
llms_txt applies the same containment when collecting prose pages. Block
HTML comments join fences and code spans as inert content, so commented-out
prose is neither validated nor rewritten. Link openings tolerate the
whitespace CommonMark allows after '(' - previously ']( page.md)' and
']( <x>)' dodged both the classifier and the angle-form rejection. And
check_crossrefs fails up front on a missing site directory instead of
scanning zero pages as a clean site.
| # Every markdown link/image target: `](target#anchor "title")`. Each target is | ||
| # classified in `_rewrite_links` — there is deliberately no shape-based | ||
| # pre-filter here, so no link can dodge validation by its spelling. Zensical's | ||
| # own link validation only covers .md targets (a missing image or | ||
| # directory-style link builds green even under --strict; MkDocs failed the | ||
| # build), so everything else is validated here. | ||
| _LINK = re.compile(r'(\]\()([^)\s]+?)(#[^)\s]*)?( +(?:"[^"]*"|\'[^\']*\'|\([^()]*\)))?(\))') | ||
| # CommonMark forms the classifier deliberately rejects rather than models: | ||
| # angle-bracket destinations `](<target>)` and reference-style definitions | ||
| # `[label]: target` (footnote definitions `[^label]:` are a different, | ||
| # supported syntax). Either would otherwise dodge validation by its spelling; | ||
| # failing loud keeps the guarantee without modelling unused syntax. | ||
| _ANGLE_LINK = re.compile(r"\]\(<") | ||
| _REF_DEFINITION = re.compile(r"^[ \t]*\[(?!\^)[^\]]+\]:", flags=re.MULTILINE) | ||
| # A scheme-prefixed target (https:, mailto:, tel:, ...) is external — the | ||
| # `://` shorthand misses scheme-only URIs like mailto:. | ||
| _EXTERNAL = re.compile(r"[a-zA-Z][a-zA-Z0-9+.-]*:") | ||
| # Fenced code blocks and inline code spans: their content is inert in the | ||
| # rendered HTML, so links inside them are illustrative text, neither validated | ||
| # nor rewritten. Fences are matched line-based in `_code_intervals` (closer at | ||
| # least as long as the opener, unclosed runs to EOF, per CommonMark) and spans | ||
| # only in the text between fences; a span cannot cross a blank line, so a | ||
| # stray unpaired backtick cannot swallow the paragraphs (and links) after it. | ||
| # Known approximations of the renderer's block model: 4-space-indented | ||
| # content is treated as prose, because in this corpus indentation is | ||
| # admonition/list body whose links must stay validated — a link in a true | ||
| # indented code block is over-validated (fails loud or gets rewritten in the | ||
| # rendition), never under-validated; and span pairing is bounded by blank | ||
| # lines rather than full block structure. | ||
| _FENCE = re.compile(r"^[ \t]*(`{3,}|~{3,})") |
There was a problem hiding this comment.
🟡 Three residual spots where the hand-rolled link/fence model still diverges from python-markdown (the engine Zensical renders through): (1) whitespace-padded or empty destinations like [t](servers/tools.md ) or [t]() match neither _LINK nor the loud-reject scans, so they are silently neither validated nor rewritten — a padded broken asset target ships as a silent 404 and a valid padded .md link ships source-relative into the renditions; (2) a CommonMark-valid balanced-paren destination [x](foo(1).md) is mis-captured as foo(1) by [^)\s]+?, so the build aborts with a misleading "cannot resolve link target 'foo(1'" naming a path that isn't in the page; (3) _FENCE treats a line-initial multi-backtick code span like pip install mcp as a fence opener (CommonMark forbids backticks in a backtick fence's info string), and the phantom fence runs to EOF, silently exempting every later link from validation/rewriting. Nothing under docs/ trips any of these today; fixes are to allow optional whitespace around the destination/title (or loud-reject padded forms), allow one level of balanced parens in the target (or loud-reject '(' like the angle-bracket case), and make the backtick fence-opener branch reject backticks in the info string, e.g. ```{3,}(?[^\n]*`)``.
Extended reasoning...
What the bugs are. The classifier comment above _LINK promises totality ("no link can dodge validation by its spelling"), and unmodeled CommonMark forms are deliberately rejected loudly (_ANGLE_LINK, _REF_DEFINITION). Three residual shapes — all verified empirically against the compiled patterns at HEAD and against python-markdown 3.10.2, the renderer Zensical uses — break that contract in three different ways: a silent dodge, a loud mis-parse, and a silent phantom fence.
1. Whitespace-padded and empty destinations (silent dodge). _LINK requires the target to start immediately after ]( and the closing ) to follow the target/title with no intervening whitespace, and needs at least one target character. CommonMark and python-markdown accept padding and emptiness: [t](servers/tools.md ), [t]( servers/tools.md), [t](servers/tools.md\t), [t](target 'Title' ), [i](img/missing.png ), and [t]() all render as working <a>/<img> elements, yet _LINK.search() returns no match on every one, and neither loud-reject scan fires (_ANGLE_LINK needs the literal ](+< adjacency, so even [t]( `````<a b.md>`````) slips past it). Proof: _rewrite_links('See [diagram](img/missing.png ) here.', ...) returns the page byte-identical with no error, while the unpadded twin raises the designed _BuildError. Consequences: a padded broken asset target ships as a silent 404 through a fully green build (this script is the pipeline's sole guard for non-.md targets — Zensical only validates .md links, and check_crossrefs sees a resolved element with no bracket text), and even a valid padded .md link is never rewritten to an absolute URL, so the published .md rendition and llms-full.txt carry a source-relative link that 404s at the rendition's directory-URL depth while the HTML site looks fine.
2. Balanced-paren destinations (loud mis-parse with a misleading message). CommonMark allows balanced parentheses in a bare destination, and markdown.markdown('[x](foo(1).md)') renders <a href=\"foo(1).md\">. But _LINK's target class [^)\s]+? cannot consume ), so _LINK.search('[x](foo(1).md)').group(2) == 'foo(1' (reproduced) — the (\)) group grabs the inner paren. The truncated 'foo(1' is not an anchor, not scheme-prefixed, not absolute, and (DOCS / 'foo(1') exists as neither dir nor file, so the build aborts with cannot resolve link target 'foo(1' — a missing-file error naming a path that appears nowhere in the author's page. Neither loud-reject scan fires, so the failure bypasses the designed "not supported here; use X" contract. Worse, an existing parenthesized file (diagram(v2).png, a common Windows "copy (1)" export artifact) is unlinkable entirely, since the existence check runs on the truncated path — while Zensical renders the link fine.
3. _FENCE misreads a line-initial multi-backtick code span as a fence opener (silent under-validation). CommonMark — honored by python-markdown and pymdownx.superfences at the locked versions — rejects a backtick fence opener whose info string contains a backtick, so a line like \`\`\`pip install mcp\`\`\` is an inline code span in a paragraph. But _FENCE (^[ \t]*(\{3,}|~{3,})) ignores the rest of the line and treats it as an opener. Reproduced: for 'Intro.\n\n```pip install mcp```\n\nSee Tools and X.', _code_intervalsreturns one phantom interval from the span line to EOF, and_rewrite_linksreturns the page byte-identical — the broken link is never existence-checked (for asset targets this script is the sole guard, so it ships as a silent 404) and the valid link is never rewritten (source-relative 404 in the renditions). With a real fence later in the page, the phantom interval closes at that fence's all-backtick *closer*, exempting the intermediate prose and inverting code/prose parity afterward; the same mechanism defeats_prose_h1`. This directly falsifies the "a stray unpaired backtick cannot swallow the paragraphs (and links) after it" guarantee comment — here a paired triple-backtick span swallows everything. Note the tilde branch is correct: CommonMark permits backticks in a tilde fence's info string, so only the backtick branch diverges.
Why the existing fixes don't cover these. Commit 8acb8dd closed the previously reported classifier holes (extension caps, extensionless targets, single-quoted/paren titles, angle destinations, reference definitions), but every one of those keyed on the target's spelling — none touched the surrounding whitespace grammar, the paren-in-target case, or the fence-opener info-string rule. The documented "known approximations" name only 4-space-indented content and blank-line-bounded span pairing, and the posted _code_intervals review comments cover indented blocks, cross-block span pairing, and HTML comments — different mechanisms with non-overlapping fixes.
Fixes. (1) Allow optional [ \t]* around the destination and after the title in _LINK (re-emitting without the padding), plus either matching empty destinations and failing loud or a third rejected-forms scan for \]\([ \t]*\); (2) extend the target grammar with one level of balanced parens, e.g. (?:[^()\s]|\([^()\s]*\))+?, or loud-reject ( in targets so the failure at least routes through the instructive rejection message; (3) make the backtick fence-opener branch reject backticks in the info string: _FENCE = re.compile(r'^[ \t]*(\{3,}(?[^\n]*`)|~{3,})')` — a one-clause change; the closer logic needs none (closers are all-backtick lines and still match).
Severity. Nit: grep confirms nothing under docs/ trips any of the three today (consistent with the byte-identical-renditions verification), so nothing breaks at merge. The cost lands on future contributors — as a silent broken link/rendition divergence for (1) and (3), and as a loud but misdiagnosed CI abort for (2) — the same latent-classifier-hole family as the findings already confirmed and fixed on this PR.
| # Zensical runs no MkDocs plugins or hooks, so the build is three steps: | ||
| # materialise the API reference pages and the concrete config, build the | ||
| # site strictly, then generate llms.txt and the per-page markdown | ||
| # renditions. This script is the single owner of that recipe, dependency | ||
| # sync included — CI (shared.yml, docs-preview.yml) and scripts/build-docs.sh | ||
| # all call it. The toolchain detection in docs-preview.yml and build-docs.sh | ||
| # keys on this file's path and expects the site under site/. |
There was a problem hiding this comment.
🟡 The header comment says "the build is three steps" but the script now runs four: commit a483385 inserted the check_crossrefs.py site-validation step between the strict build and llms_txt.py without updating this enumeration (the sibling comment in .github/workflows/shared.yml was updated in the same commit). Reword to four steps — or drop the count — and mention the post-build cross-reference/inventory validation.
Extended reasoning...
What's stale. Lines 5–8 of scripts/docs/build.sh describe the recipe as "three steps: materialise the API reference pages and the concrete config, build the site strictly, then generate llms.txt and the per-page markdown renditions" — but the script body now executes four pipeline steps: build_config.py (line 31), zensical build -f mkdocs.gen.yml --strict (line 32), check_crossrefs.py --site-dir site (line 38), and llms_txt.py --site-dir site (line 40).
How it happened. Commit a483385 replaced the old grep -F '[<code>' guard with the standalone check_crossrefs.py checker and inserted it between the strict build and the llms.txt step. That same commit did update the parallel comment in .github/workflows/shared.yml — which now correctly names "the post-build steps [that] fail on unresolved cross-references, inventory download failures, and broken non-markdown link targets" — but the enumeration in this header was never touched, leaving it as the only stale description of the recipe.
Why the omitted step can't be read as part of "build the site strictly". The crossref check exists precisely because zensical build --strict does not cover those failure classes: the inline comment at lines 34–37 of this same file says so explicitly ("Zensical stays green even under --strict when a cross-reference fails to resolve ... or an objects.inv inventory fails to download ... Validate the built site instead"). Per the PR's own guarantees inventory, check_crossrefs.py is the sole owner of two failure classes — unresolved [text][identifier] cross-references rendering as literal bracket text, and failed inventory downloads silently degrading every inventory-driven link — so it is a distinct, load-bearing step, not a detail of the build step.
Why it matters (mildly). The header explicitly claims this script is "the single owner of that recipe", so it is the natural place a maintainer reads to understand or replicate the pipeline (e.g. in another workflow, or when deciding whether a step is "extra"). An enumeration that says the pipeline ends at llms.txt invites exactly the misreading that got the shared.yml comment fixed earlier on this PR — same comment-drift class as that already-confirmed misattribution finding.
Step-by-step proof.
- Read the header: it promises exactly three steps — config/API materialisation, strict build, llms.txt.
- Read the body after
uv sync:build_config.py→zensical build --strict→check_crossrefs.py→llms_txt.py. Four distinctuv runinvocations, four steps. - The third step is not subsumable into the second: lines 34–37 document that
--strictships both of its failure classes green, which is the whole reason the step exists. git show a483385 -- .github/workflows/shared.ymlshows the sibling comment being updated to name the post-build checks in the same commit that added the step here — confirming drift rather than a deliberate summary.
Why nit. Comment-only, zero runtime effect, and the omitted step carries its own accurate four-line explanatory comment 25 lines below in the same 40-line file — a reader of the script itself can't actually miss it; only someone relying on the header enumeration alone is misled.
How to fix. One-line reword, e.g.: "the build is four steps: materialise the API reference pages and the concrete config, build the site strictly, validate the built site's cross-references and inventories, then generate llms.txt and the per-page markdown renditions" — or drop the count entirely.
| """Generate llms.txt, llms-full.txt, and per-page markdown (https://llmstxt.org/). | ||
|
|
||
| Zensical has no equivalent of MkDocs' build hooks, so this runs as a standalone | ||
| post-build step over the source tree (`mkdocs.yml` + `docs/`) and writes | ||
| three kinds of artifact into the built `site/`: | ||
|
|
||
| - `llms.txt`: a markdown index of the documentation, one link per page, | ||
| grouped by nav section. | ||
| - a `.md` rendition of every prose page next to its HTML (e.g. | ||
| `servers/tools/index.md`), which is what the llms.txt links point at. | ||
| - `llms-full.txt`: every prose page concatenated for single-fetch consumption. | ||
|
|
||
| Page markdown is the source markdown with YAML frontmatter stripped, `--8<--` | ||
| snippet includes resolved (so the `docs_src/` code examples appear inline) and | ||
| relative links rewritten to absolute URLs. The API reference pages under | ||
| `api/` are mkdocstrings stubs with no prose source, so they are linked as | ||
| rendered HTML from an Optional section instead of being embedded. | ||
|
|
||
| Usage: | ||
| python scripts/docs/llms_txt.py --site-dir site | ||
| """ | ||
|
|
There was a problem hiding this comment.
🟡 The ~800 lines of hand-rolled parsing across the four scripts/docs/ scripts (llms_txt.py, check_crossrefs.py, gen_ref_pages.py, build_config.py) now own the pipeline's advertised correctness guarantees, yet have zero unit tests — tests/ has no directory for them, and both pyright's include list and coverage's source omit scripts/. This PR's own review history found ~20 latent edge-case bugs in exactly this parsing logic, each fixed and verified manually with no regression test pinning it; a small tests/scripts_docs/ suite exercising the pure functions (_code_intervals, _rewrite_links, _split_frontmatter, unresolved_refs, _dest_md_uri, _nav_pages/_validate_nav) would convert that accumulated verification work into a durable regression net.
Extended reasoning...
What the gap is. This PR replaces MkDocs plugin/hook machinery with four standalone scripts totalling ~830 lines: scripts/docs/llms_txt.py (~347), gen_ref_pages.py (~232), check_crossrefs.py (~164), and build_config.py (~85). They contain dense hand-rolled parsers — a markdown link classifier with code-fence/code-span interval tracking (_LINK, _FENCE, _CODE_SPAN, _code_intervals), a YAML frontmatter splitter (_split_frontmatter), a snippet resolver, an HTMLParser subclass with skip-depth/block-boundary state (_ProseTextExtractor), and a griffe alias-resolution walker (_compact_index). None of this has a single unit test: tests/ contains no directory for scripts (only cli, client, docs_src, examples, interaction, issues, server, shared, transports, types), pyproject.toml's [tool.pyright] include lists src/mcp, src/mcp-types/mcp_types, tests, docs_src and examples/* but not scripts/, and [tool.coverage.run] source omits it too. So the code that now owns the pipeline's advertised guarantees — nav validation, link validation, crossref detection, llms.txt fidelity — is invisible to type checking, to the 100%-coverage gate, and to the test suite, exercised only end-to-end by the docs CI job, which by construction covers only the happy path of the current corpus.\n\nWhy this matters empirically — the proof is this PR's own review history. Across the review rounds on this PR, roughly 20 latent edge-case bugs were confirmed and fixed in exactly this parsing logic: the _ASSET_LINK extension-length cap, the mailto:/scheme-only external-link misclassification, missing frontmatter stripping, cross-block code-span pairing, angle-bracket destinations, the subscript carve-out lookbehind in check_crossrefs.py, docs/ containment escapes in both the link classifier and the nav validator, extensionless/directory-style targets, external nav URLs ending in .md, the H1-fallback scanning into code fences, and more — plus the three residual shapes reported in the latest round. Each fix grew the regexes/state machines further, and each was verified manually (byte-identical output diffs, planted failures) rather than pinned by a regression test. Step by step, that means: (1) a future contributor refactors _code_intervals or _LINK — say, to add the angle-bracket destination support suggested in review; (2) they run ./scripts/test and the full CI matrix — all green, because nothing in tests/ touches these modules; (3) the docs job also passes, because the current corpus under docs/ happens not to contain a mailto: link, a frontmattered page, or a stray backtick; (4) the reintroduced edge case ships silently, exactly as it did the first time, until a review or a contributor trips it. Nothing in the repo prevents step (4) — the ~20 findings above each took an adversarial review pass to surface, and that pass won't re-run on the next refactor.\n\nWhy the "scripts/ is conventionally untested" refutation doesn't hold here. It's true the exclusion predates this PR (update_readme_snippets.py and gen_surface_types.py are also untested), and the deleted docs/hooks/*.py had no tests either. But the old hooks delegated the actual validation to MkDocs strict mode — upstream, tested code — whereas these scripts are the validation layer now: the PR description's own guarantees inventory assigns missing-nav-target, unresolved-crossref, and asset-link detection to them as sole owners. The exclusion was calibrated for trivial helpers; this PR relocates the pipeline's correctness enforcement itself into scripts/, which is the surface-area-increase case the convention never contemplated. Repo precedent also cuts the other way: tests/docs_src/test_shape.py deliberately duplicates a docs-build guarantee inside the ordinary pytest run, CI enforces fail_under = 100 with branch = true, and the repo ships a dedicated test-quality skill — the strongest test culture signal a repo can send.\n\nWhy it's cheap to fix. The flagged functions are pure, string-in/value-out, with no I/O beyond an optional docs/ fixture: _code_intervals(str) -> list[tuple[int,int]], _split_frontmatter(str) -> (dict, str), _dest_md_uri(str) -> str, unresolved_refs(html) -> list[str], _rewrite_links(...) (raises _BuildError on the documented rejection shapes), _nav_pages/_validate_nav. Every confirmed-and-fixed finding in this PR's review threads comes with a ready-made test case — the empirical proofs in those comments are literally executable assertions (unresolved_refs('<p>the <strong>tools</strong>[<code>0</code>] entry</p>') == [...], _LINK.search("[x](<servers/tools.md>)")", the frontmatter/H1 interactions, the containment escapes). A tests/scripts_docs/` (or similar) suite pinning those cases is mostly transcription, and would also let the modules be added to pyright's include list.\n\nImpact and severity. Nothing breaks at merge: CI is green, the scripts work on the current corpus, and this is CI-only tooling, not shipped SDK code — hence a nit, not a blocker. The cost is forward-looking: the substantial verification effort accumulated across this PR's review rounds (each fix confirmed with planted failures and byte-identical diffs) currently lives only in resolved review threads, and the next refactor of any of these state machines can silently undo any of it. Converting that work into a regression suite — even just the ~20 confirmed cases — is the cheapest possible insurance on the pipeline's guarantees, and matches the bar the rest of the repo already holds itself to.
Migrates the v2 docs build from MkDocs Material to Zensical, the Material team's successor — the same move Starlette and uvicorn made. Original migration by @Kludex; taken over to fold in review findings and empirical validation.
Why this isn't a drop-in swap
Zensical natively re-implements
search,mkdocstringsandglightbox, but runs no arbitrary MkDocs plugins or hooks and its module/extension API hasn't shipped. This repo's docs relied on three plugin/hook-based pieces, which become standalone build steps aroundzensical build(the sidecar pattern):mkdocstrings,search,glightbox, emojimkdocs.yml(unchanged semantics)mkdocs-gen-files(theapi/tree)scripts/docs/gen_ref_pages.pymkdocs-literate-nav(theapi/nav)scripts/docs/build_config.py→mkdocs.gen.ymlhooks:(llms_txt.py)scripts/docs/llms_txt.py(post-build)socialcardsscripts/docs/build.shis the single owner of the whole recipe (dependency sync → generate config/API stubs →zensical build --strict→ llms.txt):shared.yml,docs-preview.yml(via the toolchain detection merged in #3081) andscripts/build-docs.shall call it.mkdocs.ymlstays the source config;mkdocs.gen.ymlanddocs/api/are gitignored build artifacts.Guarantees that did not survive the toolchain swap — and where they moved
Zensical builds some failure classes green that MkDocs strict-mode failed, so the sidecar scripts own them now (each verified by inducing the failure and observing the loud error):
--strict→build_config.pyvalidates every nav entry againstdocs/(external URLs exempt).[name][identifier]cross-reference → renders as literal bracket text, build stays green →build.shgreps the built site for the rendered signature and fails..mdtargets →llms_txt.pyvalidates relative asset links while walking the prose.llms_txt.pyfails if a generated package index is missing from its Optional section.Config-level
strict: trueis parsed but not honored by Zensical (verified: only the CLI flag aborts), so the--strictflag inbuild.shis the enforcement point; the oldvalidation:block is a severity filter that at best restates defaults, at worst masks issues, and stays removed. Broken-anchor detection is on by default in Zensical and verified working.The API package index is now deliberate instead of accidental
mkdocstrings renders a member re-exported across a package boundary (
mcp/__init__re-exportingmcp_types) as a full duplicate of its canonical docs when the target package happens to be loaded, and silently omits it when it isn't — soapi/mcp/had 123 headings under MkDocs (the ~50mcp_typesre-exports were just absent) and 255 under Zensical (all duplicated), from the same stub, purely by rendering order.gen_ref_pages.pypins the semantics: themcpindex renders each export as a compact one-line entry linking to its canonical page, inlining full docs only for exports documented nowhere else. This also surfacedClientResult, which was invisible on both old sites despite being in__all__. The declarative alternative (mkdocstrings'summaryoption) was tested in four variants and can't do this: it produces either an empty page or full duplicate bodies with same-page anchors, dropsClientResultin every variant, and inherits the same load-order accident.preload_modulesis emitted for the cross-package targets so warm incremental rebuilds can't hitAliasResolutionError(reproduced, fixed, re-verified).Visual parity and design
Zensical's modern UI keeps Material's
md-*DOM but ships different stock styling, sodocs/extra.csswas rebuilt against it (validated by measurement on the live preview, light and dark):navigation.prunelabels collapsed rows with the page title, which fell back to "Index"; browser tabs too).Verification
All exercised on real builds in a clean environment (plus the docs CI job on this branch):
bash scripts/docs/build.sh: strict build green with zensical 0.0.48; ~35s cold, ~3s warm incremental (a capability MkDocs didn't have). MkDocs baseline was ~33s.mcplogo icon, palette toggles,--8<--snippet includes, tabs and admonitions render correctly; search index atsite/search.json.KeyError).griffelibis griffe's successor distribution by the same author — not a typo — and is imported directly by the generator.Breaking changes
None for SDK users. For docs contributors:
mkdocs servebecomesscripts/serve-docs.sh; local one-shot builds arebash scripts/docs/build.sh.Follow-up (out of scope)
[v1.x]backport of the toolchain-detectingscripts/build-docs.sh, so a v1.x-triggered Pages deploy can still build main's half of the combined site. Until it lands, deploys triggered from v1.x pushes fail (main-triggered deploys, previews and CI are unaffected).--strict.AI Disclaimer