Grammar-forced speculative drafts (#48): forced spans in constrained JSON output are free, guaranteed-accepted drafts - #70
Merged
Conversation
…trained JSON output are free, guaranteed-accepted drafts
New byte-level GBNF-subset engine (c/grammar.h: parser + set-of-stacks PDA
walker) wired into spec_decode as a third draft source ("metodo F"), tried
before MTP/n-gram. Wherever the grammar admits exactly one legal byte, the
forced span is tokenized and injected as drafts; the existing batch-union
verification confirms them, so a wrong or out-of-sync grammar can never
change the output. Lazy arming skips preambles; adaptive guard (same
pattern as MTP) disables the source below 50% acceptance; grammar-accepted
tokens no longer pollute the MTP acceptance counter.
GRAMMAR=file.gbnf enables it in run and serve modes (also with DRAFT=0 and
with the int4 MTP head from JustVugg#8); GRAMMAR_DRAFT=n caps the span (default 24).
Measured on M3 Max / int8-MTP container, greedy, MTP=0 DRAFT=0, NDJSON
classification: 0.37 -> 0.50 tok/s (1.60 tok/forward, 81 fw per 130 tok),
100% acceptance (48/48), output byte-identical to baseline.
Co-Authored-By: Claude Fable 5 <[email protected]>
JustVugg
approved these changes
Jul 11, 2026
JustVugg
left a comment
Owner
There was a problem hiding this comment.
Beautifully done — this is exactly the design the idea called for, and it's implemented with the right safety guarantees. Validated here:
- Build clean (0 warnings), oracle unchanged with no grammar (32/32 TF, 20/20 greedy).
test_grammarpasses.- Lossless proof: greedy generation with a grammar loaded vs without produces byte-identical output on the oracle (20/20 either way) — confirming the grammar only proposes drafts that normal verification accepts or rejects; a wrong or out-of-sync grammar cannot change the result.
The three properties that make this special all hold: ~1.0 acceptance on forced spans (correct by construction), engages even with the int4 MTP head from #8 (no model head needed), and composes with MTP/n-gram (grammar tried first, existing sources take over at branches). The self-contained grammar.h (byte-level GBNF subset, left-recursion detected and fails safe) keeps it clean, and the adaptive guard below 50% acceptance matches the MTP auto-off pattern.
This is the highest-leverage speculation source for structured-output/API workloads — on JSON-heavy generation the forced-token fraction is large and free. Great work turning #48 into reality. Merging.
This was referenced Jul 11, 2026
This was referenced Jul 13, 2026
fabio-rovai
added a commit
to fabio-rovai/colibri
that referenced
this pull request
Jul 13, 2026
…rafts (JustVugg#48/JustVugg#70 follow-up) schema_gbnf.h compiles a practical JSON-Schema subset (strict objects, string/ number/integer/boolean/null, enum/const, arrays with items, nesting) into the byte-level GBNF subset grammar.h parses, so structured-output workloads get grammar-forced drafts without hand-writing GBNF. Unsupported keywords fail soft: the engine runs without a grammar and output is unchanged (drafts are verified, never constraints - a wrong compile can only cost acceptance, not correctness). grammar_setup: GRAMMAR= (raw GBNF) keeps precedence; SCHEMA= feeds the compiler into the same gr_parse path. 8 test groups in tests/test_schema_gbnf.c walk compiled grammars end-to-end through the PDA (forced spans, enum disambiguation, nested instances, escapes, leading-zero rejection, fail-closed fallbacks). Co-Authored-By: Claude Fable 5 <[email protected]>
fabio-rovai
added a commit
to fabio-rovai/colibri
that referenced
this pull request
Jul 13, 2026
…rafts (JustVugg#48/JustVugg#70 follow-up) schema_gbnf.h compiles a practical JSON-Schema subset (strict objects, string/ number/integer/boolean/null, enum/const, arrays with items, nesting) into the byte-level GBNF subset grammar.h parses, so structured-output workloads get grammar-forced drafts without hand-writing GBNF. Unsupported keywords fail soft: the engine runs without a grammar and output is unchanged (drafts are verified, never constraints - a wrong compile can only cost acceptance, not correctness). grammar_setup: GRAMMAR= (raw GBNF) keeps precedence; SCHEMA= feeds the compiler into the same gr_parse path. 8 test groups in tests/test_schema_gbnf.c walk compiled grammars end-to-end through the PDA (forced spans, enum disambiguation, nested instances, escapes, leading-zero rejection, fail-closed fallbacks). Co-Authored-By: Claude Fable 5 <[email protected]>
fabio-rovai
added a commit
to fabio-rovai/colibri
that referenced
this pull request
Jul 13, 2026
…rafts (JustVugg#48/JustVugg#70 follow-up) schema_gbnf.h compiles a practical JSON-Schema subset (strict objects, string/ number/integer/boolean/null, enum/const, arrays with items, nesting) into the byte-level GBNF subset grammar.h parses, so structured-output workloads get grammar-forced drafts without hand-writing GBNF. Unsupported keywords fail soft: the engine runs without a grammar and output is unchanged (drafts are verified, never constraints - a wrong compile can only cost acceptance, not correctness). grammar_setup: GRAMMAR= (raw GBNF) keeps precedence; SCHEMA= feeds the compiler into the same gr_parse path. 8 test groups in tests/test_schema_gbnf.c walk compiled grammars end-to-end through the PDA (forced spans, enum disambiguation, nested instances, escapes, leading-zero rejection, fail-closed fallbacks). Co-Authored-By: Claude Fable 5 <[email protected]>
JustVugg
added a commit
that referenced
this pull request
Jul 14, 2026
…rafts (#148) * SCHEMA=<file.json>: JSON-Schema -> GBNF compiler for grammar-forced drafts (#48/#70 follow-up) schema_gbnf.h compiles a practical JSON-Schema subset (strict objects, string/ number/integer/boolean/null, enum/const, arrays with items, nesting) into the byte-level GBNF subset grammar.h parses, so structured-output workloads get grammar-forced drafts without hand-writing GBNF. Unsupported keywords fail soft: the engine runs without a grammar and output is unchanged (drafts are verified, never constraints - a wrong compile can only cost acceptance, not correctness). grammar_setup: GRAMMAR= (raw GBNF) keeps precedence; SCHEMA= feeds the compiler into the same gr_parse path. 8 test groups in tests/test_schema_gbnf.c walk compiled grammars end-to-end through the PDA (forced spans, enum disambiguation, nested instances, escapes, leading-zero rejection, fail-closed fallbacks). Co-Authored-By: Claude Fable 5 <[email protected]> * schema_gbnf: whitespace-tolerant emission (jws at separators) Measured on GLM-5.2 current main (#146): the greedy continuation writes sloppy JSON (spaces after colons, fences, long free text) and a compact-only grammar desyncs at the first stray space, forfeiting every span after it. jws points are not forced themselves (two legal bytes) but the multi-byte spans around them keep drafting and the walker survives non-compact output - strictly acceptance-positive for a verified draft source. Tests re-derived for the new span boundaries + a sloppy-instance walk. Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: Claude Fable 5 <[email protected]> Co-authored-by: JustVugg <[email protected]>
fabio-rovai
added a commit
to fabio-rovai/colibri
that referenced
this pull request
Jul 14, 2026
…rafts (JustVugg#48/JustVugg#70 follow-up) schema_gbnf.h compiles a practical JSON-Schema subset (strict objects, string/ number/integer/boolean/null, enum/const, arrays with items, nesting) into the byte-level GBNF subset grammar.h parses, so structured-output workloads get grammar-forced drafts without hand-writing GBNF. Unsupported keywords fail soft: the engine runs without a grammar and output is unchanged (drafts are verified, never constraints - a wrong compile can only cost acceptance, not correctness). grammar_setup: GRAMMAR= (raw GBNF) keeps precedence; SCHEMA= feeds the compiler into the same gr_parse path. 8 test groups in tests/test_schema_gbnf.c walk compiled grammars end-to-end through the PDA (forced spans, enum disambiguation, nested instances, escapes, leading-zero rejection, fail-closed fallbacks). Co-Authored-By: Claude Fable 5 <[email protected]>
ErikTromp
pushed a commit
to SensAI-PT/aviary
that referenced
this pull request
Aug 9, 2026
…ce, guaranteed-accepted forced spans, lossless + opt-in (JustVugg#48, JustVugg#70) New byte-level GBNF-subset engine (c/grammar.h: parser + set-of-stacks PDA walker) wired into spec_decode as a third draft source ("metodo F"), tried before MTP/n-gram. Wherever the grammar admits exactly one legal byte, the forced span is tokenized and injected as drafts; the existing batch-union verification confirms them, so a wrong or out-of-sync grammar can never change the output. Lazy arming skips preambles; adaptive guard (same pattern as MTP) disables the source below 50% acceptance; grammar-accepted tokens no longer pollute the MTP acceptance counter. GRAMMAR=file.gbnf enables it in run and serve modes (also with DRAFT=0 and with the int4 MTP head from JustVugg#8); GRAMMAR_DRAFT=n caps the span (default 24). Measured on M3 Max / int8-MTP container, greedy, MTP=0 DRAFT=0, NDJSON classification: 0.37 -> 0.50 tok/s (1.60 tok/forward, 81 fw per 130 tok), 100% acceptance (48/48), output byte-identical to baseline. Co-authored-by: Claude Fable 5 <[email protected]>
ErikTromp
pushed a commit
to SensAI-PT/aviary
that referenced
this pull request
Aug 9, 2026
…rafts (JustVugg#148) * SCHEMA=<file.json>: JSON-Schema -> GBNF compiler for grammar-forced drafts (JustVugg#48/JustVugg#70 follow-up) schema_gbnf.h compiles a practical JSON-Schema subset (strict objects, string/ number/integer/boolean/null, enum/const, arrays with items, nesting) into the byte-level GBNF subset grammar.h parses, so structured-output workloads get grammar-forced drafts without hand-writing GBNF. Unsupported keywords fail soft: the engine runs without a grammar and output is unchanged (drafts are verified, never constraints - a wrong compile can only cost acceptance, not correctness). grammar_setup: GRAMMAR= (raw GBNF) keeps precedence; SCHEMA= feeds the compiler into the same gr_parse path. 8 test groups in tests/test_schema_gbnf.c walk compiled grammars end-to-end through the PDA (forced spans, enum disambiguation, nested instances, escapes, leading-zero rejection, fail-closed fallbacks). Co-Authored-By: Claude Fable 5 <[email protected]> * schema_gbnf: whitespace-tolerant emission (jws at separators) Measured on GLM-5.2 current main (JustVugg#146): the greedy continuation writes sloppy JSON (spaces after colons, fences, long free text) and a compact-only grammar desyncs at the first stray space, forfeiting every span after it. jws points are not forced themselves (two legal bytes) but the multi-byte spans around them keep drafting and the walker survives non-compact output - strictly acceptance-positive for a verified draft source. Tests re-derived for the new span boundaries + a sloppy-instance walk. Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: Claude Fable 5 <[email protected]> Co-authored-by: JustVugg <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #48 — the grammar itself as a third draft source, next to MTP and n-gram lookup.
What it does
For constrained-output workloads (JSON/NDJSON, function calling, structured extraction), wherever the grammar admits exactly one legal byte — braces, quotes, key names, separators, enum bodies — that forced span is tokenized and injected as drafts that the existing batch-union forward verifies. Properties, as discussed in the issue:
DRAFT=0.Usage
GRAMMAR=schema.gbnf coli run ... # or serve; GRAMMAR_DRAFT=n caps the span (default 24)Grammar format: byte-level GBNF subset (llama.cpp style) — literals with escapes, char classes incl. negation, rule refs, groups,
| ? * +, comments, multi-line rules. Root rule isroot. Left recursion is detected and fails safe (walker off, generation proceeds undrafted).How
c/grammar.h(new, header-only liketok.h/st.h): GBNF parser + PDA walker with a set of stacks in normal form (top of every stack is a terminal byte-class, or the stack is empty = parse completable).gr_forced()extends the forced span while exactly one byte is legal and the parse cannot terminate there (where it can, the model may legitimately emit EOS, so we don't force).c/glm.c: the walker follows every emitted token (gr_feed), arming lazily at the first byte the root admits (preambles are skipped) and re-arming after a desync.spec_decodeasks the grammar first; forced bytes are encoded with the engine's own tokenizer — the tokenization boundary is not guaranteed to match the model's, which is fine: verification absorbs it (that's what the acceptance stat measures). Separategrammatica:acceptance stat; grammar-accepted tokens no longer pollute the MTP acceptance counter.c/tests/test_grammar.c: 13 scenarios (forcing, branching, enums,? * +incl. multi-byte-literal repetition, negated classes, desync, hex escapes, parse errors, left recursion, and the NDJSON shape from the issue), wired intomake check.Measured (Apple M3 Max, 128 GB, same box as #47)
Real-model A/B on the int8-MTP container (
GLM-5.2-colibri-int4-with-int8-mtp, 357 GB), greedyTEMP=0,MTP=0 DRAFT=0(grammar is the only draft source),NGEN=130,RAM_GB=60, NDJSON classification task with the grammar below, fresh process per run:GRAMMAR=fit.gbnfGrammar used:
Happy to adjust naming/placement to taste — e.g. if you'd rather the serve/API layer compile JSON schemas to GBNF (the OpenAI
response_formatseam you mentioned), that composes naturally on top of this engine-side mechanism and I can follow up with it.🤖 Generated with Claude Code