C++ API

TinyLoop's C++ layer is the canonical runtime surface. Python bindings (tinyloop_py) are a thin pybind11 wrapper over this API, and the HTTP server is an orchestration layer built on top of the Python wrapper.

This section documents C++ API behavior from the implementation (include/tinyloop.h, include/model.h, include/kv_cache.h, src/inference.cpp, src/generate.cpp) rather than from wrapper semantics.

Namespace and header contract

Public surface:

#include <tinyloop/tinyloop.h>

Everything documented here lives under namespace tinyloop.

tinyloop.h is the stable entry point for:

• lifecycle (load_model, free_model)
• scoring (score* family)
• generation (generate* family)
• cache handles (PrefixCache, ResumeHandle, PrefixPool)
• runtime memory probes (vram_usage_bytes)

Internal headers (model.h, kv_cache.h, runtime_internal.h) expose implementation details and are intentionally not the long-term compatibility boundary.

API topology

tinyloop::Model* lifecycle
  ├─ score / score_last_token / score_logit_lens / score_with_*
  ├─ generate / generate_stream / generate_batch
  ├─ generate_speculative / generate_tree_speculative
  ├─ PrefixCache build + decode reuse
  ├─ ResumeHandle build + resume_generate
  └─ PrefixPool register + generate_with_pool

How the runtime executes

For a standard decode call, the state machine is:

1. Model load

   • Parse .tinyloop header (magic, version, dims, loop defaults).
   • Upload packed INT2/INT4 + scales/zeros + LN/output tensors.
   • Allocate reusable buffer pool (BufferPool) sized by max_seq_len and optional prefill_chunk.

2. Prefill

   • Embed prompt tokens.
   • Run pre-blocks (n_pre_blocks).
   • Run shared loop block L times.
   • Populate runtime KV cache layers.

3. Decode

   • Per emitted token: append one KV row/layer, compute next-token logits from last hidden, sample.
   • Stop by max_tokens, EOS, stop-sequence tail match, grammar dead-end, or callback abort (streaming path).

4. Teardown

   • Free per-call caches/temporaries.
   • Keep model weights + buffer pool alive for reuse across requests.

The architecture is optimized for repeated inference calls on one loaded model, not one-shot load/execute/free loops.

Why this design exists

TinyLoop deliberately keeps C++ API breadth narrow while keeping runtime modes rich.

• Narrow ABI reduces compatibility churn.
• Most complexity (quantization paths, KV modes, speculative variants) is encoded in config/env/runtime logic, not in deep class inheritance.
• This fits the project goal: high-performance runtime for one architecture family (weight-shared looped transformers), not a generic model-zoo interface.

Major trade-offs

• Performance-first ownership model: raw pointers and explicit free APIs keep overhead low but push correctness responsibility to callers.
• Single-model mutable buffers: one Model* is not concurrently re-entrant for inference without external synchronization.
• Feature velocity via env flags: mode composition is powerful but increases operational complexity.
• Specialization over generality: architecture-specific wins (warm-start, shared-weight speculative, store-h KV) at the cost of broad checkpoint compatibility.

Reference

Each page below is a self-contained reference for one API family. Pick the one that matches what you need, or read in order for a full tour.

Section	Covers
Lifecycle and Memory Model	load_model, free_model, vram_usage_bytes, ownership rules, prefill_chunk, thread-safety.
Scoring APIs	score, score_last_token, score_logit_lens, score_with_uncertainty, score_with_consistency_escalation.
Generation APIs	generate, generate_stream, generate_batch, generate_speculative, generate_tree_speculative, GenerateConfig.
Cache Reuse and State Transition APIs	PrefixCache, ResumeHandle, PrefixPool — reuse prefill state across calls or upgrade loop depth on the same prompt.
Batching and Integration Patterns	generate_batch constraints, service-level orchestration patterns, runtime-mode composition matrix.