Model

The Model class is the entry point for everything else on the Python surface. Construct one per .tinyloop artifact, reuse it across many requests, and release its VRAM either via close() or the with protocol.

tinyloop_py.Model(path, max_seq_len=2048)

import tinyloop_py

model = tinyloop_py.Model("model.tinyloop", max_seq_len=2048)

Alternative helper that is a pure alias for the constructor:

model = tinyloop_py.load("model.tinyloop", max_seq_len=2048)

Parameters

Name	Type	Default	Description
path	str	—	Path to a .tinyloop artifact on disk. See Model format for the file layout. Relative paths are resolved against the current working directory.
max_seq_len	int	2048	Upper bound on total sequence length (prompt + generated). Sizes buf.main (FP32 residual stream, seq-wide) at construction. When prefill_chunk == 0 this also sizes the FP16 scratch buffers (norm, scratch1, attn_down). Must be > 0.
prefill_chunk	int	0	When > 0, the FP16 scratch buffers are sized to prefill_chunk rows instead of max_seq_len. Prefill of seq_len > prefill_chunk runs through the chunked path (see Chunked prefill below); shorter prompts still run single-shot. Unlocks long-context prefill on tight VRAM at the cost of modest wall-clock overhead. Score-style APIs (score, score_last, score_logit_lens, score_trajectory) require seq_len <= prefill_chunk; generate / generate_stream work at any length. 0 (default) preserves the old full-seq scratch behaviour.

Returns

A live tinyloop_py.Model instance. The constructor blocks until:

1. The artifact header is parsed and validated.
2. Weights are uploaded to GPU memory (INT2 / INT4 packed, plus FP16 scales / zeros / LayerNorm params / embedding / head).
3. Runtime buffers (buf.main FP32 residual stream, buf.norm / buf.scratch1 / buf.attn_down FP16 scratch) are allocated to max_seq_len.
4. Auto tile sizing runs (reports loop-block vs L2 cache residency in the load log).

Typical load time on a 407M GPTQ-INT4 artifact on H100: ~200 ms including the CUDA context warmup. 1B-effective is ~400 ms.

Raises

• RuntimeError — path does not exist, is not a valid .tinyloop artifact, or the GPU has insufficient free memory to upload the weights and buffers. The error message names the failure point (header mismatch, missing section, allocation failure with requested size) so the caller can distinguish config vs capacity issues.
• ValueError — max_seq_len <= 0.

VRAM cost

The Model holds:

• Weights. INT2 / INT4 packed data + FP16 scales + FP16 zeros per quantized layer, plus full-precision (FP16) LayerNorm weights, embeddings, output head. Example: 1B-effective INT4 → 217 MB.
• Runtime buffers. buf.main is FP32 and always sized max_seq_len × D × 4 bytes. The FP16 scratch buffers (norm, scratch1, attn_down) are sized to scratch_rows × D × 2 bytes each, where scratch_rows = prefill_chunk when the override is active, else max_seq_len.
• FP16 body cache (opt-in via TINYLOOP_EXPERIMENTAL_FP16_BODY=1, read at first block forward). Adds ~3× weight VRAM (407M: 216 → 619 MB; 1B: 222 → 633 MB).
• KV caches — not owned by Model directly. Each call to generate / build_prefix_cache / build_resume_handle allocates its own RuntimeKVCache (released at the end of that call, or when the handle goes out of scope).

Measured on 1B-effective / H100:

Config	max_seq_len	prefill_chunk	Buffer VRAM
default	4096	0	117 MB
long-context tight	4096	512	44 MB (−62 %)

Inspect live VRAM use with Model.vram_usage_mb().

Chunked prefill

When prefill_chunk > 0 is set, long prompts (seq_len > prefill_chunk) run through a chunk-first driver: the prompt is split into chunks of prefill_chunk tokens and each chunk runs through all pre-blocks + loop iterations in turn, with attention growing over the cache across chunks.

Compatibility:

• Greedy output is bit-exact vs single-shot prefill for FP16 KV and FP16-h cache modes (W_k · h reconstruction is exact).
• INT8-h and INT4-h drift by the quantization reconstruction envelope (~0.6 % / ~10 % L2-rel respectively). This is a numerical path difference, not a bug — single-shot prefill uses exact FP16 K/V during attention; chunked uses reconstructed-from-INT K/V. Both are valid samples from the quantized posterior.
• INT8 KV (non-h) is not yet wired for chunked prefill — it falls back to single-shot, which may overflow a chunk-sized scratch. Set prefill_chunk=0 when using TINYLOOP_KV_INT8=1.

Wall-clock cost: at chunk=256 and 1024-token prompt (4 chunks), overhead is ~+42 % vs single-shot on 1B. At chunk=64 (16 chunks) overhead explodes to ~+5000 % because the total CUDA kernel launch count scales with chunks. Pick prefill_chunk >= 256 for production; smaller chunks only make sense when VRAM is the binding constraint.

Score-API gating: the single-shot score paths read norm[seq_len × D], which doesn't fit in a chunk-sized buffer. Those APIs refuse cleanly with seq_len > prefill_chunk and direct you to either shorten the prompt or reload the Model with prefill_chunk=0. generate / generate_stream / resume_generate work at any length because prefill is chunked and decode is BT=1.

Context manager

Model supports the with protocol and releases all GPU memory deterministically on block exit:

with tinyloop_py.Model("model.tinyloop") as model:
    logits = model.score_last(tokens, loops=8)
# VRAM freed here; calling model.* below raises RuntimeError.

This is the recommended pattern for evaluation scripts and tests — it guarantees the model is released even if an exception escapes the block.

Model.close()

model.close()

Release all GPU memory (weights + runtime buffers + FP16 body cache if active) synchronously.

• Idempotent. Safe to call multiple times; second and subsequent calls are no-ops.
• Post-close behaviour. Every public method (generate, score_*, build_*, config, vram_usage_mb, etc.) raises RuntimeError on a closed model, naming the method and instructing the caller to construct a new instance. This replaces the pre-2026-04-17 behaviour of segfaulting on post-close access.
• GIL. Released while running the CUDA teardown.

Model.__exit__(exc_type, exc_value, traceback)

Protocol method called by with. Delegates to close(). Never suppresses exceptions from the body.

Model.config()

config = model.config()

Returns a dictionary-like object with the artifact's static architecture metadata. No forward pass is required — the values are read directly from the .tinyloop header.

Returned keys

Key	Type	Meaning
dim	int	Hidden dimension D. Residual-stream and QKV projection width.
n_heads	int	Attention heads. head_dim = dim / n_heads. For INT8-h / INT4-h, head_dim must be even (the cache modes enforce this).
ffn_dim	int	Feed-forward hidden dimension. For SwiGLU this is the width of the gate and up projections.
vocab_size	int	Token vocabulary size. Output head is [vocab_size, dim].
embed_factor_dim	int	Factorized embedding dimension. 0 means full-rank embedding.
n_pre_blocks	int	Count of pre-loop blocks (weight-unshared transformer blocks before the shared loop). Default 2 for the reference architecture.
default_loops	int	Default loop depth baked into the artifact. Overridable per-call via the loops / n_loops kwargs on every scoring and generation API.

Example

cfg = model.config()
print(f"{cfg['dim']}-dim, {cfg['n_heads']} heads, {cfg['ffn_dim']} FFN, "
      f"L_default={cfg['default_loops']}")
# 2048-dim, 16 heads, 8192 FFN, L_default=8

Model.vram_usage_mb()

vram_mb = model.vram_usage_mb()

Returns the current VRAM footprint in MB as a Python float. Counts everything the Model object holds:

• Packed INT2 / INT4 weights + per-group FP16 scales / zeros
• FP16 LayerNorm parameters, embedding, output head
• Runtime residual + scratch buffers
• FP16 body cache (if active)
• Any PrefixCache / ResumeHandle objects still alive under this Model

Not counted: CUDA context overhead, cuBLAS / cuDNN internal workspaces. Use nvidia-smi for the full-process number.

Snapshot is point-in-time — calling during decode reflects the KV cache growth up to that moment.

Tokenization

TinyLoop does not ship a tokenizer. Do it in Python with whatever library matches the checkpoint — the example below uses HuggingFace's GPT2TokenizerFast:

import numpy as np
from transformers import GPT2TokenizerFast
import tinyloop_py

tok = GPT2TokenizerFast.from_pretrained("gpt2")
model = tinyloop_py.Model("model.tinyloop", max_seq_len=2048)

prompt_ids = np.asarray(
    tok.encode("Looped transformers are", add_special_tokens=False),
    dtype=np.int32,
)
out_ids = model.generate(prompt_ids, max_tokens=32, loops=8, temperature=0.0, top_k=50)
print(tok.decode(out_ids))

All TinyLoop APIs take np.ndarray[int32] or compatible array-like objects — never byte strings. Input type conversions are performed via numpy.asarray(x, dtype=np.int32) inside the binding; lists of Python ints work but add one copy.

Thread safety

A single Model instance is not thread-safe for concurrent inference. It owns:

• A single CUDA stream for all forward-pass work.
• A single set of buf.main / buf.norm / buf.scratch1 / buf.attn_down residual / scratch buffers.

Two threads calling generate / score_* / resume_generate on the same instance will race on those buffers.

Safe patterns:

1. Lock per model. Wrap calls in a threading.Lock. Sufficient when request concurrency is small and request latency is the main signal.
2. Model-per-worker. Each worker thread constructs its own Model pointing at the same .tinyloop file. Each gets independent buffers; CUDA handles the device-side kernel queuing. Scales linearly in weight VRAM — OK for 407M / 1B on an H100 but not for larger models.
3. Process-per-replica. Multiple Python processes, each with one Model. Avoids the GIL entirely; costs IPC for result marshalling.

Future work tracked in the production roadmap: continuous batching so a single Model can serve many requests simultaneously without per-request buffer duplication.

Lifetime and reuse

A single Model is designed to be long-lived:

• Construct once per .tinyloop artifact, per process.
• Reuse across thousands of generate / score_* calls.
• Only close() explicitly when you need the VRAM back (e.g. switching artifacts mid-process or releasing before a memory-hungry non-TinyLoop operation).

Constructing a second Model does not invalidate the first — both coexist on the same GPU subject to VRAM budget.

Determinism

All Model APIs produce bit-identical output across repeated calls with the same inputs when greedy decoding is used (temperature=0.0):

• score, score_last, score_logit_lens, score_trajectory — deterministic regardless of sampling kwargs; they do not sample.
• generate, generate_stream, generate_speculative — deterministic when temperature=0.0 (top_k / top_p become no-ops under greedy).
• resume_generate — deterministic under greedy and bit-exactly matches a fresh generate(loops=new_L) on the same prompt.

Non-determinism enters only through temperature > 0 sampling, which uses a per-call PRNG seed derived from Python's random module. If you need reproducible sampling, seed random before the call.

The forward pass itself does not depend on any PRNG (there is no dropout at inference time), so weights + tokens + loop depth determine logits completely.