A lossless compression research bench, built around one question: can algebraic representation compete with entropy coding?
The answer is no, and the repo now measures the whole design space that proves it — from the original Chebyshev codec at 35.8 bits per character down to a pure-Python PPM that beats bz2.
Bits per character on the Gospel of Matthew (121 KB, 28-symbol lowercase). Lower is better.
| bpc | what it is | |
|---|---|---|
| ppm4 | 2.062 | PPMC, escapes + exclusion — ours |
| bz2 | 2.094 | stdlib |
| bwt | 2.204 | BWT + MTF + order-1 — ours |
| ctw16 | 2.211 | context tree weighting — ours |
| lzma | 2.252 | stdlib |
| order3 | 2.255 | adaptive context model — ours |
| zlib -6 | 2.545 | stdlib |
| order-1 entropy floor | 3.220 | (bound) |
| order-0 entropy floor | 4.124 | (bound) |
| bezier, annealed alphabet | 4.647 | cubic Bézier + optimised rank axis — ours |
| bezier, spectral alphabet | 4.902 | Fiedler-ordered rank axis — ours |
| packed5 | 5.000 | 5 fixed bits per symbol |
| bezier, alphabetical | 5.538 | the same curve, conventional order — ours |
| partition bound | 5.841 | (bound) |
| bezier, raw ASCII | 6.964 | the same curve, ASCII code points — ours |
| subset bound | 7.247 | (bound) |
| raw ASCII | 8.000 | |
| poly-compress, GF(p) | 8.901 | exact finite-field roots |
| delta + varint | 10.524 | the PRD's own baseline |
| poly-compress, float Q=32 | 35.852 | the original Chebyshev codec |
PPM beats every stdlib codec here; BWT and CTW beat lzma and zlib but not bz2. The two entries at the bottom are the subject of the original study.
On the full 4 MB KJV, ppm6 reaches 1.648 bpc against bz2's 1.766, and BWT edges ahead too at 1.764. On the first megabyte of enwik8 — a standard benchmark corpus — ppm6 gets 2.226, beating bz2 (2.251) and lzma (2.326).
1. The algebraic codec was crippled by floating point, not by algebra. Doing the identical thing in GF(p) instead of over the reals is 4.2× smaller (35.8 → 8.6 bpc), 20× faster, and exact — and it removes every pathology at once: no quantizer, no delta stream, no escape blocks, no boundary guard, no LAPACK reproducibility hazard. The entire Chebyshev apparatus existed to manage a numerical conditioning problem that does not exist off the real line.
2. Even at its ceiling the idea loses, and it loses before the algebra starts. The
subset and partition bounds contain no polynomials; they are the optimal cost of the
per-block positional model itself. Both sit above plain 5-bit fixed-width coding. And the
remaining gap between GF(p) and those bounds can only be closed by ranking the image of the
coefficient map — which is the combinatorial number system, at which point the polynomial
has vanished. Any injective encoding of a position set costs at least log2 C(L,k) bits;
the polynomial is one inefficient injection among many.
3. Most of what the geometric approach was losing was the alphabet, not the geometry.
Every arm that puts a symbol's rank on an axis inherits an assumption nobody stated: that
alphabetical order is a sensible axis. It is not — q precedes r by scribal convention,
and a fitted curve pays the square of every jump. Reordering the 28 symbols to minimise
Σ Mᵢⱼ (π(i) − π(j))² over the bigram counts cuts the arrangement cost by 78% and takes
the same Bézier arm from 5.538 bpc to 4.647 — past fixed 5-bit packing, though still
short of the order-0 entropy floor. The curve never changed. Only the axis did.
The optimiser is two strategies that compose: the Fiedler vector of the graph Laplacian is the exact optimum of the relaxed problem, so it seeds simulated annealing on the discrete one rather than merely serving as a baseline. On Matthew it produces
alphabetical abcdefghijklmnopqrstuvwxyz_/ cost 41,340,800
spectral quni_/hstodrbaelmfgwypcvkjzx cost 16,334,734
annealed gcw/dnihta_seorlmuyfbpvkjzxq cost 8,938,146
with the vowels collected into a_seo and the rare letters exiled to kjzxq.
Two guards keep this honest. The 28-entry ordering table ships inside every stream, so a
table trained on the corpus it compresses is paid for in the size it reports. And relabelling
symbols is a bijection, so it cannot change the source's order-0 entropy — test_arrangement.py
asserts exactly that. The arm gets smaller because the residuals became more predictable,
never because the text did.
Full working, every number traceable to a CSV: docs/ANALYSIS.md.
pip install -e ".[dev]"
python data/fetch_corpora.py # KJV tiers (public domain) + enwik8 prefixes
# the algebraic codec, in either representation
poly-compress compress --input data/tier2_genesis1.txt --output g.crce --coeff-mode gf
poly-compress decompress --input g.crce --output restored.txt
poly-compress inspect --input g.crce # where did the bits go?
# the full ladder: baselines, bounds, and every codec
poly-compress benchmark --input data/tier3_matthew.txt --codecs --coeff-mode both \
--export-csv bench.csv --export-md bench.md
# a standard corpus, over raw bytes
poly-compress benchmark --input data/enwik8_1mb.bin --byte-level --codecs
# why Q could never simply be lowered
poly-compress benchmark --input data/tier3_matthew.txt --conditioning
# what is the alphabet's ordering worth? four rank axes, verse/chapter/book
poly-compress arrangement --input data/tier1_john316.txt \
--input data/tier2_genesis1.txt \
--input data/tier3_matthew.txt \
--export-csv results/arrangement.csv \
--plot results/trajectory_comparison.png
# parallel speedup, worker telemetry, Amdahl fit
poly-compress scaling --input data/tier3_matthew.txt --workers 1,2,4,8src/polycompress/
coder.py arithmetic coder (Witten-Neal-Cleary), ~1 bit overhead per 200k symbols
models.py adaptive order-N context models
ppm.py PPMC with escapes and exclusion
ctw.py context tree weighting, log-space Bayesian mixture
bwt.py suffix array, Burrows-Wheeler, move-to-front
ordering.py bigram matrix, spectral and annealed alphabet arrangements
bezier.py cubic Bezier over a configurable rank axis, exact integer residuals
arrangement.py the four-axis study, its table and its figure
cgr.py chaos game representation, provably the order-N model in disguise
grid.py the 1D-to-2D index mapping and Hilbert traversal
subset.py the combinatorial bounds, as running codecs
gf.py exact finite-field polynomial roots
cheb.py the original Chebyshev path, kept as the control
encoder.py .crce container, both coefficient modes
decoder.py parsing, root solving, corruption detection
baselines.py entropy floors, combinatorial bounds, stdlib codecs
benchmark.py measurement harness, attribution tables, Pareto front
parallel.py multiprocess encode/decode with worker telemetry
viz/ the infographic set, rendered from the engine itself
docs/
SPECIFICATION.md the .crce v2 format, plus ten defects in the original PRD spec
ANALYSIS.md full results and working
A codec that scores well by cheating is worthless for a study like this, so these are enforced by tests rather than assumed:
- The size ledger is exact. Every bit is attributed to one category, and
test_ledger_conservation.pyasserts the categories sum to the real file size across every tier, block size, and quantization depth. The attribution tables are facts. - Coder overhead is separated from model quality. The arithmetic coder reports
-sum log2 palongside its real output, so a model's loss can never hide behind a loose coder. Measured overhead is under 64 bits on a 4 MB file. - Escapes cannot flatter the result. Every report shows escape rate and
bpc_poly_onlybeside the headline number, and the "escape whenever it's cheaper" guard is off by default. - Losslessness is guaranteed, including across machines. The full 4 MB KJV round-trips byte-identically. GF(p) is exact by construction; the float path escapes any root near a rounding boundary rather than risk a different LAPACK build decoding it differently.
- Skips are reported, never projected. CTW declares a practical size limit and the benchmark prints "skipped" rather than extrapolating a number that would read as measured.
Every frame of the animation is generated by calling the engine, and tests/test_viz.py
re-derives each number it displays and compares — so the pictures cannot drift away from
the code they depict.
python -m polycompress.viz --out docs/media
pytest -q # 1,238 tests, about 95 seconds
pytest -m slow -q # 117 more: full L x Q sweeps and the 4 MB corpus, ~11 minutes
MIT. Corpora are the King James Version via Project Gutenberg (public domain) and prefixes of enwik8.