Turns scholarly PDFs into Markdown whose footnotes are still attached to the sentence they belong to.
PDFs are a messy thing. The format was made to carry our rich book culture into the digital age, never to be read by a machine. A vast treasure of knowledge still sits between dusty covers and inside barely legible OCR disasters, and opening up the latter is a stated aim of the Archilles project. On some volumes, recognising everything correctly is not a matter of trying harder: the evidence is no longer on the page. What is left is patient, meticulous work — fault-tolerant routines for many layouts and many kinds of OCR error, and an honest way of managing the uncertainty that remains. Scriptor is built around that residue rather than around the happy case.
Page-local notes become document-wide Pandoc footnotes, each one attached where its marker stood. The identity of a note is its page and its number — never the bare digit.
A model may turn pixels into text. Whether something is a marker, and which sentence owns it, is decided in code you can audit. For a citation, “usually correct” is the most expensive kind of correct.
Every doubtful marker lands in a review file and an audit sidecar — with its candidates, their scores, and the reason each was scored that way. Under-flagging is the cheaper failure.
A roman-paginated preface stays citable as [p. xiv] instead of being renumbered or dropped. The decoded ordinal only ever serves to put the pages in order.
One volume was scanned once, in one typeface, by one engine — so the same misreading repeats. Scriptor counts the glyphs standing in the document’s own gaps and weights the candidates accordingly.
Put an x in a box in the decision file and run the reflow again. Because no model sits in the loop, the replay reproduces every certain choice untouched.
A PDF breaks its lines where the column ends, not where the thought does. Scriptor takes a histogram of body-line lengths, reads the peak as this volume’s full measure, and walks left until the count falls below a quarter of it. Lines shorter than that edge close a paragraph. Nothing is hard-coded — a wide folio calibrates differently from a narrow octavo.
Before a single line is joined, the pages are sorted: front matter, table of contents, main text, bibliography, register. Reflowing an index would destroy it, so it is left raw. The contents are preserved and linked rather than flattened. The headings that mark these transitions are recognised in five languages.
A running head sits above every page and belongs to none of them. Scriptor finds the lines that recur across pages — by similarity rather than equality, because OCR never repeats itself exactly — and drops them before the reflow joins anything. Words hyphenated across a line break are closed back up.
Borrowed from error propagation in physics: name the uncertainty at its source and carry it through the calculation, rather than discovering at the end that the answer does not add up. An unclaimed footnote is classified by what the candidate search finds in the gap — and the class travels with it, into the review file and the audit.
| Class | Meaning | In the review file |
|---|---|---|
| certain | Marker present, sequence intact. | Set cleanly, no flag |
| suggested | Exactly one plausible candidate. | [?FN:6|&] |
| guessed | Several candidates, or weak evidence. | [??FN:6|&:0.7] |
| orphan | No candidate at all. | [?FN:6] |
Three ingestion paths, the reflow, Pandoc footnotes, the confidence layer, glyph learning within a document and across a corrected corpus, the decision loop, a translation profile, and a DOCX footnote binder. 224 tests, about 3,900 lines of Python. Run against real volumes — Baynes, Byzantium; Snell, Die Entdeckung des Geistes — which is where most of the edge cases came from.
The OCR backend is still a stub: extract/ocr_backend.py raises NotImplementedError. A real backend comes next, chosen by testing candidates against a volume that actually resists — and Scriptor will orchestrate Marker, MinerU, Docling and olmOCR behind that seam rather than reimplement them.
Scriptor is the layer beneath the rest of the workshop. Its output feeds Archilles, where retrieval is grounded in page-level citations, and Archillator, which translates the result — which is what the translate-prep step exists for.
And the gain compounds. Every volume Scriptor gets right enters the Archilles index with its citations intact, so the answers a scholar draws from their own library improve with each book that passes through here. The less the retrieval has to guess, the more it can cite. The other heroes →