Prepares a scholarly library so the machines can understand it — and cite it as a scholar would.
PDFs are a messy thing. The format was made to carry our book culture into the digital age, never to be read by a machine. A vast treasure sits between dusty covers and inside barely legible OCR disasters. On some scanned PDFs, recognising everything correctly is not a matter of trying harder: the information is no longer on the page. What it takes is patient, meticulous work — fault-tolerant routines for many layouts and many kinds of OCR error, and an honest way of carrying the uncertainty that remains.
In the humanities, the argument sits in the body and the evidence sits in the footnotes. Every PDF converter handles the prose acceptably and the apparatus badly, for a typographic reason: footnote markers are tiny superscript digits, and OCR misreads them. The note survives at the foot of the page; its anchor in the sentence does not. Nothing downstream repairs that — no chunker, no translator, no language model. A footnote silently attached to the wrong sentence is worse than a missing one: you find out years later, when you follow the citation. Scriptor makes that anchor hold.
Page-local notes become document-wide Pandoc footnotes, each one attached where its marker stood. The identity of a note is its page and its number — never the bare digit.
A model may turn pixels into text; whether something is a marker, and which sentence owns it, is decided in code you can audit. So correction is a decision, not an edit: put an x in the decision file, run again, and every certain choice replays untouched.
Every doubtful marker lands in a review file and an audit sidecar — candidates, scores, and the reason each was scored that way. And since one volume was scanned once, in one typeface, by one engine, the same misreading repeats: the book itself teaches the run. Under-flagging is the cheaper failure.
A roman-paginated preface stays citable as [p. xiv] instead of being renumbered or dropped. The decoded ordinal only ever serves to put the pages in order.
A PDF does not know where a line ends. Google Books hands over one printed line in four pieces; other engines hand over one piece per word. Scriptor joins them at their baseline. Only then does a histogram of line lengths mean anything: its peak is this volume’s full measure, and a line short of that closes a paragraph. Nothing is hard-coded — a wide folio calibrates differently from a narrow octavo.
Before a single line is joined, the pages are sorted: front matter, table of contents, main text, bibliography, register. Reflowing an index would destroy it, so it is left raw. The contents are preserved and linked rather than flattened. The headings that mark these transitions are recognised in five languages.
A running head sits above every page and belongs to none of them. Scriptor finds the lines that recur across pages — by similarity rather than equality, because OCR never repeats itself exactly — and drops them before the reflow joins anything. Words hyphenated across a line break are closed back up.
Borrowed from error propagation in physics: name the uncertainty at its source and carry it through the calculation, rather than discovering at the end that the answer does not add up. An unclaimed footnote is classified by what the candidate search finds in the gap — and the class travels with it, into the review file and the audit.
| Class | Meaning | In the review file |
|---|---|---|
| certain | Marker present, sequence intact. | Set cleanly, no flag |
| suggested | Exactly one plausible candidate. | [?FN:6|&] |
| guessed | Several candidates, or weak evidence. | [??FN:6|&:0.7] |
| orphan | No candidate at all. | [?FN:6] |
None of it demands a second pass. Taken as it comes, a single run is the best reading the evidence allows: every note kept, the certain ones anchored, the doubtful ones flagged, the unplaceable ones set down at the end of their paragraph — marked, never hidden. The decision loop waits for the volumes you care enough about to correct; the rest are simply done.
Three ingestion paths, the reflow, Pandoc footnotes, the confidence layer with its decision loop, a translation profile, and a DOCX footnote binder. 259 tests, about 4,400 lines of Python. Run against volumes that resist — Seeck, whose text layer arrives one word at a time; Thil-Lorrain, a Google scan in French; Zuckerman, with Hebrew in the notes. That is where the edge cases came from.
The OCR backend is still a stub: extract/ocr_backend.py raises NotImplementedError. Most PDFs already hold a text layer, so this is the exception rather than the road in. When it comes, the engine will be chosen by what it reports — boxes per line, confidence per character — not by how well it reads. A model that returns Markdown returns no page number, and the page number is the citation.
Scriptor is the layer beneath the rest of the workshop. Its output feeds Archilles, where retrieval is grounded in page-level citations, and Archillator, which translates the result — which is what the translate-prep step exists for. Every volume Scriptor gets right enters the index with its citations intact: the less the retrieval has to guess, the more it can cite. The other heroes →