Portfolio of

Mostafa Anwari
00

Transcript Archive

Decades of a newspaper survive only as scans. This is the system that reads them: OCR, article reconstruction, an index, and a question-answering layer that cites the page for every claim. The page below walks through it step by step, on one real front page.

Runs in production behind a daily cost cap, per-IP rate limits, prompt-injection defenses and architectural tests that enforce import boundaries. Each phase and component was selected through real trial before assembly.

Python · Neon Postgres · Next.js 16

The Ask the Archive screen: a research prompt with suggested questions about 1955 majors, 1968 campus protests and Earth Day 1970, noting answers cite primary sources across 351 editions and 11,705 articles.
ask the archive — cited answers
The edition reader open to The Transcript for Wednesday, January 13, 1960, showing scanned front-page stories beside a section index, a weather report and a January 1960 top-ten panel.
edition reader — page by page
The pipeline end to end, drawn in ink on the ground: the scan, OCR, structure, merge, enrich, triage and publish, the reader, the index, ask and answer.
The pipeline end to end, drawn on the ground.
01

It starts with a photograph of a page.

A scanned front page, as the scanner delivered it. To the computer it is pixels: nothing on it can be searched, quoted or counted.

02

It works one edition at a time.

Every page of an edition shares one date, and that date keys everything the pipeline makes for it: text, images, articles, and the citations that point back to them. Editions queue in any number.

03

Every page takes the same path.

One year or a century, one paper or many: each edition is one run of the same pipeline. Runs are independent, so an archive grows by adding editions, not by changing the code.

04

Every word gets a position and a confidence.

Google Document AI reads a lossless greyscale copy of the page. Nothing is sharpened, upscaled or guessed. Words under 80 percent confidence are flagged, like the one in red, instead of silently kept.

05

A model rebuilds the articles from the words.

Gemini turns the OCR words into articles, ads, headlines, bylines and captions, in reading order. The page image is for layout only: the prompt forbids adding any word read from the picture.

06

Stories are joined across pages by ID, never by rewriting.

The model may only say which fragments belong together. Code makes the join, and repairs a seam only where the two ends match at 90 percent or better. Anything less stays marked unresolved.

07

Ads get categories, edge cases get a second look, and the edition goes live in one move.

Every ad is classified, and any phone, address or price the ad’s own text does not contain is cleared. Triage sends only rule-selected items back to the model, which may change a type or category and nothing else. The edition is validated four times and swapped live in a single rename, with the previous version kept for rollback.

08

Each edition comes back as a paper you can read.

Sections, headlines, bylines and every article in full, laid out from the structured file. The scan stays behind as the source. The text is what you read and search.

09

Articles are cut into overlapping passages and embedded.

Passages of about 3,200 characters overlap by whole sentences, so no idea is split at a boundary. Each passage and each photograph becomes a vector in Postgres with pgvector. An index build is immutable: a change means a new build, activated whole.

10

Search runs two ways and merges the results.

A vector search by meaning and a full-text search by exact words run on every question, and reciprocal rank fusion merges the two lists. A Gemini judge scores each candidate against the question from 0 to 10 and keeps 4 and up.

11

Every claim cites a page, or it is cut.

The generator may only state what its sources say, and each claim carries a citation number that resolves to a page. A citation that resolves to nothing is removed in code, not by prompting. If the best sources score under 5, it declines instead of guessing.

12

One pipeline, from scan to cited answer.

The steps above run once per edition, in order, with validation between each hand-off. Any archive with dated scans can go in. The links below open the live archive built from this one.