Building OCR Model Arena, then finding out where the first baseline actually breaks

OCR comparisons are easy to make look scientific: send a few pages to several models, eyeball the Markdown, and turn the outputs into a leaderboard. The difficult part is making every page, corruption, prompt, output contract, timing, failure, and retry mean the same thing. OCR Model Arena is the evaluation system I built to make that comparison reproducible. This is a report on the working platform and its first measured baseline, not a claim that all of its model adapters have already been benchmarked.

• What is real
• Architecture
• Dataset
• Corruptions
• Scoring
• Exact tests
• Results
• Model comparison
• Limitations
• Next benchmark

Start with the evidence boundary

The most important result is not a score. It is a precise boundary around what the repository proves. A registry entry proves that an adapter can be configured. An injected runtime test proves that the adapter understands an upstream response shape. Neither proves that the heavyweight package, checkpoint, CUDA kernel, prompt, or serving stack works in the target environment. I keep those levels separate:

The code repository is private during this first integration pass, so the article ships a public, machine-readable benchmark summary and robustness table, but not a public source link. The status should change only after the real checkpoint and cloud rows exist.

The system: page in, auditable experiment out

The core interfaces know nothing about Modal or RunPod. A model loads and predicts; a dataset yields samples; a metric compares typed truth with a typed prediction; and a backend submits jobs. Provider code only transports the same portable job envelope. This keeps a model usable on a laptop, in a persistent service, or in a cloud worker without copying its inference logic into three places.

dataset adapter
  image + SHA-256 + MIME + plain truth + Markdown + ordered blocks
        │
        ▼
seeded corruption engine ──▶ transformed PNG + complete provenance
        │
        ▼
deterministic OCR job id = hash(dataset, truth, model, backend,
                                corruption, metric contract, seed, repeat)
        │
        ├──────── local batch ────────▶ reused in-process model
        ├──────── Modal batch ────────▶ versioned deployed worker
        └──────── RunPod batch ───────▶ queue endpoint / persistent worker
                                           │
                                           ▼
raw response + plain text + optional Markdown + timings + model metadata
        │
        ▼
normalization ──▶ metrics ──▶ deterministic failure flags
        │
        ▼
atomic page JSON + separate raw output + rebuildable Parquet
        │
        ▼
leaderboard + robustness + failures + latency/cost Pareto reports

A benchmark expands the Cartesian product of pages, models, corruption configurations, inference settings, and repeats. Existing records are parsed and identity-checked before they count as complete. Local workers reuse a loaded model. Remote workers accept batches so one page does not pay for one GPU container startup. Cloud submissions persist an uncertainty intent and then the provider handle before polling; if the process dies in the dangerous window where a provider may have accepted paid work but no handle was committed, the runner stops for reconciliation instead of silently submitting it again.

Storage treats individual JSON records as the transactional source of truth. Raw outputs live beside them, and Parquet is regenerated as an analytical view. Writes use a temporary file, file synchronization, atomic replacement, and directory synchronization. A filesystem lease prevents two local processes from driving the same run at once.

A tiny dataset that makes the whole repository runnable

The first dataset is synthetic on purpose. It avoids a multi-gigabyte download and gives the evaluator exact text, Markdown, block type, order, and bounding boxes. The canonical corpus contains eight 1275 × 1650 RGB pages at 300 DPI, rendered with DejaVu Sans, Bold, and Mono. Every page has a fixed research header and a page footer.

Generation uses a dedicated deterministic random stream derived from the global seed and template index. Tests generate the seed-42 dataset twice in separate directories and compare complete truth objects and image hashes. Prepared images are still the reproducible artifact: font files, Pillow, and PNG encoders can change bytes across environments even when the pseudo-random sequence is identical.

Robustness is a controlled experiment, not an “ugly scan” folder

Every transformation validates and stores its resolved parameters. A typo fails instead of quietly falling back to a default. Each step records its own seed and input/output pixel hashes, which means a failure can be replayed exactly from the original page. The measured sweep used one corruption family at a time:

Within one family, parameter lists form a Cartesian product; across families, this run uses one_at_a_time. That isolates a blur curve from a rotation curve. A separate cartesian mode can compose families for stress tests, but its scores answer a different question.

Crop and occlusion preserve the original full-page reference. I treat those as document recovery tasks—“recover the page despite missing pixels”—rather than visible-only OCR. The report labels this policy, and 32 records in the run changed visibility. A later visible-only track should derive its reference from block bounding boxes and a visibility mask.

Scoring without letting output formatting choose the winner

Text normalization uses Unicode NFKC, removes invisible formatting characters, and collapses whitespace and line breaks. It preserves case and punctuation. Line-wrap differences should not decide a model comparison; changing Case to case should.

CER = Levenshtein(reference characters, prediction characters)
      ─────────────────────────────────────────────────────────
                    number of reference characters

WER uses whitespace-delimited words instead of characters.
Normalized similarity = 1 - edit_distance / max(reference_len, prediction_len)

The default suite includes CER, WER, normalized edit similarity, exact match, reading order, heading/list signature, table shape, and Markdown block-sequence similarity. CER and WER are reported both as page-level macro means and as micro scores that sum edit counts and reference lengths. Structure metrics mark whether the reference makes them applicable; a page with no table cannot inflate table accuracy merely because the prediction also contains no table.

Structure scoring is deliberately described as heuristic. Reading order fuzzy-matches ordered reference blocks to locations in the output and measures the longest increasing subsequence. Table similarity currently compares table count and row/column shape, not cell transcription. Markdown similarity compares block-type order, not semantic content. There is not yet a geometric bounding-box IoU score or a specialized formula-recognition score.

Failure flags are also deterministic diagnostics rather than a learned hallucination oracle. They cover empty and excessive output, repeated sequences, duplicate paragraphs, missing headings, unexpected tokens, broken order, malformed tables/Markdown, and complete generation collapse. Raw output is always retained so a flag can be audited.

Exactly how I tested it

The repository is pinned to Python 3.12 through uv. The local benchmark ran on CPython 3.12.12, Linux x86-64, Tesseract 4.1.1 with Leptonica 1.82.0, English, OEM 1 (LSTM-only), PSM 3 (automatic page segmentation), plain-text output, and a 120-second timeout. It used a local CPU, batch size one, one in-flight batch, seed 42, and one repeat.

uv sync --locked --extra dev

uv run ocr-arena datasets prepare synthetic-v1 \
  --root data/synthetic-v1 \
  --count 8 \
  --seed 42

uv run ocr-arena sweep \
  --config configs/robustness.yaml \
  --artifacts artifacts

The sweep expansion was checked before inference:

8 unique pages × 1 model × (34 corruptions + clean) × 1 repeat
                                                           = 280 evaluations

Every request rereads and verifies the expected SHA-256 and MIME type before execution and rejects images above 50 megapixels. Tesseract starts one subprocess per page, so its measured latency includes process startup. The report distinguishes warm model inference from driver end-to-end timing where an adapter can measure both; this baseline has no persistent model server, network, queue, cold-start, token, GPU, or VRAM component.

The automated suite

The final checkout passed 111 tests with 82% statement coverage, plus Ruff formatting/linting, strict mypy over 52 source files, a frozen dependency-lock check, package build, Docker Compose validation, two Docker image builds, and a one-page Docker Tesseract smoke. GitHub Actions repeats the locked Python checks and a two-page fixture CLI benchmark on every push.

Cloud tests use fake provider transports. The environment had no Modal or RunPod credentials, and the local NVIDIA installation had a driver/library mismatch, so I did not run a GPU model and do not report cloud latency, peak VRAM, or cost.

What the first 280 OCR evaluations said

The run completed all 280 jobs with zero execution errors in 163.75 seconds wall time. A resume immediately afterward skipped all 280 OCR jobs, though it still regenerated the corruptions before discovering the durable records, taking 46.78 seconds. That inefficiency is visible and should be fixed by checking experiment identity before materializing variants.

I exclude the report's legacy gpu_seconds_per_page field here. In this CPU-only run it contains mean inference seconds, not GPU utilization. Cost and VRAM were null.

Clean documents: one layout failure dominates the average

On the eight clean pages, macro CER was 0.06586 and normalized similarity was 0.93414. Seven templates were transcribed with a combined macro CER of 0.00776. The two-column page alone had CER 0.47253.

Clean-page CER; bars are scaled to the worst page, not to 1.0.

Tesseract did not mainly misread the characters in the column page. It interleaved the left and right lines, while the reference reads the complete left column and then the complete right. The reference and prediction both contain 455 characters, but their global order differs:

Column one describes acquisition.
Every source page receives a content
hash before any transformation is ...

Column two describes evaluation.
Text accuracy, document structure,
latency, memory, and cost are ...

Column one describes acquisition. Column two describes evaluation.
Every source page receives a content Text accuracy, document structure,
hash before any transformation is latency, memory, and cost are ...

The reading-order score for that clean page was 0.6. This is the concrete reason the arena keeps transcription and order metrics separate: a recognizer can see nearly every glyph and still produce a document that is wrong to read.

Corruption response

Large clean synthetic text is unsurprisingly tolerant of JPEG compression and random pixel noise. Strong geometric and low-frequency image changes hurt much more:

The full 34-condition table is available as CSV. Do not over-read monotonicity: there is one page per type, one seed, and one repeat. Occlusion location changes with its seed, so a larger occluder can happen to cover less important text. Document content and corruption are also crossed only eight times, making these curves useful diagnostics rather than confident causal estimates.

How the model families differ—and what that might change

A fair arena cannot pretend all registry names describe the same kind of system. The bare PaddleOCR-VL or GLM visual-language checkpoint is not the same product as its full page pipeline, which also detects regions, crops them, recognizes elements, and reconstructs reading order. The registry gives those variants different identities.

Upstream descriptions and links were verified Aug 12, 2026. Upstream benchmark claims are not reproduced here and are intentionally not mixed into this project's result table.

Six mechanisms that could explain a future difference

1. Segmentation boundary. A detector/recognizer works region by region; a page VLM can condition on the whole document; a page pipeline combines both. Missed regions, crop order, and reconstruction can dominate the recognizer itself.
2. Visual resolution. A 1288-pixel page render, dynamic high-resolution crops, and a downsampled whole page expose very different numbers of pixels per glyph. Tiny text may improve exactly where compute and context fragmentation increase.
3. Decoder behavior. Classical and discriminative recognizers do not generate a long response token by token. VLMs can use language context to repair ambiguous text and express Markdown, but can also invent plausible content, repeat, or stop early. Latency scales with output length.
4. Task-specific training. OCR VLMs are trained for documents, tables, formulas, and reading order. A general VLM may understand broader semantics but have weaker glyph-level and layout priors. Model size alone does not settle this.
5. Pre/postprocessing. Orientation classification, dewarping, retry policies, header removal, prompt construction, and Markdown cleanup can move the score while the checkpoint stays fixed. They belong in the named model identity.
6. Serving system. A local subprocess, direct CUDA generation, and a persistent vLLM/SGLang endpoint have different cold starts, batching, queues, network overhead, and memory accounting. Warm inference and end-to-end latency must remain separate.

What these results do not prove

• There is no empirical Tesseract-versus-VLM comparison yet and no meaningful multi-model Pareto frontier.
• The corpus has eight unique synthetic pages, one page per document type, one repeat, one language, and no handwriting or photographed real document.
• OmniDocBench, OCRBench, and other external benchmarks were not downloaded or run.
• Tesseract 4.1.1 was the installed binary; upstream identifies 5.x as the current stable major series.
• No GPU, Modal, RunPod, quantization, dtype, VRAM, generated-token, or dollar measurement exists.
• The structure scores are heuristic and do not yet include bounding-box IoU or a dedicated formula metric.
• Crop and occlusion use the full original truth, which measures recovery rather than visible-only OCR.
• The result uses one seed, so it has no repeat variance, confidence interval, or paired bootstrap uncertainty.

The source-snapshot caveat

The saved run is complete and its per-page images, hashes, transformation provenance, outputs, and records agree internally. However, it was produced during development before the source was committed. Its recorded arena-source digest does not match the current private repository commit, and no commit in the short Git history matches that digest. The English Tesseract trained-data hash was also not persisted. The identical command resumed all 280 evaluations at the time, but running it now would create a new run identity.

I therefore call this a real development benchmark, not an exact release benchmark reproducible from a public commit. The correct fix is to rerun from a tagged source revision and pinned container, capture the language-data hash, and publish the manifest and result bundle together.

The next benchmark I would trust

1. Tag the evaluator, freeze the dependency lock, and record the CPU/GPU worker image digest and immutable checkpoint revision.
2. Rerun the Tesseract baseline on that tag, using current Tesseract 5.x as a separately named row rather than overwriting 4.1.1.
3. Add a larger synthetic corpus and a license-reviewed, held-out real-document set. Keep benchmark test examples out of any future training data.
4. Deploy one dependency-isolated worker each for PP-OCR, PaddleOCR-VL, GLM-OCR, DeepSeek-OCR, olmOCR, and the generic VLM control.
5. Run a plain-transcription track and a structured-parsing track with identical fixed images and explicit prompts.
6. Use deterministic decoding where supported and at least three repeats for generative systems; report page-paired bootstrap confidence intervals.
7. Warm each runtime, then record cold-start, warm model, batch-service, and driver end-to-end latency separately on one fixed GPU SKU.
8. Attach a dated price source, token counts, VRAM scope, and cost coverage before drawing a three-axis accuracy/latency/cost frontier.
9. Mine the worst paired disagreements, inspect the raw outputs, and only then form claims about why one architecture wins.

A standalone site becomes valuable at that point. The article is the durable narrative and methods record; the separate benchmark explorer should let a reader choose model revision, dataset, document type, corruption, metric, hardware, and pricing snapshot, then inspect individual page failures. The public summary JSON is deliberately shaped as the first static data source for that future explorer.

References

1. Tesseract OCR upstream repository and version documentation.
2. PP-OCRv6 official architecture documentation.
3. PaddleOCR-VL 1.6 official documentation.
4. GLM-OCR official model card.
5. DeepSeek-OCR official repository.
6. olmOCR official toolkit and model documentation.
7. Hugging Face's SmolVLM 256M/500M announcement and model overview.