Accuracy Benchmarks

Results sourced from committed evaluation artifacts. Dates shown are the measurement dates.

OCR Engine Upgrade: PP-OCRv5 → PP-OCRv6

Measured 2026-07-09 on a local RTX 3090 GPU running the production image (paddleocr==3.7.0, CUDA 12.9). Subjects: a real 24 MP HEIC camera document (Polish text, human-transcribed ground truth) and the repository sample invoice/receipt PDFs (pdftotext ground truth). Scoring is order-insensitive multiset recall and precision.

The deployed configuration is PP-OCRv6 medium with text_det_box_thresh=0.5. The lower box threshold recovers faint lines from gloss/perspective regions in camera photos (+20 pt word recall) with zero regression on clean rendered pages.

Metric	PP-OCRv5 (previous)	PP-OCRv6 medium + box_thresh 0.5 (current)
Camera-photo word recall	0.670	0.865 (+29 %)
Camera-photo Polish-diacritic recall	0.562	0.775 (+38 %)
Camera-photo word precision	0.986	0.955
Receipt numeric recall	0.818	1.000 (+22 %)
Dense printed invoice (word recall / precision)	1.000 / 1.000	1.000 / 1.000
Warm GPU latency (RTX 3090, dense page)	~0.55 s	~1.13 s

Word recall on printed rendered pages (invoice and receipt PDFs) is 1.000 on both engine versions — the improvement is concentrated on camera photos where perspective, gloss, and compression stress the detector. Numeric recall (amounts, dates, identifiers) is the closest proxy for field-extraction accuracy, since fields are matched from those tokens.

Photo Engine: Camera-Document Preprocessing

Measured 2026-07-09 (same hardware and image as the OCR engine bench). The photo engine applies geometric unwarping and orientation classification to camera uploads before OCR. The decision was to enable it for IMAGE uploads only — unwarp applied to already-flat rendered pages causes a regression from 1.000 to 0.635 word recall by distorting aligned pixels.

Metric	Without preprocessing (baseline)	With unwarping (camera uploads)
Camera-photo word recall	0.865	0.921 (+6.6 %)
Camera-photo Polish-diacritic recall	0.775	0.854 (+10.2 %)
Camera-photo word precision	0.955	0.961
Dense printed page word recall	1.000	0.635 (regression, photo-only by design)

The unwarping gain is real but requires the OCR service to return the transformed page image so bounding boxes are drawn in the correct (transformed) coordinate space. The photo-only restriction prevents the dense-page regression. The feature is gated by the OCR_PHOTO_TRANSFORMS_ENABLED environment variable.

Multi-Invoice PDF Split Detection

Measured 2026-07-21 using the synthetic labeled evaluation set generated by scripts/gen_split_eval.py and scored by scripts/eval_doc_split.py. The detector uses the embedded text layer (pdftotext) to find page-boundary signals (invoice headers, date-label patterns, document-type keywords) before any OCR or LLM call, so boundary detection is free.

Condition	Boundary precision	Boundary recall	F1	Documents evaluated
Clean text (no OCR noise)	1.000	1.000	1.000	660
Character-level OCR noise (2 %)	-	-	0.969	660

The eval set mixes multi-invoice PDFs with single-invoice negatives. Precision and recall are computed on exact page-index boundary matches. The 2 %% OCR noise model applies per-character confusion, dropping, and swapping (including colon loss, which breaks label regexes). At that noise level the detector achieves F1 0.969, which motivated the suggest-and-confirm UX: boundaries are offered to the user for review rather than applied silently.

Methodology Notes

All numbers are sourced from committed evaluation artifacts in the repository (docs/ocr-calibration/*.json, docs/OCR_IMAGE_CALIBRATION.md). Split-detection results are in docs/ocr-calibration/split_detection_results.json. No numbers are estimated or extrapolated.
OCR scoring uses order-insensitive multiset recall and precision: the metric counts whether each ground-truth word token appears in the OCR output (recall) and whether each OCR output token appears in the ground truth (precision), without penalizing for reading-order differences. This is a better proxy for field-extraction accuracy than sequence-similarity metrics, which are sensitive to line segmentation changes that do not affect token content.
Camera-photo ground truth is a human transcription of the test document (docs/IMG_2375.HEIC). Printed-page ground truth is derived from pdftotext, the same tool used in the digital-PDF fast path.
Split-detection precision and recall measure exact page-index matches on synthetically generated multi-invoice PDFs with known boundaries. The eval set covers a range of invoice types, date formats, and vendor names.
Benchmark re-runs after any engine change can be triggered with the scripts in scripts/. Fresh results land in dev-artifacts/ (gitignored) and are committed after review.

Questions about the methodology or results: hello@synairo.com