How Occlira benchmarks its PII detection
Accuracy you can’t measure is a promise you can’t keep. So we hold every release of Occlira to a demanding, fully-transparent benchmark built for the hardest surface we handle: real-world legal documents. This page explains exactly how that benchmark works and what it tells us.
At a glance
| What we measure | Result on our legal benchmark |
|---|---|
| Sensitive values located (any PII found at all) | ~97% |
| Values assigned the exact expected type | ~95% |
| National IDs handled correctly (found, never mis-labelled) | ~98% |
| Contact / network / financial IDs (email, phone, URL, IP, IBAN, card) | ~100% |
| Overlap-level F1 across every PII type in the benchmark | ≈ 0.90 |
Every figure comes from running the real Occlira engine end-to-end over 200 synthetic legal documents with ~5,500 human-defined PII annotations — then confirming it holds on a second, completely independent document set the engine had never seen.
1. A corpus that looks like the work
Generic PII test sets (a name here, an email there) don’t resemble the documents our users actually process. So we authored a corpus that does: 200 fictional but realistic legal documents, split into two independent 100-document sets.
- 10 practice areas — litigation, employment & HR, healthcare, banking & tax, real estate, family law, corporate/M&A, IP & data protection, immigration & criminal, and insurance/consumer/estate.
- 10 jurisdictions — US, UK, Germany, France, Italy, Spain, Finland, Cyprus, EU and international.
- The PII entity types this benchmark scores — from people, organisations and addresses to 20 national and structured identifiers (SSNs, NHS/NIN numbers, passports, tax IDs, fiscal codes, IBANs, payment cards, and more).
- Real document shapes — court captions, letterheads and signature blocks, field/value intake forms, tables, embedded email threads and multi-page contracts, ~150 to ~2,000 words.
- Realistic density — from sparse notices with a handful of identifiers to dense KYC forms and estate schedules carrying 30–40.
Every name, number, address and identifier is synthetic. Structured identifiers are drawn from a fixed pool of valid-checksum values, so a detection reflects the recogniser’s real ability — not a lucky random digit.
2. Ground truth that can’t drift
Each document is written with the sensitive spans marked inline. A build step strips the markers and records the exact character offset of every PII value. Because the plain text is reconstructed marker-by-marker, the ground truth is correct by construction — it can never disagree with what the engine actually receives. The result is ~5,500 precisely-located annotations we score against.
We also plant deliberate distractors that a good detector must ignore: docket and case numbers, statute citations, invoice and matter references, monetary amounts, section numbers, dates, and product code-names. Getting these wrong is just as measurable as getting PII right.
3. We test the real engine, not a stand-in
The benchmark drives the production Occlira detection pipeline over the very same interface the app uses — full NER plus the structured-recogniser layer, deduplication, boundary cleanup and false-positive filtering. Nothing is mocked or simplified. What we measure is what ships.
4. What “correct” means
We score at two levels and report both, because they answer different questions:
- Localization — did the engine find the sensitive span at all? This is the number that matters most for privacy: a located value gets redacted.
- Typing — did it assign the exact expected type (e.g. French social-security number vs. Spanish tax ID)?
For structured identifiers we also report the metric that reflects real product behaviour: “handled correctly.” An identifier counts as handled correctly when it is found and given either its precise type or an honest, neutral one — and is never given a wrong label. On our benchmark, national identifiers reach ~98% handled-correctly with essentially zero mis-labelling — the engine would rather say “an ID document” than guess the wrong country.
5. What the numbers say
- Contact, network and financial identifiers are effectively solved — email, URL, phone, IP address, IBAN and payment card are detected at ~100% span and type recall, with high precision, across prose, labelled fields and tables.
- People, organisations and locations are localised at 97–99%, with the overwhelming majority typed correctly.
- National and structured identifiers — the hardest category in multi-jurisdiction legal text — are located at ~97% and handled correctly at ~98%, with mis-labelling driven essentially to zero.
- Overlap-level F1 across every type is ≈ 0.90.
6. The test that matters most: generalisation
It’s easy to look good on the data you tuned against. So we do something stricter: after improving the engine on the first 100-document set, we generated a second, fully independent 100-document set — new parties, new facts, new document structures, authored separately — and ran the engine on it cold. The core results held: localisation ~97%, correct-type recall ~95%, and national identifiers again handled correctly ~98% with near-zero mis-labelling. That’s the signal we care about — the accuracy is a property of the engine, not of any one test.
7. Transparency and scope
We publish our method because transparency is part of the product. A few honest notes on what this benchmark is — and isn’t:
- It is a synthetic, deterministic, reproducible regression benchmark. Every value is fictional; no real personal data is used or exposed.
- It is designed to be re-run on every change, so accuracy can only move in one direction release to release.
- It measures text-native legal documents. OCR, scanned pages and audio go through their own dedicated evaluations.
- It is our internal engineering benchmark, not an independent third-party audit — and we’re clear about that distinction.
8. Reproducibility
The corpus generator, the exact-offset builder, the engine runner and the scoring scripts are kept together and version-controlled. Any result on this page can be regenerated from the same two commands, against the same frozen engine — so the numbers are auditable, not anecdotal.
See it on your own files
Free for 14 days on Windows 10 and 11. No account, no key. One-time license, no subscription.
All identities, identifiers and documents in the Occlira benchmark are fictional synthetic test data. Figures reflect Occlira’s internal legal-document benchmark and are provided for transparency into our detection quality.
Related: anonymize before ChatGPT · for law firms · data & privacy practices