Confidence Scores, Not Guesses: Designing an Extraction Pipeline That Flags Instead of Fabricates
The short answer
An extraction pipeline that always returns a value and never a flag isn't more accurate — it's hiding its uncertainty. Faced with a smudged digit, a model optimized to fill every cell will pick the most plausible character rather than admit it isn't sure, and that plausible-looking wrong value is more dangerous than a blank, because a blank gets checked and a guess doesn't. Good extraction design treats "I'm not confident" as a valid, first-class output — with a confidence score and a reason — instead of optimizing to eliminate blanks.
Vendors love a screenshot of a fully filled-in spreadsheet with zero blank cells. It's the wrong thing to be proud of. A completely full output isn't evidence the tool got everything right — it's evidence the tool never admits when it didn't. Real handwritten documents have genuinely ambiguous, genuinely illegible, genuinely missing values in them. A pipeline that returns zero blanks on real data is a pipeline that's guessing somewhere and not telling you.
The blank-cell fallacy
Extraction tools get judged, informally, by how complete their output looks. A filled cell reads as success; a blank reads as failure. That incentive pushes a model toward always producing something — the most statistically plausible character for an ambiguous digit — rather than toward admitting uncertainty. The output looks more impressive and is, in a specific and costly way, less honest.
Why a confident guess is worse than a flagged gap
A blank cell or a flagged value gets scrutiny — someone looks at it, checks the source, and resolves it. A confident, plausible-looking guess gets none of that, because nothing about it signals a problem. It's indistinguishable from a correct value right up until it corrupts an analysis, and by then it's often too late to trace back to the page it came from. The errors that do the most damage are the ones that never look like errors.
What good confidence scoring looks like
A single headline accuracy number for the whole document tells you almost nothing about which specific cells to trust. Useful confidence scoring is granular — attached to each field or cell, not the document as a whole — and it comes with a reason, not just a number: illegible handwriting, an ambiguous shorthand token, or two sources that disagree with no rule to break the tie. A reason tells a reviewer exactly what to check. A bare number just tells them to worry.
Designing a pipeline that flags instead of fabricates
- Make "not confident" a valid output. Give the model explicit permission to flag uncertainty instead of treating anything short of a clean answer as a failure to route around.
- Attach a reason to every flag. A bare confidence score is hard to act on. A reason tells the reviewer exactly what to look at.
- Route flags to human review, not silent autofill. A low-confidence value should stop for a person, not get filled in and shipped anyway.
- Track flag rate as a first-class metric. A pipeline with a suspiciously low flag rate on real, messy data isn't more accurate than one that flags honestly — it's probably just not flagging enough.
How to spot a tool that doesn't do this
Feed it something genuinely illegible or ambiguous on purpose — a digit you can't confidently read yourself — and watch what comes back. If it returns a clean, confident-looking value with no flag and no lower confidence signal, you've found a tool that's optimized to look complete rather than to be honest about what it doesn't know.
NoteMerge is built around exactly this trade. Low-confidence cells are flagged for your review with a reason attached, instead of filled in with the model's best guess. In independent validation against hand-built targets across three multi-page studies, it reached 97.6% cell-level agreement — and the remaining gap was flagged, not silently wrong. That pairs with the determinism boundary between judgment and rendering: a flagged decision is logged data, resolved once by a human, then rendered the same way every time after.
See what gets flagged, not just what gets filled in
Snap photos of your pages and get a reconciled spreadsheet back, with every low-confidence value flagged for review. Free 14-day trial, no card required.
Join the waitlistFrequently asked questions
What is a confidence score in AI data extraction?
A per-value signal for how sure the pipeline is about a specific extracted value, distinct from an overall accuracy number. A good confidence score is tied to a specific field or cell, not just a headline stat for the whole document.
Why would flagging a value as "unclear" be better than guessing it?
Because a flagged value gets checked and a plausible wrong guess doesn't. The guess looks exactly like a correct value to anyone reviewing the output, which means it's the error most likely to survive review and end up baked into an analysis.
How does an extraction pipeline decide when to flag something?
It should flag whenever the evidence genuinely supports more than one reading — a smudged digit that could be two different numbers, a value with no source to confirm it, or two sources disagreeing with no documented rule to break the tie. The flag should come with a reason, not just a score.
Isn't a tool that never returns blanks more accurate?
No — it's just hiding its uncertainty better. A tool that always fills every field is optimized to look complete, not to be correct. The number to watch is the flag rate on genuinely ambiguous data, not the absence of blanks.
What should happen to a flagged, low-confidence value?
It should route to a human reviewer with the reason attached, alongside the original source, so a person can resolve it once — and that resolution should feed back into the record as a logged decision, not just a corrected cell.
Does NoteMerge flag uncertain values instead of guessing them?
Yes. Low-confidence cells are flagged for review with a reason rather than filled in with a best guess. In independent validation, the tool reached 97.6% cell-level agreement against hand-built targets — and the remaining gap was flagged, not silently wrong.