Why We Never Let an LLM Write the Final Number
The short answer
LLMs are useful for judgment calls — resolving which of two conflicting values is correct, or whether an unlabeled page belongs to a given record — because those require reasoning over ambiguous evidence. They're a liability for the very last step: writing the final number into the output. LLM output isn't guaranteed to be identical across runs, and reproducibility is non-negotiable for any dataset you might need to defend. The fix is a hard line: LLM for decisions, deterministic code for rendering. Re-running the same input should always produce the same file.
"AI-powered" gets read as "an LLM touches every step," including the number that lands in your spreadsheet. For anything you need to trust, that's exactly backwards. The step that most needs intelligence is deciding what a document means. The step that most needs to be boring, predictable, and the same every single time is writing that decision down.
Extraction is two different jobs, not one
Turning a messy document into a clean dataset is really two jobs wearing one label. The first is judgment: reading ambiguous evidence and deciding what it means — does this page belong to that record, which of two conflicting values is correct, is this genuinely illegible. The second is mechanical: taking a decision that's already been made and writing it into a cell, applying a formula, formatting a file. Treating those as one undifferentiated "AI step" is where reliability problems start.
What LLMs are actually good at
The judgment half is exactly where an LLM earns its keep. Deciding whether an unlabeled continuation page belongs to a given record, or which of two shorthand tokens should win a conflict, requires reasoning over context — not a lookup, a judgment call. That's a fundamentally different task than reading pixels, and it's the one worth spending model capability on, paired with a confidence score and a written reason for anything a human should double-check.
Why an LLM shouldn't write the final number
Two problems show up the moment an LLM is the last thing to touch a value. First, non-determinism: the same input isn't guaranteed to produce the same output, especially once you account for model updates behind an API you don't control. "The model wrote a slightly different value this time" is a fine failure mode for a chat assistant and an unacceptable one for a dataset you'll be asked to reproduce. Second, silent helpfulness: a model asked to "clean up" a value can round it, reformat it, or normalize it in ways nobody specified — useful in a conversation, corrosive in a pipeline where every transformation needs to be intentional and auditable.
Drawing the line: decisions vs. rendering
The architecture that actually works keeps a hard boundary between the two jobs. An LLM assists with the decision — with a confidence score and a reasoning trace — and that decision gets logged as data, not as prose buried in a chat transcript. Deterministic code then takes the logged decision and renders it: applies the fixed formula, writes it to the fixed cell, formats the file the same way every time. Feed the same logged decisions back through the renderer, and you get the same file, byte for byte. The intelligence lives entirely upstream of the part that has to be boring.
A simple test for whether a pipeline respects this
- Run the same input through the pipeline twice. Use the identical source document both times.
- Diff the two output files. Compare every value, not a summary or a spot check.
- Treat any unexplained difference as a failure. If anything differs beyond a timestamp, the determinism boundary isn't actually enforced — a non-deterministic step is still touching the final output.
NoteMerge is built on exactly this split. Reconciliation — identity, conflicts, confidence — may use an LLM, and every decision it makes is logged with its reasoning. Rendering the workbook from those logged decisions is deterministic code, full stop. Run the same notebook through twice and you get the same file both times. In independent validation against hand-built targets across three multi-page studies, it reached 97.6% cell-level agreement — a number that means something precisely because the pipeline behind it is reproducible.
Get the same file every time you run it
Snap photos of your pages and get a reconciled spreadsheet back — rendered deterministically, so it's reproducible on demand. Free 14-day trial, no card required.
Join the waitlistFrequently asked questions
Can an LLM be trusted to produce the same output twice?
Not reliably. LLMs generate output through a process that isn't guaranteed to be identical across runs, especially as models are updated behind an API. For a chat response, that's harmless. For the final number in a dataset you need to reproduce, it's disqualifying.
Why is reproducibility important for a data extraction pipeline?
Because you need to be able to defend any value in the output — to a reviewer, an auditor, or your future self doing a re-analysis. If re-running the same source document can produce a different result, you can no longer prove the output reflects the source, only that it reflected the source once.
What should an LLM be used for in a data pipeline, if not the final output?
The judgment calls: deciding whether an unlabeled page belongs to a given record, which of two conflicting values should win, or whether a value is genuinely illegible. Those require reasoning over ambiguous evidence, which is what LLMs are good at — as long as the decision is logged, not just acted on silently.
What's the difference between an LLM making a decision and an LLM rendering the output?
Making a decision means resolving an ambiguous judgment call and logging the reasoning behind it. Rendering means taking an already-resolved, logged decision and writing it into the final file — applying fixed formulas and formatting with no room for the output to vary. The first benefits from an LLM's reasoning; the second should never touch one.
How do I test whether an "AI-powered" tool is actually deterministic?
Run the exact same input through it twice and diff the two output files, cell by cell. If anything differs beyond a timestamp, the tool doesn't enforce a real boundary between judgment and rendering — it's letting a non-deterministic step touch the final output.
Does NoteMerge use an LLM to produce the final spreadsheet values?
An LLM helps make the judgment calls during reconciliation — identity, conflicts, confidence — and every one of those decisions is logged. Rendering the workbook from those logged decisions is deterministic code. Re-running the same notebook produces the same file every time.