Document extraction · millions of freight records · every discrepancy, not a sample

Freight Invoice Reconciliation at Document Scale

Freight audit is usually a sampling exercise, because reading invoices is expensive and there are millions of them. That framing quietly concedes that most overcharges are never found. The interesting constraint is that the extraction has to be good enough that a discrepancy is a discrepancy rather than a parsing artefact. A false discrepancy costs a carrier relationship, so the system needs to know the difference between 'these disagree' and 'I could not read this'.

Reconstructed

Reconstructed from the résumé rather than from source I hold. The design reasoning is my own account.

Outcome figures are attested; anything used to make the scale legible is illustrative.

2022–2023datamlevaluation

What I owned

Primary architect. I owned the end-to-end design: extraction, the canonical charge model, the reconciliation logic, the confidence gate that separates disagreement from illegibility, and the dispute output the client's team worked from.

open the diagram →

Discrepancy identification: 100%attestedDiscrepancy identification
Annual savings: $2.5MattestedAnnual savings
Records reconciled: MillionsattestedRecords reconciled

9components
2request paths
3load stages
3decisions, argued
1failures, owned

Built onPythonSQL

01Problem 02My role 03Architecture 04Decisions 05What broke 06Impact

The problem

What was actually wrong

The transport management system knows what a shipment was contracted to cost. The carrier's invoice states what they are charging for it. Reconciling the two is arithmetic, and it was not being done at scale because the invoice side arrives as documents rather than as data.

The default response is to sample: audit a few percent, extrapolate, and negotiate on the extrapolation. That works as a negotiating posture and fails as an audit, because the discrepancies that matter are not uniformly distributed and a sample designed for an average tells you nothing about a systematic one.

The hard part is not parsing. It is knowing when the parse is untrustworthy, because a discrepancy raised against a misread line item costs more than the overcharge it imagines.

Before this existed

Sample-based manual invoice audit with extrapolated recovery estimates.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

Documents, not data: Invoice formats vary by carrier and change without notice. Anything that assumes a stable layout will be wrong within a quarter.
A false discrepancy is expensive: Raising a dispute against a carrier over a misread line item damages a commercial relationship. Precision matters more than recall here, which is the reverse of most detection problems.
Complete rather than representative: The whole point is to stop sampling. Any design with a per-document cost high enough to force sampling has failed at the premise.

My role

Primary architect. I owned the end-to-end design: extraction, the canonical charge model, the reconciliation logic, the confidence gate that separates disagreement from illegibility, and the dispute output the client's team worked from.

Designed

The canonical charge model that both the invoice side and the system-of-record side normalise into, so reconciliation compares like with like
The confidence gate: a parse below threshold goes to review rather than to reconciliation, so 'I could not read this' never becomes 'these disagree'
The per-carrier extraction strategy with a shared fallback, so a new format degrades to review rather than to silent misreads
The discrepancy classification that separates rate errors, accessorial charges and duplicate billing, because they are three different conversations with a carrier

Built

The extraction and parsing pipeline across carrier invoice formats
The reconciliation engine and the discrepancy classifier
The confidence-gating logic and the review queue behind it

Led

Set the rule that an unreadable document is an explicit outcome rather than a silently skipped record, which is the difference between a complete audit and one that looks complete

Architecture

How it is put together

Two normalisation paths into one canonical charge model, with a confidence gate between extraction and reconciliation. The gate is the load-bearing component: it is what lets the system claim completeness honestly, because unreadable documents are counted rather than dropped.

The same architecture, in prose

Carrier invoices arrive as documents in per-carrier formats.

Extraction parses each document into line items, using a per-carrier strategy where one exists and a shared fallback where it does not.

A confidence gate evaluates the parse. Anything below threshold routes to a review queue rather than onward, and is counted as unreconciled rather than as agreeing.

The transport management system supplies the contracted cost for the same shipment.

Both sides normalise into a canonical charge model (base rate, accessorials, fuel, adjustments), so reconciliation compares comparable quantities.

The reconciliation engine matches shipments and computes the difference per charge component.

The discrepancy classifier separates rate errors from accessorial disputes from duplicate billing, because each is a different conversation with a carrier.

Classified discrepancies reach the client's audit team as a dispute-ready record with the evidence attached.

Every path, written out (2)· the walkthroughs above, as text

One invoice, reconciled

A carrier invoice arrives, is parsed, gated, compared against the contracted cost, and becomes a dispute.

01
The invoice arrives
A billing document lands in the carrier's own format.
This side of the reconciliation only ever exists as a document.
02
Parse to line items
A per-carrier strategy extracts charge lines, with per-field confidence attached.
Confidence has to be produced at extraction time. It cannot be inferred later.
Fails by: A layout shift produces a confident read of the wrong field.
03
Decide whether this is readable
Below-threshold parses route to review and are counted as unreconciled.
This is the component that makes 'we found 100% of discrepancies' a defensible statement rather than a hopeful one.
Fails by: A threshold loosened to shrink the review queue turns illegibility into false agreement.
04
Normalise both sides
Invoice charges and contracted costs are mapped onto shared components.
Otherwise the two systems disagree over vocabulary rather than money.
Fails by: An unmapped carrier accessorial folds into base rate and conceals a genuine discrepancy.
05
Compare component by component
Shipments are matched and per-component differences computed.
A component-level difference points at a cause; a total that disagrees points at nothing.
06
Name the discrepancy type
Rate error, accessorial dispute or duplicate billing.
Each is a different conversation with the carrier, and the classification is what makes the output actionable.
07
Hand over a dispute, not a difference
The audit team receives the discrepancy with its evidence attached.
Recovery is the outcome. A list of numeric differences is an intermediate artefact.

Read wrongly, with confidence

The document parses cleanly, the mapping has no entry for one of its charges, and two errors cancel into agreement.

01
The invoice parses cleanly
Every field comes out with high confidence. There is nothing illegible about the document.
The confidence gate exists to catch documents the extractor could not read. This is not one of them.
02
The gate passes it, correctly
Confidence is above threshold, so the record goes straight to canonicalisation with no review.
The gate measures legibility, not comprehension. It has no view on whether the charges were understood.
Fails by: This is where the failure becomes invisible: the one control that would have involved a person declines to, for a good reason.
03
An unmapped surcharge folds into base rate
The canonical schema has no component for this carrier's accessorial, so its value is absorbed into the base-rate field.
Mapping is where all carrier-specific knowledge lives, which also makes it where all carrier-specific gaps live.
Fails by: Nothing reports the fold. The record is complete-looking and internally consistent.
04
The comparison agrees
Inflated base rate is compared against a contracted cost that the carrier had also rolled the surcharge into, and the two match.
The arithmetic is correct. The inputs are not.
Fails by: Two errors cancelling produce agreement, which is indistinguishable from correctness at this layer.
05
No discrepancy is raised
The record is dropped from the output as clean, and the carrier is never asked about the charge.
The system reports 100% coverage and is, on this invoice, silently wrong.
Fails by: The fix was a separate arithmetic identity (reconciled component sums against the invoice total), plus publishing mapping coverage per carrier, so thin coverage reads as a gap rather than as silence.

What changes as load grows· the scale stages, as text

The sampling audit it replaced

illustrative

a reviewed fraction of invoices · manual keying

This is the baseline, not a stage of the system: a person reads a subset of invoices and compares them by hand. Coverage is whatever the review budget buys, and every invoice outside the sample is assumed correct.

Gives first: Reviewer hours. The audit is bounded by how many invoices a team can read, which is why most overcharges were never found.

Every invoice, every period

attested

millions of records reconciled per cycle

Extraction runs on everything, the confidence gate routes only what it could not read to a person, and reconciliation compares each line against the contracted cost. Review stops being the audit and becomes the exception queue; its volume is now a measure of extraction quality rather than of audit coverage.

Gives first: The confidence gate's threshold. Set it high and the exception queue swamps the reviewers; set it low and a parsing artefact reaches a carrier as a disputed charge. That threshold is the whole system's tuning knob.

More carriers, more formats

illustrative

new document layouts arriving continuously

The failure mode moves from volume to variety. Each new carrier format degrades extraction confidence until it is learned, so the exception queue spikes per onboarding rather than per period. The canonical schema is what stops that spike from reaching reconciliation logic: a new layout changes the extractor, never the comparison.

Gives first: Extraction generalisation. Throughput is fine; what does not scale is a person confirming that an unfamiliar layout was read correctly before its discrepancies are trusted.

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

Extraction confidence gates reconciliation. Below-threshold documents are counted as unreconciled and routed to review, and the queue depth is published with the results.

Context

The natural implementation drops or best-guesses documents it cannot parse. Both produce an audit that reports completeness it has not achieved.

Alternatives, and why not

Best-effort parse everythingConverts illegibility into false agreement. The invoices that parse worst are disproportionately the unusual ones, which is where the discrepancies live.
Drop unparseable documents and report on the restReintroduces sampling through the back door, with a sample selected by parser difficulty, the worst possible selection mechanism.

Rationale

The claim being made is completeness. A completeness claim is only as good as the accounting for what was not covered.

What it cost

A real manual review queue with a real cost, published rather than absorbed.

Outcome

100% discrepancy identification, with the unreconciled volume visible as a number rather than as an absence.

What it moved in the diagram

Confidence gate
Manual review queue

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

A carrier surcharge nobody had mapped reconciled perfectly

What happened: A carrier's invoices came back consistently clean against the contracted cost while an accessorial charge on them was genuinely unbilled-for. The reconciliation was arithmetically correct and commercially wrong.
Root cause: The canonical mapping had no entry for that surcharge, so the extractor folded it into base rate. Base rate then matched the contracted cost because the surcharge had been quietly added to it. Two errors cancelled, and the output was agreement.
What I did: Reconciled totals were checked against invoice totals as a separate arithmetic identity, which surfaced the carriers where component sums and document totals disagreed. That is a different question from 'do we agree with the carrier', and it is the one that catches mapping gaps.
What changed in the architecture: Mapping coverage became a reported figure rather than an assumption: the share of each invoice's value that landed in a named canonical component, per carrier, published beside the discrepancy results. An unmapped charge now shows up as thin coverage rather than as silence.
What it taught me: A completeness claim needs a second, independent way of being wrong. The confidence gate covered documents the system could not read; nothing covered documents it read incorrectly but confidently, and those are the expensive ones, because they look like success.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

100%attested

Discrepancy identificationComplete rather than sampled

$2.5Mattested

Annual savingsFor the logistics client

Millionsattested

Records reconciledFreight records across carrier formats

Business: An audit that had always been a sample became complete, and the recovery followed from the coverage.
Engineering: A confidence gate between extraction and comparison, which is the component that lets a completeness claim be made honestly.

Leadership and hindsight

What I influenced, and what I would change

What I would do differently

The classifier folded novel discrepancy types into the nearest known category rather than flagging them as novel. I would make 'unrecognised' a first-class output.
Retroactive rate-table updates were a persistent source of phantom discrepancies and I handled them reactively rather than versioning the reference side properly.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

Python: Extraction, reconciliation and classificationThe document-parsing and modelling ecosystem in one place.
SQL: The system-of-record side and the reconciliation joinsThe contracted-cost side already lived in the warehouse.

3 more sections are written and hidden: constraints, leadership, stack.