Skip to content
Kiran
← Work

Hybrid ensemble · Airflow · Snowflake · a gold set that lied

Freight Pricing Anomaly Detection

Three detectors with different failure modes, weighted and deliberately asymmetric: a hierarchical statistical detector that only scores above the mean, carrying overcharge, and two bidirectional ML detectors carrying everything else. The statistical tier falls back through four levels of business-context specificity with a confidence multiplier at each, and uses a median-absolute-deviation z-score rather than a standard-deviation one, because the distribution it is measuring is contaminated by the very anomalies it is looking for. The part I would put on a slide, though, is the evaluation: the gold set had been recording the model's own predictions as reviewer ground truth, and the fix was rebuilding the label pipeline around what reviewers wrote rather than what the interface pre-ticked.

2025–2026mlevaluationdataeconomics

What I owned

Owner. Designed and built the pipeline, the ensemble, the feature engineering, the feedback loop and the evaluation, including the audit that invalidated the first version of the evaluation.

Reduction in manual pricing review hours
90%attestedReduction in manual pricing review hours
Share of investigator effort directed at the top 10% highest-dollar deviations
90%attestedShare of investigator effort directed at the top 10% highest-dollar deviations
Detectors in the ensemble
3codeDetectors in the ensemble
Built onAirflowSnowflakescikit-learn and a neural autoencoderpandas, numpy and a postal-code geocoderCloud Run and object storage
01Problem02My role03Architecture04Decisions05What broke06Impact

The problem

What was actually wrong

The transportation management system processes thousands of shipping transactions a day at rates that vary legitimately by customer, route, mode, service level, rate offering and ship method. Some fraction of them are wrong. An overcharge produces a customer dispute; an undercharge produces silent revenue leakage.

Manual detection does not scale and is not consistent between reviewers. But the naive automation (flag anything far from the average) is useless here, because the variance is mostly legitimate. A premium expedited shipment on a long-haul lane is supposed to cost more. The problem is not finding outliers; it is finding outliers relative to the right comparison group, when the right comparison group is sometimes nine attributes deep and sometimes does not exist at all.

Before this existed

Manual review of transactions by pricing analysts, inconsistent between reviewers and unable to cover the volume.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

The reference distribution is contaminated
You are estimating the normal rate for a lane from data that contains the anomalies you are hunting. Any estimator that is not outlier-resistant is partly fitted to the thing it is meant to detect.
Specific comparison groups run out of data
The most precise business context (one customer, one full route, one mode, one service, one rate offering) often has too few historical transactions to say anything about.
Reviewer attention is the scarce resource
Reviewers work the top of a sorted list. Anything below the fold is effectively never seen, which shapes both the product and what the evaluation can honestly claim.
Over- and undercharge are not symmetric problems
Overcharge is a customer-facing dispute with a clear business context; undercharge is quieter and shows up as a pattern rather than a threshold breach. One detector should not be asked to do both.

My role

Owner. Designed and built the pipeline, the ensemble, the feature engineering, the feedback loop and the evaluation, including the audit that invalidated the first version of the evaluation.

Designed

  • The asymmetric three-detector ensemble and its weighting
  • The four-tier hierarchical statistical detector with per-tier confidence weighting
  • The serving contract: engineered features never leave the model boundary
  • The evaluation methodology, including what it deliberately refuses to report

Built

  • The orchestration pipeline: extraction, training, prediction, evaluation, write-back and notification
  • Around forty engineered features across business ratios, value tiers, mode-consistency flags and geospatial distance
  • The rebuilt label pipeline: verdict derivation from reviewer free text, theme matching, gold-set construction and seeded audit sampling
  • Regression gating and alert-volume projection over model versions

Led

  • Wrote the label-noise finding into the codebase rather than into a private note, and made the corrected precision the number the programme used

Architecture

How it is put together

A scheduled pipeline: extract a training window and a rolling prediction window from the warehouse, engineer features, score with three detectors, apply business-rule overrides, classify risk, and merge results back. A separate evaluation task re-scores from raw features against a reviewer-derived gold set and gates on regression.

The same architecture, in prose

A scheduler runs the pipeline daily.

Extraction pulls a 60-day training window that deliberately excludes the most recent eight days, and a 7-day rolling prediction window that excludes the current day.

Feature engineering derives around forty features: cost-to-sell ratios, rate per kilogram, accessorial-to-base ratios, cost complexity, value-tier percentages, mode-consistency flags, and geospatial features from postal-code distance including rate per kilometre and distance efficiency against the median.

Three detectors score in parallel: a hierarchical statistical detector at 35 percent, an autoencoder at 30 percent, and an isolation forest at 35 percent.

The statistical detector matches each transaction to the most specific business context that has enough history, falling back through four tiers and applying a confidence multiplier that rewards specificity.

Scores combine with context-aware weighting and tier-based threshold calibration, then business-rule overrides apply for known legitimate patterns such as absorbed cost and percent-declared-value pricing.

Risk classification produces the reviewer-facing ranking, sorted so the highest-dollar suspected errors surface first.

Predictions merge into the warehouse on a transaction key. Original columns and model scores are written; engineered features deliberately are not.

Reviewers work the top of the list in a feedback application, recording a verdict.

Evaluation re-scores from raw features rather than reading stored scores, compares against the reviewer-derived gold set, projects alert volume, and checks for regression against the previous model version.

Model artifacts persist to object storage, with a local fallback for development.

Every path, written out (3)· the walkthroughs above, as text

A daily scoring run

Yesterday's freight transactions need scoring and ranking for review.

  1. 01

    Pull two windows, both with a gap

    A 60-day training window excluding the last eight days, and a 7-day rolling prediction window excluding today.

    Recent transactions are still being corrected. Training on them teaches the model that in-flight adjustments are normal, which is exactly backwards.

  2. 02

    Derive ratios, not absolutes

    Around forty features: cost ratios, rate per kilogram, accessorial share, value tiers, mode-consistency flags, and postal-code distance features including rate per kilometre.

    Absolute cost is dominated by legitimate variation. Ratios and per-unit rates are comparable across shipments that otherwise share nothing.

  3. 03

    Find the most specific comparison group that exists

    Match on a nine-attribute key if there is enough history; otherwise fall back through five-to-six, then two-to-four attributes, then the global distribution, applying a confidence multiplier at each tier.

    A deviation within a precise business context means more than the same deviation against everything. The multiplier makes that belief explicit instead of hiding it in a threshold.

    Fails by: A new lane falls to the global tier and is scored with the lowest confidence. Visible in the output rather than silently treated as equivalent.

  4. 04

    Combine three detectors with different blind spots

    Statistical at 35 percent for overcharge, autoencoder at 30 percent for implausible attribute combinations, isolation forest at 35 percent for raw outliers.

    The statistical detector scores only above the mean. Undercharge is carried entirely by the two bidirectional detectors, which is a division of labour rather than an ensembling trick.

  5. 05

    Suppress the known-legitimate

    Business-rule overrides remove absorbed-cost and percent-declared-value patterns that look anomalous and are not.

    These rules change on a business timeline. Keeping them outside the model means the people who own them can read them.

  6. 06

    Rank by dollar exposure, not by score

    Risk classification sorts the queue so the highest-value suspected errors are at the top.

    Reviewer attention is the scarce resource. The ranking is the product; the score is an input to it.

  7. 07

    Write scores, not features

    Original columns and model scores merge into the warehouse. Engineered features stay inside the model boundary.

    The feature set changes every time the model improves. The serving contract must not.

Failure path: the evaluation was measuring the interface

Reported precision looked acceptable. It was several times better than the truth, and the gap was entirely an artefact of how labels were collected.

  1. 01

    The checkbox was pre-ticked from the model's own prediction

    The review form pre-populated its anomaly flag from the prediction being reviewed, and saved the whole row whenever any field changed.

    A reviewer who typed "the sell cost is correct" in the notes and left the pre-ticked box alone produced a stored row marking a false alarm as a confirmed anomaly.

    Fails by: The system was scoring its own agreement with itself and calling it ground truth.

  2. 02

    Nearly a third of labels contradicted their own checkbox

    Across the labels collected over five months, a substantial minority said in free text that the cost was correct while carrying an affirmative anomaly flag.

    The free text is what the reviewer actually did. The boolean is what the interface did.

  3. 03

    Rebuild the label pipeline around the text

    Verdicts are derived from reviewer free text, with theme matching over the derivation and a fixed-seed audit sample so the derivation itself can be checked by a human.

    Corrected precision came out several times lower than the boolean-derived figure. That number became the number the programme used.

  4. 04

    Report the corrected figure, and the themes behind it

    The evaluation reports precision and the dominant false-alarm themes, which is the part that tells you what to fix.

    Precision tells you how bad it is. Themes tell you what to do about it. A gold set that only yields a scalar is a scoreboard, not an instrument.

Failure path: the stored score is not any model's score

Evaluating by reading the score column from the prediction table would have produced a number that no model ever generated.

  1. 01

    Merging on a transaction key overwrites the score

    The write-back merges on shipment and order-release keys, overwriting non-key columns.

    A transaction inside the 7-day rolling window is therefore re-scored on roughly seven consecutive runs, and what is stored is whichever ran last.

  2. 02

    And the labels pool across model versions

    The labels collected over five months span six model versions.

    Even if the stored score were stable, the collection is not a sample from one model.

  3. 03

    Re-score from raw features instead

    Evaluation recomputes scores from raw features under the version being evaluated rather than reading the stored column.

    Reading those columns would measure a chimera. The extra compute is the price of measuring one thing at a time.

  4. 04

    Say plainly what cannot be measured

    Recall is reported for completeness and explicitly marked untrustworthy, because reviewers only worked the top of a sorted queue.

    Unflagged transactions are unlabeled, not negative. A recall number computed as though they were negative is not conservative. It is wrong in a flattering direction.

What changes as load grows· the scale stages, as text

The nightly run

code

1 window per day · full lane set · batch scoring

Airflow opens the window, extraction pulls the period, features are built, all three detectors score in parallel, the ensemble weights them, business rules override where they must, and predictions are written back before the pricing team logs in. The whole thing is embarrassingly parallel and finishes inside its slot.

Gives first: Extraction. Everything downstream is cheap compared with reading the window out of Snowflake.

Ten times the lanes

modelled

an order of magnitude more series per window

The hierarchical statistical detector is the one that feels it, because it fits per level of the hierarchy rather than once. Its cost grows with the number of groups, not the number of rows. The autoencoder scores in a single pass and barely notices. Feature engineering becomes the memory constraint rather than the CPU one. The gold set does not grow with the lanes, so evaluation coverage silently thins: the same number of labelled cases now speaks for ten times as much traffic.

Gives first: The statistical detector's per-group fitting, and (less visibly but more dangerously) gold-set coverage. The first shows up as a slow DAG; the second shows up as confidence in a model nobody has actually checked on the new lanes.

Intraday scoring

modelled

windows every few hours instead of nightly

Batch assumptions stop holding. A window that is a fraction of a day has too few observations for the hierarchical detector's higher levels, so it either widens its window and stops being intraday or reports on thinner evidence. Reviewer feedback arrives during scoring rather than between runs, so the feedback loop has to tolerate labels landing mid-window. Risk classification thresholds tuned on daily volumes flag differently on partial ones.

Gives first: Statistical power per window. This is the point where the honest move is to split the system: keep the nightly run as the authority and add a cheap, deliberately less confident intraday signal beside it, rather than pretending one model serves both cadences.

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

The statistical detector scores only above-mean rates and carries overcharge. The two machine-learning detectors are bidirectional and carry everything else, including undercharge.

Context

Over- and undercharge are different business problems with different evidence. A dispute has a clear counterparty and a threshold; leakage is a pattern.

Alternatives, and why not

  • One bidirectional statistical detectorSymmetric thresholds on a skewed cost distribution produce a flood of low-value undercharge flags, most of them legitimate discounting.
  • One model doing everythingThe business context that makes an overcharge legible (this customer, this lane, this service level) is exactly the context a general outlier model discards.

Rationale

Ensemble weights here are a division of labour, not a tuned hyperparameter. Each detector is responsible for the failures it is good at, and the weighting says so.

What it cost

Undercharge detection has no business-context expert behind it, so it is weaker and it is known to be weaker.

Outcome

Flag composition became explainable: you can say which detector raised a given transaction and why, which is what makes a reviewer trust the queue.

What it moved in the diagram

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

The gold set was recording the model's predictions as reviewer ground truth

What happened
The review interface pre-populated its anomaly checkbox from the model's own prediction and saved the whole row whenever any field changed. Reviewers who wrote that the cost was correct, but did not untick the box, produced rows stored as confirmed anomalies. Nearly a third of the collected labels contradicted their own free text.
Root cause
The field used as ground truth had the system's own output as its default value. Agreement was the path of least resistance for a reviewer whose actual work was writing the note.
What I did
Rebuilt the label pipeline: verdicts derived from free text, theme matching across derivations, gold-set construction, and a fixed-seed audit sample so the derivation itself is human-checkable.
What changed in the architecture
Downstream code uses the derived verdict and is not permitted to read the reviewer boolean at all.
What it taught me
Corrected precision was several times worse than what we had been reporting. The finding was written into the codebase next to the code it invalidated, because a measurement-integrity discovery that lives in someone's notes will be rediscovered the hard way.

The stored score belonged to no model in particular

What happened
Because the write-back merges on a transaction key, a transaction in the 7-day rolling window is re-scored on roughly seven consecutive runs, and the persisted score and version are whichever ran last. The collected labels also span six model versions.
Root cause
A serving table optimised for latest-state was being read as an experiment log.
What I did
Evaluation re-scores from raw features under the version under test rather than reading the stored columns.
What changed in the architecture
Evaluation was decoupled from the serving table entirely.
What it taught me
Before trusting a stored metric, ask what the write pattern does to it. Idempotent upserts and measurement are quietly incompatible.

Recall was not measurable and had been reported anyway

What happened
Reviewers only worked the top of a critical-sorted queue, so the unflagged population is unlabeled. Any recall computed by treating unflagged as negative is inflated.
Root cause
The labelling process is a function of the ranking, so the labels are not a sample of the population.
What I did
Recall is reported with an explicit statement that it is untrustworthy and why, and precision plus dominant false-alarm themes became the metrics the programme acted on.
What changed in the architecture
The honest fix, a randomly sampled review queue alongside the ranked one, was named and costed rather than quietly skipped.
What it taught me
Selection bias in the labelling process invalidates the metric, not just the sample. Say which number you cannot compute; the alternative is that someone else computes it wrongly.

Thresholds drifted upward as anomalies accumulated

What happened
Deviation estimates computed from the training window widened as genuine overcharges entered it, raising the flagging threshold over time.
Root cause
A standard-deviation estimator on a contaminated reference distribution is partly fitted to the anomalies it is meant to find.
What I did
Switched to a median-absolute-deviation robust z-score, with a mixture model for genuinely multimodal lanes.
What changed in the architecture
Per-tier threshold calibration, so the threshold reflects the confidence of the comparison group rather than one global cut.
What it taught me
In anomaly detection, the reference distribution is never clean. Choose estimators that assume contamination rather than estimators that assume it away.

Evaluation

How I knew whether it worked

The evaluation exists as a pipeline task with regression gating, and its most important property is what it refuses to claim.

It re-scores rather than reading

Scores are recomputed from raw features under the version being evaluated, because the stored score is an artefact of merge ordering across roughly seven re-runs and six model versions.

It states what it cannot measure

Precision and dominant false-alarm themes are measurable. Recall is not, because the label set is a function of the ranking, so it is reported for completeness and marked untrustworthy in the report itself.

Themes, not just a scalar

The report groups false alarms by theme, which is the part that tells you what to fix. A gold set that yields only a number is a scoreboard rather than an instrument.

Regression gating on a model

A new version is checked against the previous one before it ships, and projected alert volume is computed alongside precision. A model that improves precision by flagging half as much is a different product, not a better one.

The derivation is itself audited

Because verdicts are derived from free text, a fixed-seed sample is set aside so a human can check that the derivation matches what the reviewer meant.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

90%attested
Reduction in manual pricing review hoursBusiness-reported outcome. Not instrumented in the system; presented as the organisation's assessment rather than a measurement I can reproduce.
90%attested
Share of investigator effort directed at the top 10% highest-dollar deviationsBusiness-reported. The ranking is by dollar exposure by design; the effort concentration is the organisation's account of the result.
3code
Detectors in the ensembleWeighted 35 / 30 / 35.
4code
Fallback tiers in the statistical detectorNine-attribute key down to global, with confidence multipliers 1.2 / 1.1 / 1.0 / 0.9.
~40code
Engineered features
60 dayscode
Training windowExcluding the most recent eight days, which are still being corrected.
105observed
Reviewer labels in the corrected gold setCollected over five months, spanning six model versions.
Business
Suspected pricing errors are surfaced daily and ranked by dollar exposure, so review effort concentrates where the money is instead of spreading across the transaction volume.
Engineering
A full ML lifecycle in one pipeline (extraction, training, scoring, evaluation, write-back and notification), with regression gating on the model and a serving contract that survives feature-set changes.
People using it
Reviewers get a ranked queue with an explainable reason per flag, and a feedback path whose labels are now derived from what they actually wrote.

Leadership and hindsight

What I influenced, and what I would change

Technical leadership

  • Audited my own system's evaluation and found it several times more optimistic than the truth, then published the corrected figure rather than the flattering one.
  • Wrote the label-noise finding into the codebase, next to the code it invalidated, so it could not be rediscovered the hard way.
  • Named the cost of measuring recall honestly (a randomly sampled review queue) instead of quietly reporting a number that would not survive scrutiny.
  • Kept business pricing rules in a form the pricing team can read and change, rather than absorbing them into a model only I could modify.

What I would do differently

  • The scoring history should have been append-only from the start. A serving table that upserts on a transaction key is the right shape for serving and the wrong shape for measurement, and one table was doing both.
  • A pre-populated ground-truth control should have been caught in review of the feedback app. The general rule I now apply: a system's own output must never be the default value of the field that judges it.
  • Undercharge detection is the weakest part of the system. It has no business-context expert behind it, only the two bidirectional detectors, and it deserves a hierarchical detector of its own rather than borrowing one built for the opposite direction.
  • The ensemble weights were set by reasoning about division of labour rather than by fitting them. That was defensible early and should now be revisited against the corrected gold set, which did not exist when the weights were chosen.

Go deeper

The detail, for people who want it

Collapsed by default. The case study stands without any of it.

Why the training window has an eight-day hole in it

Training uses 60 days, excluding the most recent eight. Prediction uses a 7-day rolling window, excluding today.

The exclusions exist because freight transactions are still being adjusted for days after they post: accessorials arrive late, corrections land, rates are amended.

Training on that unsettled tail teaches the model that in-flight corrections are the normal state of a transaction, which raises tolerance for exactly the deviations it should be flagging.

The gap is a data-maturity boundary, and it is worth writing down as one. An arbitrary-looking constant with no comment is the kind of thing a future maintainer removes.

What the tier confidence multipliers actually encode

A match at the most specific tier carries a 1.2 multiplier; the next tiers carry 1.1, 1.0, and 0.9 at the global level.

Nothing here was fitted. The multipliers encode a stated belief: a 3-sigma deviation within one customer's specific lane, mode and service level is stronger evidence of an error than the same deviation against the entire book of business, because the comparison group actually resembles the transaction.

Putting the belief in a multiplier rather than in a threshold means it stays visible. Anyone can read what the system thinks specificity is worth, and argue with the number.

The alternative, one threshold per tier, tuned independently, hides the same belief inside four unrelated constants.

The general shape of the label-noise failure

The specific bug was a pre-ticked checkbox. The general shape is worth more than the specific bug.

Any time a system's output becomes the default value of the field used to judge that output, the evaluation drifts toward measuring agreement with itself. The drift is invisible because the labels look like labels.

Two properties make it detectable: collect a second, independent signal (here, free text the reviewer had to write anyway), and periodically check whether the two signals agree. They disagreed on nearly a third of rows.

The same theme runs through the measurement-platform work: a suite of assertions that assert nothing returns green, and a gold set that records predictions returns precision. Both are confidence you have not earned, and both are only found by auditing the instrument rather than the system.

Geospatial features from postal codes

Distance is derived from origin and destination postal codes rather than taken from the transaction, because the transaction does not carry it.

That yields rate per kilometre, a distance-efficiency ratio against the median for comparable shipments, a long-haul flag and a distance category.

Rate per kilometre is the single most useful of these: it makes a short expensive move and a long cheap move comparable, which raw cost does not.

It also introduces a dependency on postal-code reference data, which is a real maintenance obligation and the reason the feature set is not persisted downstream: the derived values are only as current as the reference data that produced them.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

Airflow
Daily orchestration of extraction, training, prediction, evaluation, write-back and notification.Separate tasks make failures attributable and re-runs partial, which matters when a stage is a model training run.
Snowflake
Source transactions and the prediction table.The merge semantics on the prediction table are the reason evaluation re-scores rather than reads: the same property that makes serving correct makes measurement wrong.
scikit-learn and a neural autoencoder
Isolation forest and reconstruction-based detection.Two detectors with different assumptions, deliberately chosen so their blind spots do not overlap with the statistical detector's.
pandas, numpy and a postal-code geocoder
Feature engineering, including distance-derived features.Ratios and per-unit rates make shipments comparable across lanes; distance has to be derived because the source data does not carry it.
Cloud Run and object storage
Execution and model-artifact persistence, with a local-storage fallback for development.The same pipeline runs on a laptop without cloud credentials, which is what keeps the training path debuggable.

5 more sections are written and hidden: constraints, evaluation, leadership, deeper, stack.

Next