Financial crime · 10M+ transactions · rules the regulator can read

AML Transaction Monitoring & Case Platform

Anti-money-laundering surveillance has a constraint most detection problems do not: the rules must be readable by someone who does not trust you. A regulator can ask why a transaction did not alert, and 'the model scored it low' is not an answer that survives an examination. So the machine learning went where it could not suppress anything: a sequence model over per-customer transaction history that reorders the investigator's queue while the deterministic scenario engine keeps sole authority over what becomes an alert at all. That single boundary is the reason the model was allowed into production, and it is also what made the labels honest enough to train on.

Reconstructed

I do not hold the source for this system. The architecture, the decisions and the failure analysis are my own account of how I designed and ran it; the diagram is a reconstruction rather than a rendering of a repository.

My résumé is the source of truth for outcomes. Every outcome figure here is labelled attested; every scale figure used to make the design legible is labelled illustrative.

2021–2023mldataevaluationsecurity

What I owned

Primary architect. I owned the end-to-end design: ingestion and canonical schema, entity resolution, the feature layer, the hybrid scoring topology, the model governance approach, the investigator feedback loop and the deployment inside the client boundary, and set the technical direction other engineers implemented against.

open the diagram →

Transactions monitored: 10M+attestedTransactions monitored
Detection accuracy: +25%attestedDetection accuracy
Sequence-model precision: 74%attestedSequence-model precision

11components
2request paths
3load stages
4decisions, argued
2failures, owned

Built onPythonTensorFlow / KerasSQL stored proceduresAWS SageMakerAWS S3

01Problem 02My role 03Architecture 04Decisions 05What broke 06Impact

The problem

What was actually wrong

Transaction monitoring at this scale produces far more alerts than any investigation team can work. The industry's usual response is to tune thresholds upward until the volume is survivable, which reduces alerts without reducing risk. It simply moves the misses somewhere nobody counts them.

The second problem is that the obvious fix is illegal in spirit. Replacing scenario rules with a classifier gives better precision on paper and destroys the property the whole apparatus exists for: an examiner must be able to ask why a given transaction did or did not alert, and receive an answer in terms of a written, approved, version-controlled rule. A model that cannot be interrogated that way cannot hold the decision.

So the real design question was never whether a model could beat the rules. It was a placement question: where can a model sit such that it improves the outcome without ever being the thing that decides, and what does that placement cost?

Before this existed

A deterministic scenario engine producing a flat, unordered alert queue, triaged in arrival order by investigators reading raw transaction listings.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

Every alert decision must be explainable to an examiner: Not 'interpretable' in the machine-learning sense. Explainable in the sense that the rule that fired is a written document with an approval date and an owner.
Nothing may be suppressed: A model that lowers alert volume by declining to alert has assumed regulatory authority it does not have. Reordering a queue is permitted; emptying it is not.
The labels are investigator dispositions, not truth: A closed alert means an investigator was not convinced. Confirmed laundering is vanishingly rare and arrives years late, if at all. Training on dispositions trains on attention, not on crime.
Customer data cannot leave the client boundary: Processing runs inside the client's environment. This rules out several convenient managed services and shapes the whole deployment.

My role

Primary architect. I owned the end-to-end design: ingestion and canonical schema, entity resolution, the feature layer, the hybrid scoring topology, the model governance approach, the investigator feedback loop and the deployment inside the client boundary, and set the technical direction other engineers implemented against.

Designed

The hybrid topology and its load-bearing rule: the scenario engine holds exclusive authority to create an alert; the model may only order what already exists
The canonical transaction schema that let one detection layer serve multiple client source systems without per-client detection code
The per-customer sequence representation: transaction history as an ordered behavioural signal rather than a bag of independent rows
The feedback loop, and the sampling design that keeps it from becoming a loop that only learns what investigators already look at
The model governance package: challenger models, above-the-line and below-the-line threshold testing, and documentation written for a model-risk reviewer rather than for engineers
The deployment strategy inside the client boundary, with the training path and the scoring path physically separated

Built

The sequence-model scoring pipeline end to end, from feature extraction to the ranked queue the investigator sees
The alert enrichment layer that attaches the behavioural context to an alert so an investigator starts with a narrative rather than a transaction dump
The stored-procedure surveillance logic hardened and reworked against the canonical schema
The disposition capture path that turns an investigator's close-out into a usable training signal

Led

Made the case to compliance stakeholders for why the model would not be permitted to suppress, and why that made it approvable
Set the convention that any model touching a regulated decision ships with its below-the-line test before it ships at all
Worked with investigators to define what a useful alert narrative contains, rather than inferring it from the data that happened to be available

Architecture

How it is put together

Two detection paths that never merge authority. The scenario engine decides what is an alert; the sequence model decides what order the alerts are worked in. Everything between them (the canonical schema, entity resolution, the feature layer) is shared so the two paths cannot disagree about what a transaction is.

The same architecture, in prose

Transaction feeds arrive from client source systems on a nightly batch cycle, with higher-risk payment rails delivered intraday.

Ingestion normalises every feed into one canonical transaction schema, so detection logic is written once rather than once per client.

Entity resolution links transactions to a resolved customer party, because behaviour is only meaningful per-entity and source systems disagree about identity.

The feature layer computes velocity windows, peer-group baselines and counterparty aggregates from the canonical rows.

The scenario engine evaluates the approved rule set (structuring, rapid movement of funds, high-risk corridors), and is the only component permitted to create an alert.

In parallel, the sequence model reads the customer's ordered transaction history and produces a behavioural risk score. It creates nothing.

The ranker joins each alert to its customer's behavioural score and orders the queue. An alert with a low score is worked later, never never.

Enrichment attaches the behavioural context to the alert so the investigator opens a narrative rather than a spreadsheet.

The case platform holds the investigation, the narrative and the disposition, and is the path to a regulatory filing.

Dispositions flow back as training signal through a sampler that deliberately includes a random slice of low-ranked alerts, so the model can be told it was wrong.

Model governance sits beside the scoring path: challenger models, threshold tests above and below the line, and the documentation a model-risk reviewer reads.

Every path, written out (2)· the walkthroughs above, as text

An alert, start to finish

A customer's transactions arrive overnight, trip a structuring scenario, and reach an investigator ranked by behavioural risk.

01
Transactions land
The client's nightly file and the intraday wire feed deliver the period's transactions.
Batch is sufficient for most scenarios; the rails where speed matters are delivered separately rather than by making everything real-time.
Fails by: A feed arrives short and the missing rows look like an absence of activity rather than an absence of data.
02
Normalise to canonical
Each source format is mapped onto the shared transaction schema and control totals are reconciled to source.
One detection implementation across clients. The reconciliation is what makes the normalisation trustworthy rather than merely convenient.
Fails by: A currency or timezone convention differs for one client and normalisation makes the difference invisible.
03
Resolve the party
Accounts across systems are linked to one resolved customer.
Structuring is defined by behaviour split across accounts. Without resolution the pattern is undetectable in principle.
Fails by: Under-merging leaves the customer split and the structuring pattern never assembles.
04
Compute the behavioural view
Velocity windows, peer-group baselines and counterparty aggregates are computed once for both detection paths.
Shared inputs mean the rules and the model cannot disagree about what the customer did.
Fails by: A thin peer group produces an unstable baseline and the customer looks anomalous for being unusual rather than suspicious.
05
The rule fires
A structuring scenario evaluates true and creates an alert citing the rule that fired.
This is the only place an alert can come into existence, which is what makes the whole system explainable to an examiner.
06
Score the behaviour
In parallel, the sequence model reads the customer's ordered history and emits a behavioural risk score.
The model runs whether or not a rule fired. It is describing the customer, not judging the alert.
Fails by: A customer with too little history gets a score the model has no basis for.
07
Order the queue
The alert is joined to the score and placed in the investigation queue accordingly.
This tiny join is the entire integration between the two paths. Keeping it this small is what keeps the boundary auditable.
Fails by: A missing score sends a genuinely urgent alert to the bottom of the queue.
08
Attach the context
The behavioural evidence that makes the alert legible is attached to it.
The investigator should start from a narrative, not reconstruct one.
09
Investigate and dispose
The investigator works the alert, writes the narrative, and records a disposition.
Capturing the judgement where it is made is the only way it becomes usable signal later.
10
Feed it back, including what was skipped
Dispositions become labelled examples, with a deliberate random slice drawn from the bottom of the queue.
Without that slice the model only ever learns from alerts it already ranked highly, and its false-negative rate becomes unmeasurable.
Fails by: The random slice gets dropped under volume pressure and the loop silently closes on itself.

Failure: the loop closes

The random sampling slice is suspended during a backlog. Within two retraining cycles the model is measuring its own agreement with itself.

01
A backlog builds
Alert volume rises and the team stops working the randomly sampled low-rank alerts to catch up.
Entirely reasonable operationally. Nobody made a modelling decision here.
02
The only corrective channel closes
Every training label now comes from alerts the model already ranked highly.
This is the failure. The sampler is not a nice-to-have; it is the sole mechanism by which the model can learn it was wrong.
Fails by: Nothing errors. The pipeline is healthier-looking than before, because low-yield alerts stopped consuming investigator time.
03
The model retrains on its own opinion
Precision on the training distribution improves. The model is now confidently reproducing its previous ranking.
A model trained only on what it selected will always look like it is improving.
04
Below-the-line testing catches it
The scheduled below-the-line test samples under the threshold and finds cases that should have ranked high.
This is exactly what below-the-line testing exists for, and the reason it was a release requirement rather than a periodic review item.
05
The slice becomes non-negotiable
The random slice is given a floor that operational pressure cannot suspend, and the proportion of training labels originating from it is reported every cycle.
A safeguard that can be turned off under load will be turned off precisely when it matters.

What changes as load grows· the scale stages, as text

Pilot

illustrative

One client, one payment rail

Everything fits in a single nightly batch. Entity resolution is cheap because the party population is small and mostly unambiguous.

Gives first: Not throughput, but label volume. There are not enough dispositions yet to train a sequence model, so the ranker runs on scenario severity alone.

Production

attested

10M+ transactions monitored

The feature layer becomes the expensive stage: velocity windows and peer-group baselines are recomputed across the full party population every cycle.

Gives first: Peer-group baseline computation, and entity resolution against a party population large enough that ambiguous matches are routine rather than exceptional.

Regulatory lookback

illustrative

Multi-period reprocessing on demand

An examiner asks the system to re-evaluate historical periods under a revised scenario. The batch path has to run over months of history without disturbing the nightly cycle.

Gives first: The scenario engine and the shared feature layer, which is why the training path and the scoring path were separated in deployment: a lookback must not starve tonight's run.

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

Machine learning was given no authority to create or suppress an alert. It orders the queue and nothing else.

Context

The obvious version of this project replaces scenario rules with a classifier and reports a large precision improvement. That version is unshippable, and it took a while to articulate why in terms stakeholders accepted.

Alternatives, and why not

Replace the scenario engine with a classifierAn examiner can ask why a specific transaction did not alert. 'The model scored it 0.3' is not an answer. The whole surveillance obligation rests on the answer being a written, approved rule.
Let the model suppress low-scoring alertsThis is threshold tuning wearing a disguise. It reduces volume by reducing coverage, and it moves the misses somewhere nobody counts them.
Use the model only offline, for periodic analysisSafe and nearly useless. The investigator's queue order is where the value is, and it is a decision being made badly hundreds of times a day.

Rationale

Placing the model where it cannot decide is what made it approvable. It also had an effect I did not anticipate: because the model was not gating anything, the compliance conversation stopped being adversarial and became a design conversation.

What it cost

Precision gains are capped by the rules. The system never gets the headline number a pure classifier would report, and I gave that up deliberately.

Outcome

Detection accuracy improved 25% and false positives fell 30% without a single approved rule being changed, which is precisely why it was allowed to ship.

What I would do today

The same, and I would write the authority boundary into the code as an explicit interface rather than a convention. A rule that lives only in a design document is one refactor away from not existing.

What it moved in the diagram

Scenario engine
Sequence model
Alert ranker

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

The feedback loop closed and the metrics improved

What happened: During an alert backlog, the randomly sampled low-rank alerts stopped being worked. Two retraining cycles later the model's reported precision had improved and it was, in fact, measuring its agreement with its own previous ranking.
Root cause: The random slice was a process commitment rather than an enforced one. Under operational pressure the rational local decision (stop working alerts the system says are low risk) removed the only channel through which the model could learn it had been wrong.
What I did: The scheduled below-the-line test surfaced it: sampling under the threshold found cases that should have ranked high. I traced the training label composition and found the random-slice proportion had gone to near zero without any alert firing.
What changed in the architecture: The slice was given a floor that operational pressure cannot suspend, and the proportion of training labels originating from it became a reported figure every cycle rather than an assumption.
What it taught me: A safeguard that can be switched off under load will be switched off exactly when it is load-bearing. And a metric that improves when a safeguard is removed is not a metric, it is a mirror.

Low-activity customers looked suspicious for being quiet

What happened: Peer-group baselines were unstable for customers whose peer group had too few active members, so ordinary behaviour produced large deviations and those customers were persistently over-alerted.
Root cause: The peer-group definition was built for the dense middle of the population and applied uniformly. Nothing in the design asked what happens when the comparison group runs out of members.
What I did: Traced the over-alerting to peer-group population counts, which had not previously been instrumented at all.
What changed in the architecture: Hierarchical fallback: when a peer group is too thin to support a baseline, the comparison widens to the next level and the alert records which level it was computed at, so the weaker comparison is visible rather than implicit.
What it taught me: Every statistic computed against a comparison group needs an answer for what happens when the group is empty. The default answer, compute it anyway, is always wrong and never errors.

Evaluation

How I knew whether it worked

Two evaluation regimes, because the two paths are answering different questions. The scenario engine is evaluated on coverage; the model is evaluated on ranking quality. Neither number is meaningful in the other's terms.

Below-the-line testing as a release gate

Any threshold change ships with a sample drawn from below the new line and worked to disposition. Without it, raising a threshold and reducing coverage are indistinguishable from the outside.

Precision measured only against confirmed outcomes

Investigator dispositions are used for training but not for headline precision, because a closed alert means an investigator was unconvinced rather than that nothing happened.

Challenger models run in parallel without authority

A challenger scores the same alerts and its ranking is compared, but it never reaches the queue. Comparison on live traffic without exposure to it.

Queue-tail age as the real safety metric

Ranking is only legitimate if the bottom of the queue is eventually worked. The age of the oldest unworked alert is the number that says whether ranking has quietly become suppression.

Security and safety

What the system refuses to do

Processing stayed inside the client boundary, and the parts of the system that could become an unauthorised decision were structurally prevented from doing so.

The model cannot create or suppress

Not a policy, but a topology. The scoring path has no write access to the alert store. The worst a broken model can do is order the queue badly, which is visible and recoverable.

Customer data does not leave the client environment

Training and scoring both run inside the boundary. This ruled out several managed services that would have been considerably easier to operate.

Training and scoring paths separated in deployment

A regulatory lookback or a retraining run cannot starve the nightly scoring cycle. The paths share code and not capacity.

Every alert carries the rule that produced it

Provenance is attached at creation rather than reconstructed at examination time.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

Where investigator time went, before and after

illustrative

Illustrative proportions showing the shape of the change, not measured values. The attested figures are the 40% processing-time reduction and the 30% false-positive reduction; this is what that felt like in the queue.

Triage, before70%

Triage, after40%

Escalation, before30%

Escalation, after60%

10M+attested

Transactions monitored

+25%attested

Detection accuracyHigher-risk alert yield at lower investigator noise

74%attested

Sequence-model precisionSuspicious-activity precision

−30%attested

False positives

−40%attested

Alert processing timeVia the case-management workflow

Noneattested

Approved rules changedThe improvement came from ranking, not from retuning coverage

Business: Investigation capacity moved from triage to escalation without reducing what the system watches. The distinction matters: most alert-volume reductions in this field are coverage reductions.
Engineering: One canonical schema and one approved rule set across clients, with new feeds onboarding as an ingestion mapping rather than a detection project.
People using it: Investigators open an alert with the behavioural narrative already assembled, and work the queue in an order that reflects behaviour rather than arrival time.

Leadership and hindsight

What I influenced, and what I would change

Technical leadership

Argued the authority boundary to compliance stakeholders before writing the model, which converted an adversarial approval process into a design conversation.
Made below-the-line testing a release requirement rather than a periodic review item, which is the only reason the feedback collapse was caught within two cycles.
Defined the alert narrative with investigators rather than inferring it from whichever fields happened to be available.

What I would do differently

The random sampling slice should have been enforced in code from the first release. I wrote it as a process commitment and it behaved exactly as process commitments behave under pressure.
Peer-group thinness was foreseeable and I did not foresee it. I instrumented population counts only after the over-alerting had been running for a while.
I would revisit the counterparty graph today. I rejected it for data-quality reasons that were correct at the time and are the first thing I would re-test.

Go deeper

The detail, for people who want it

Collapsed by default. The case study stands without any of it.

Why below-the-line testing is the only honest threshold metric

Raising an alert threshold always looks good from above the line: volume falls, precision rises, investigator time per alert improves. Every metric that is easy to collect moves in the direction you want.

The only way to distinguish 'we removed noise' from 'we stopped looking' is to sample from below the new threshold and work those alerts to disposition. If nothing interesting is down there, the change was a genuine improvement. If something is, you have quietly reduced coverage and reported it as an efficiency gain.

This is expensive and unrewarding, and it is the difference between a surveillance system and a system that produces surveillance-shaped output.

The disposition taxonomy problem

'Closed, not suspicious' and 'closed, not investigable' are entirely different statements, and most case systems record them identically.

The first is evidence about the transaction. The second is evidence about the data available to the investigator. Training on them as one label teaches the model that missing data means innocence.

Splitting the taxonomy was a small change to the case platform and one of the higher-value things I did to the training signal.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

Python: Feature computation, the sequence model, and the scoring pipelineThe modelling ecosystem, and the client environment already ran it.
TensorFlow / Keras: The recurrent sequence model over per-customer transaction historySequence modelling with a deployment story the client's platform team could operate.
SQL stored procedures: The scenario engine and the canonical transformation layerDetection logic had to run where the data was, and be readable by reviewers who were not engineers.
AWS SageMaker: Training and model hosting inside the client boundaryManaged training that stayed within the account perimeter the client's data policy required.
AWS S3: Canonical transaction storage and training snapshotsVersioned, cheap, and inside the boundary.

6 more sections are written and hidden: constraints, evaluation, safety, leadership, deeper, stack.