A schedule matrix that makes the problem affordable

Sanctions Screening Automation

Screening customers against sanctions lists is a cross-join, and the entire engineering problem is which side of it you are allowed to take a delta on. Screen new customers against the full list daily. Screen the full customer base against newly designated parties when designations change. Do the complete cross-join quarterly. Take deltas on both sides and you miss the case that matters most: a newly designated party matching a customer you onboarded two years ago.

2025–2026mldatasecurity

What I owned

Owner. Pipeline, schedule design, matching strategy and the model-adjudication layer.

Screening schedules: 3codeScreening schedules
Pipeline stages: 3codePipeline stages

2decisions, argued

Built onAirflow and FastAPI on Cloud RunSAP HANA and object storageMy own multi-provider batch runtime

01Problem 02My role 03Decisions 04Impact

The problem

What was actually wrong

Every customer must be screened against sanctions and denied-party lists. Both sides change: customers are onboarded continuously, and designations are added by regulators without notice.

A full comparison of every customer against every listed party, every day, is not affordable. But the naive optimisation (compare only what changed on both sides) has a hole in it precisely where the compliance risk is highest.

Before this existed

Manual and partial screening, with the completeness of a given run depending on who ran it.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

Both sides of the comparison change independently: New customers arrive daily; new designations arrive on the regulator's schedule. The two events need different responses.
Name matching is inherently fuzzy: Transliteration, aliases, corporate suffixes and abbreviations mean exact matching finds almost nothing and loose matching finds almost everything.

My role

Owner. Pipeline, schedule design, matching strategy and the model-adjudication layer.

Designed

The three-schedule matrix
The cheap-filter-then-model adjudication sequence

Built

The extraction and parsing stages, the fuzzy matcher, and the model validation stage over my own batch runtime

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

Quarterly: the full customer base against the full list. Daily: new customers against the full list. On designation change: the full customer base against the new designations.

Context

A daily full cross-join is unaffordable; a delta-against-delta comparison is unsafe.

Alternatives, and why not

Daily full comparisonThe cost scales with the product of two growing sets, for a result that is unchanged for the overwhelming majority of pairs.
Compare only what changed on both sidesIt misses a newly designated party matching an existing customer, which is the single highest-consequence case in the whole system.

Rationale

Take a delta on exactly one side at a time, and pick which side based on which thing changed. The quarterly full run exists as a backstop against anything the event-driven runs missed.

What it cost

Three code paths and three schedules to operate, rather than one.

Outcome

Complete coverage of both change events at a fraction of the cost of daily full screening.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

3code

Screening schedulesQuarterly full, daily new-customer, and designation-triggered.

3code

Pipeline stagesParsing and extraction, fuzzy candidate generation, model adjudication.

Business: Continuous screening coverage of both new customers and new designations, without the cost of a daily full comparison.
Engineering: Reused my own batch runtime for the model stage, so rate limiting, throttling and credential handling came for free.

Leadership and hindsight

What I influenced, and what I would change

What I would do differently

The quarterly full run is a backstop that has never been used to catch anything the event-driven runs missed. That is either evidence the matrix is correct or evidence the backstop is not being checked properly, and I did not build the instrumentation to tell those apart.

Go deeper

The detail, for people who want it

Collapsed by default. The case study stands without any of it.

Screening and investigation are different products

This system and the compliance investigation platform look adjacent and are architecturally opposite.

Screening is batch, deterministic-first, high-volume and completeness-driven. Its failure mode is a missed match, and its design problem is cost at scale.

Investigation is per-entity, agentic, low-volume and depth-driven. Its failure mode is a confident wrong conclusion, and its design problem is defensibility.

Putting an agent on the screening problem would be expensive and worse. Putting a batch matcher on the investigation problem would answer a different question. The interesting thing is that the same compliance function needs both.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

Airflow and FastAPI on Cloud Run: Scheduling and execution of the three pipelines.Three schedules over one implementation, so the matrix is configuration rather than duplicated code.
SAP HANA and object storage: Customer data and list artefacts.Lists arrive as files and need parsing; customers live in the warehouse.
My own multi-provider batch runtime: The model adjudication stage.Rate limiting, throttle re-queuing and credential resolution were already solved, and vendoring it kept this deployment self-contained.

4 more sections are written and hidden: constraints, leadership, deeper, stack.