Multi-agent research · sanctions, adverse media, ownership

AI Compliance Investigation Platform

An orchestrator constructed per request delegates to five specialist sub-agents, all sharing one wall-clock deadline, one source registry and one middleware stack. The agent graph is the least interesting part. The engineering is in the harness around it. Four independent budgets at the tool-call boundary, a soft and hard deadline that survives a human-approval pause, durable checkpoints on blob storage, a cross-instance control channel using compare-and-swap, and fail-static authorization that serves last-known-good grants within a bounded age. Every one of those exists because something specific broke in production.

2025–2026agenticsecurityevaluationplatform

What I owned

Primary architect and technical lead. I owned the end-to-end system design: topology, orchestration, agent harness and middleware, integrations, authorization, the evaluation approach and the deployment strategy, and the failure fixes that changed it.

open the diagram →

Sanctions regimes screened: 6codeSanctions regimes screened
Specialist sub-agents: 5codeSpecialist sub-agents
Independent budgets in the guard middleware: 4codeIndependent budgets in the guard middleware

17components
3request paths
3load stages
6decisions, argued
7failures, owned

Built onLangGraph + a deep-agent frameworkGemini with search grounding, and Claude, both on VertexBlob storage as the agent filesystemCloud Run

01Problem 02My role 03Architecture 04Decisions 05What broke 06Impact

The problem

What was actually wrong

A trade-compliance analyst screening a counterparty has to check six sanctions regimes, search for adverse media, resolve aliases and transliterations, map corporate ownership up to the beneficial owner, and then produce something a lawyer will sign off on. Done properly it is hours of work per party, and it is repeated constantly.

The obvious application of an agent here is also the dangerous one. In a sanctions tool, a confident wrong answer is not a bad user experience. It is a false designation attached to a real company, or a missed one attached to a sanctioned party. The determination has to be reproducible across runs, traceable to sources, and honest about what it did not manage to check.

Before this existed

Manual research across public registries, sanctions lists and news, with the reasoning living in an analyst's head and a spreadsheet.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

The answer has to be defensible: An analyst cannot defend, and counsel cannot review, a rating whose inputs change between runs. Reproducibility is a product requirement, not an engineering nicety.
Investigations run for minutes, on infrastructure that evicts: A full research run takes minutes against a 1800-second request timeout, on a container platform that scales between two and twenty instances and can evict at any point.
Cancel arrives at the wrong instance: With multiple instances behind a load balancer, a cancel request almost never lands on the instance holding the open stream.
Absence of evidence is not a clean result: A screening that found nothing because coverage was incomplete looks identical, in a summary, to a screening that found nothing because there is nothing to find.

My role

Primary architect and technical lead. I owned the end-to-end system design: topology, orchestration, agent harness and middleware, integrations, authorization, the evaluation approach and the deployment strategy, and the failure fixes that changed it.

Designed

The orchestrator-plus-five-specialists topology and the per-request construction invariant
The middleware harness: budgets, deadlines, citation registry, progress, run control
Fail-static authorization with a bounded-age mirror
The risk rubric as code over a closed factor list, and the source-tiering model

Built

A durable checkpointer over blob storage, extending the framework's in-memory saver rather than reimplementing checkpoint semantics
An agent filesystem backend on blob storage with per-user roots and compare-and-swap primitives
Fact polarity, the fix for the worst bug the system ever had
An in-process evaluation harness whose assertions read the guard counters directly

Led

Turned the repository's own incident history into the evaluation suite
Set the pattern of moving domain knowledge out of prompts into reviewable skill files

Architecture

How it is put together

Two modes share one harness. Quick search is a single agent. Full research constructs an orchestrator per request, delegates to five specialists in parallel where they are independent, and holds them all to one deadline, one source registry and one set of budgets.

The same architecture, in prose

A request enters through the web application and the API, authenticates via enterprise single sign-on, and exchanges that for a signed token.

Authorization is checked against a permissions table in the data warehouse, with a fail-static mirror on blob storage covering warehouse outages within a bounded age.

The orchestrator is constructed fresh for this request. That is load-bearing: it lets the deadline, the source registry and the checkpointer be per-turn without any locking.

Before planning, the system recalls what this analyst already knows about this party, dated, staleness-flagged, and never a substitute for a live check.

The orchestrator plans, and if the plan is broad it pauses at a human-approval gate. The deadline pauses with it.

It then delegates to five specialists: sanctions screening across six regimes, adverse media, corporate structure, identity resolution and digital footprint. They run in parallel where independent, each carrying the same middleware stack and sharing the run-level deadline and source registry.

Evidence is written to a per-user investigation directory on blob storage: evidence, state, audit and summaries.

Sources are tiered as official or corroborating, with publisher names collapsed to canonical form.

The risk band is computed in code by a rubric from the factor identifiers the model returns. The model never emits the band itself.

A budget-capped verification pass runs, dangling citations are stripped, and the answer streams to the client while counters and traces are recorded.

Every path, written out (3)· the walkthroughs above, as text

A full investigation

An analyst submits a counterparty name for full research.

01
Authenticate
Enterprise single sign-on establishes identity and issues a signed token.
Identity has to exist before anything else, because the investigation's storage root is keyed on it.
02
Authorize, or degrade visibly
The role is resolved from the warehouse permissions table.
Unknown users are denied by default rather than defaulted to a read-only role.
Fails by: If the warehouse is unavailable, the fail-static mirror serves last-known-good grants within a bounded age. Past that age it refuses rather than granting forever.
03
Construct the orchestrator for this request
A fresh orchestrator is built, with its own deadline, source registry and checkpointer bound to this thread.
Per-request construction is what removes locking from the entire concurrency story.
04
Recall what this analyst already knows
Prior findings about this party for this analyst are surfaced, dated and staleness-flagged.
Matching is exact-substring rather than fuzzy. Surfacing the wrong company's prior screening is worse than surfacing nothing.
Fails by: Recall is context, never a substitute for a re-check. Sanctions lists move, so every query still pays for a live screening.
05
Plan, and pause if the plan is broad
The orchestrator produces a plan. A broad plan stops at a human-approval gate.
The deadline pauses here, so time the analyst spends reading the plan does not eat the research budget.
06
Delegate to five specialists
Sanctions, adverse media, corporate structure, identity resolution and digital footprint run in parallel where independent.
Each carries the same guard middleware and shares one deadline and one source registry, so the run has a single budget rather than five.
Fails by: A specialist finding nothing must report a coverage gap. Unknown is not low.
07
Register every source once
Each unique source gets one identifier for the investigation's lifetime, stamped into the text the model reads.
This is the fix for citations resolving to the wrong page, and it has to happen at the middleware layer because specialists cannot see each other's numbering.
08
Compute the band in code
The model returns factor identifiers; the rubric computes the band.
Same factors, same band, every time. A determination an analyst can defend and counsel can review.
Fails by: A novel factor outside the closed list does not move the band. That is visible rather than silent.
09
Write evidence and audit trail
Evidence, state, audit log and summaries are written under the analyst's own storage root.
Auditability of the evidence, not just of the action. The tenant boundary is the storage prefix.

Failure path: memory manufactures a sanctions hit

A clean screening two turns ago becomes, in this turn, established prior knowledge that the party is designated. In a sanctions tool this is the worst thing the system can do.

01
The screening comes back clean
The sanctions specialist checks the party and finds no designation.
This is the correct result. Everything after this is the bug.
02
The negation is silently dropped
The finding is stored as a subject-predicate-object triple: party, listed-on, sanctions list. The 'not' lives in an extra field the schema does not declare, and validation drops undeclared fields.
A triple has no place to put a negation. The structure cannot represent the answer, so it stores the opposite of it.
Fails by: Nothing in the pipeline notices, because a dropped field is not an error.
03
Two turns later, it is a fact
Recall surfaces the stored triple as established prior knowledge. The model is told the party is on the designation list.
Recall is trusted precisely because it is supposed to be the system's own verified prior work.
04
Quarantine, do not guess
The fix: every fact carries explicit polarity (affirmed, negated, or unknown), and anything not explicitly affirmed or negated is never rendered into a prompt.
Fail closed on ambiguity. There is deliberately no migration of the old triples: they genuinely are ambiguous, no offline pass can recover their direction, and guessing is exactly what caused the bug.

Failure path: the agent runs away

A verify-then-revise cycle re-ran a full screening four or more times over ten minutes, producing a different risk verdict each time.

01
The identical-argument guard does not fire
The loop guard short-circuits repeated calls with identical arguments, but the delegation instruction text varied slightly on every call.
A loop guard that keys on exact arguments only catches the exact repetition. An agent rephrasing itself escapes it.
02
Self-verification eats the run
Separately, the verification tool was called sixteen times, consuming roughly a third to a half of total wall time re-checking its own draft instead of gathering evidence.
Verification feels productive to the model. Nothing in the loop distinguishes checking from progressing.
03
Budget kinds of call, not identical calls
Four independent budgets: identical arguments, total tool calls, delegations, and verifications, each per turn.
Each budget exists because the agent found a different way to run away. Counting kinds rather than repetitions is what generalises.
04
And a wall clock behind all of them
A specialist doing slow back-to-back research burns time, not call count, so a soft deadline refuses new research tools and a hard deadline ships the partial answer.
Better a partial cited answer at minute nine than a platform timeout returning nothing at minute thirty.

What changes as load grows· the scale stages, as text

One investigation

code

1 subject · 5 specialists · sources queried once each

An analyst signs in, authorization scopes what they may investigate, and the orchestrator fans out to the five specialists. Each queries the source registry, findings are tiered and polarity-tagged, the rubric scores them, and the investigation is checkpointed as it goes. The run deadline exists but is never reached.

Gives first: The slowest external source. The platform waits on other people's systems more than on its own.

A screening batch

modelled

hundreds of subjects submitted together

Source rate limits become the governing constraint, and they are per-source rather than global, so a batch does not slow down uniformly, it stalls behind whichever tier-one source throttles first. The run deadline starts firing, which is the design working: a partial investigation that says which sources it could not reach is usable, and an investigation that hangs forever is not. Checkpoints stop being a crash-recovery feature and become the thing that lets a stalled run resume where it stopped.

Gives first: Source tiering. A batch is only as fast as the highest-authority sources permit, and downgrading to faster secondary sources changes what the finding is worth, so it is a compliance decision, not a throughput knob.

Sustained regulatory load

modelled

continuous submissions · investigations reopened and revised

Investigations stop being one-shot. A subject screened last quarter has to be re-screened against sources that have since changed, which means storage has to hold not just the finding but the evidence state it was true against. Guard middleware and the deadline are constantly active rather than exceptional, and run control (pause, resume, cancel) becomes an operator surface rather than a debugging aid.

Gives first: Evidence staleness. The expensive question at this scale is not 'can we screen this subject' but 'which of our existing conclusions are no longer supported', and answering it re-reads history rather than adding capacity.

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

The model returns factor identifiers from a closed list. A rubric in code computes the band.

Context

The same party was being rated differently across turns, on different grounds each time.

Alternatives, and why not

Let the model output the band directly against described criteriaAn analyst cannot defend, and counsel cannot review, a rating whose inputs change between runs. Non-reproducible is the same as wrong for this use case.
Ask the model to explain its band and audit the explanationsA post-hoc rationale for a band the model already chose is not the reasoning that produced it.

Rationale

Reproducibility is the product. Fix the factor vocabulary, let the model do what it is good at, deciding which factors are present, and let code do what it is good at, which is applying the same rule every time.

What it cost

Rigid. A genuinely novel risk factor does not move the band until someone adds it to the list. That is visible and reviewable, which is better than a rating that moves for unknown reasons.

Outcome

Reproducibility became a checked property, with a script that verifies the same factors produce the same band.

What I would do today

I would add an explicit 'novel factor observed' output so the model can flag something outside the vocabulary without being able to score it. Today that signal is lost rather than escalated.

What it moved in the diagram

Risk rubric

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

The memory manufactured a sanctions designation

What happened: A clean screening of a large asset manager was stored, and two turns later the model was told as established prior knowledge that the party appeared on three designation lists. In a sanctions tool that is the worst thing this system can do.
Root cause: A subject-predicate-object triple cannot express 'no'. The finding was written as party / listed-on / sanctions list, with the negation carried in an extra field the schema did not declare. Validation dropped the undeclared field, leaving an affirmative statement of the exact opposite of the finding.
What I did: Introduced explicit fact polarity (affirmed, negated, unknown) as a required part of every stored fact, with fail-closed rendering: anything not explicitly affirmed or negated is never placed into a prompt.
What changed in the architecture: Deliberately no migration of existing facts. The old triples genuinely are ambiguous, no offline pass can recover their direction, and guessing is what caused the bug in the first place. They are quarantined permanently.
What it taught me: A schema that cannot represent a negative will store the positive. When you find data whose direction you cannot recover, quarantine it. The instinct to backfill is the same instinct that produced the incident.

A verification loop re-ran a full screening four times, with a different verdict each time

What happened: A stuck verify-then-revise cycle ran for over ten minutes. Separately, the verification tool was called sixteen times in one turn, consuming roughly a third to a half of total wall time re-checking its own draft rather than gathering evidence.
Root cause: The identical-argument loop guard never fired, because the delegation instruction text varied slightly on each call. The guard caught exact repetition; the agent was paraphrasing.
What I did: Added budgets that count kinds of call rather than identical calls: total tool calls, delegations, and verifications, each capped per turn, each returning an actionable error to the model rather than killing the graph.
What changed in the architecture: Four independent budgets at the tool-call boundary, applied to the orchestrator and every specialist.
What it taught me: Each budget in that stack exists because the agent found a different way to run away. A loop guard keyed on exact arguments only catches the least creative failure.

Wall-clock runaway invisible to every call budget

What happened: A specialist doing back-to-back web research consumed the whole request window without ever tripping a call-count budget, and the platform's 1800-second timeout returned the analyst nothing at all.
Root cause: Call budgets measure calls. Slow tools burn time, not calls.
What I did: A run-level deadline with two stages: a soft deadline that refuses new research tools and instructs the model to answer from what it has, and a hard deadline that breaks the graph loop and ships the partial answer.
What changed in the architecture: One deadline shared by the orchestrator and all five specialists, per turn, not per agent, that pauses and resumes around the human-approval gate.
What it taught me: A partial, cited answer delivered at minute nine is worth more than a timeout at minute thirty. Design the degraded output before you need it.

Citations linked to the wrong page about as often as the right one

What happened: Every tool numbered its own sources from one and restarted on the next call, including inside specialists the orchestrator never saw. The interface mapped citation N to position N-1 of a score-sorted list.
Root cause: Source numbering was a local concern in a system where the final answer is assembled from many local scopes.
What I did: A per-investigation source registry owned by middleware. One identifier per unique source for the investigation's lifetime, stamped into the text the model reads, with dangling citations stripped from the final answer.
What changed in the architecture: Citation identity moved from the tool layer to the run layer, which is the only layer that can see all of it.
What it taught me: In a cited-output system, the citation is part of the answer. A wrong link is a wrong answer, and it is the failure most likely to survive review because it looks right.

Wikipedia was the third most-cited domain

What happened: Over three days of production traces, Wikipedia had 15 citations against 16 for the national treasury site and 7 for the federal register. Two hostnames for the same government department were counted as different publishers.
Root cause: Nothing in the system had an opinion about source quality, and publisher identity was the hostname string.
What I did: Explicit closed tier tables (official versus corroborating), plus canonical publisher collapse, so the same authority under two hostnames counts once.
What changed in the architecture: Tiers are advisory rather than a filter, because hiding a source from the analyst is worse than showing them a weak one with a label.
What it taught me: The question 'how much of this answer rests on the official list' was unanswerable until publisher identity was canonical. Aggregation over uncanonicalised keys quietly inflates corroboration.

An eviction eight minutes into a twelve-minute investigation discarded all of it

What happened: With in-memory checkpointing, any crash, deploy or platform eviction lost the entire run.
Root cause: Long-running work on ephemeral instances with no durable state.
What I did: A durable checkpointer over blob storage, built by extending the in-memory saver rather than reimplementing checkpoint semantics.
What changed in the architecture: Debounced flushes, because the persist call fires around thirty times per deep run with a payload that grows throughout.
What it taught me: Resume granularity is a cost decision, not a correctness one. Ten seconds of replay is cheap; every super-step persisted is not.

A warehouse key rotation stopped every analyst

What happened: Authorization queried the warehouse directly. A routine credential rotation made the permissions table unreachable and locked out the entire user base.
Root cause: A shared analytical warehouse had become an availability dependency for a real-time product path.
What I did: A fail-static mirror on blob storage, refreshed as a whole-table snapshot, serving last-known-good grants within a bounded age.
What changed in the architecture: Permissions-unavailable became an explicit error type. The mirror expires rather than granting forever, and revocations write through immediately so the dangerous direction is the fast one.
What it taught me: When a system you do not control sits in your request path, decide in advance what you do when it is gone, and make sure that answer expires.

Evaluation

How I knew whether it worked

The evaluation suite is written against this repository's own regression history rather than against generic prompts. Every scenario traces back to something that actually broke.

Scenarios derived from incidents

Runaway verification loops, re-delegated screenings returning a different verdict each time, citations pointing at the wrong source, and a greeting that used to trigger a full investigation. Each incident became a scenario, so the regression that produced it cannot return silently.

Deterministic assertions run first, and are free

Required content, forbidden content, minimum and maximum source counts, maximum duration, citations-resolve, no-budget-exhaustion, and expected risk level. They are free and never flaky, so the judge is only invoked for scenarios that declare a rubric.

The harness asserts on the harness

The runner reads the guard and deadline counters directly rather than inferring behaviour from answer text. An assertion that no budget was exhausted is a statement about the run, not about the prose.

A deliberate false-positive test

One scenario screens the company that operates the tool. A false positive there scores the minimum: hedging is fine, inventing a hit is not. Paired with a true-positive scenario on a genuinely sanctioned entity and an ownership-chain scenario that has to traverse a parent company.

The judge is told today's date

Without it, a correctly cited recent event gets marked as a hallucination because it postdates the judge's training. A small detail that otherwise produces a steady drip of false failures.

Reproducibility as a checked property

A dedicated script verifies that the same factor set produces the same risk band, which is the mechanical form of the defensibility requirement.

Security and safety

What the system refuses to do

Authorization, isolation and auditability are structural: the properties come from how the system is built rather than from checks that have to be remembered.

Tenant isolation by construction

The agent filesystem is rooted at a per-user prefix in blob storage. Cross-user access is not blocked by a check; it is not expressible. That is also why the run-control channel needs no authorization layer of its own.

Deny by default

Roles are administrator, analyst, or unauthorized. An unknown user is unauthorized rather than defaulting to read-only.

Visible degradation

Permissions-unavailable is an explicit error type. The system never silently substitutes a weaker authorization answer for a stronger one.

Bounded staleness, fast revocation

The mirror serves stale grants only within a bounded age and then refuses. Revocations are written through immediately, so removal of access propagates in seconds while addition is eventually consistent.

Auditability of the evidence, not just the action

A per-investigation audit log plus the source registry and provenance tracking means the record shows not only what the system concluded but what it read to conclude it.

Secrets never in code

All credentials resolve from the platform secret manager at runtime.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

6code

Sanctions regimes screened

5code

Specialist sub-agents

4code

Independent budgets in the guard middleware

125code

Test files

~20attested

Analyst hours saved per dayBusiness-reported. No instrumentation exists in the system to measure this, and it is presented as an estimate rather than a measurement.

Business: Multi-source counterparty research that took hours per party returns in minutes, with a cited audit trail attached.
Engineering: A reusable agent harness (budgets, deadlines, citation registry, durable checkpoints, cross-instance control) that generalises well beyond compliance.
People using it: Analysts get a determination they can defend, with the sources visible and the coverage gaps stated rather than smoothed over.

Leadership and hindsight

What I influenced, and what I would change

Technical leadership

Turned the system's own incident history into its evaluation suite, so every production failure became a permanent regression check rather than a fix.
Established that domain knowledge belongs in reviewable files a compliance specialist can read, not in prompts only an engineer can change.
Made reproducibility a checked property with a dedicated script, rather than a claim in a design document.
Set the failure-direction convention the codebase follows throughout: fail closed on ambiguity, fail open on infrastructure.

What I would do differently

Fact polarity should have been in the schema from the first day. Adding it later cost almost nothing in code, and a permanently quarantined body of otherwise useful memory.
I would give the rubric an explicit escape valve for factors outside the closed list. Today a novel risk factor is simply not counted; it should be surfaced to the analyst as an unclassified observation.
The four budgets accreted one incident at a time. Knowing what I know now, I would start from a single budget object with several dimensions rather than four independent counters.
The parallelism across specialists is coarse: they run in parallel where independent, but independence is declared rather than derived. A dependency-aware scheduler would shorten the long tail of investigations.

Go deeper

The detail, for people who want it

Collapsed by default. The case study stands without any of it.

Why the orchestrator is constructed per request

The orchestrator is built fresh for every request rather than being a process-wide singleton. It looks wasteful and it is the decision the rest of the concurrency design rests on.

Because the object is per-request, the run deadline, the source registry and the checkpointer binding can all be per-turn fields on it. Nothing is shared between concurrent investigations, so nothing needs locking.

The alternative, a shared orchestrator with per-request context threaded through every call, puts a correctness burden on every future contributor. This way, getting it wrong requires deliberately hoisting state out of the request scope.

The quick-search mode does use a process-wide singleton, because it is single-agent, stateless and short. The two modes make the trade-off explicit rather than picking one globally.

The four budgets, and what each one caught

Identical-argument loop guard: the agent calls the same tool with the same arguments repeatedly. The cheapest failure to detect and the least common in practice.

Per-turn tool-call budget: the agent keeps inventing new calls rather than repeating one. Invisible to the loop guard by definition.

Per-turn delegation budget: the orchestrator re-delegates the same investigation with slightly reworded instructions. This is the one that produced four different risk verdicts in ten minutes.

Per-turn verification budget: the agent re-checks its own draft instead of gathering evidence. Sixteen calls in one turn, a third to a half of wall time.

All four return an actionable error to the model rather than killing the graph, so the turn degrades to an answer from partial evidence instead of returning nothing.

Recall, and why matching is exact

Before planning, the system surfaces what this analyst already knows about this party, dated and flagged for staleness.

Matching is exact-substring rather than fuzzy. Fuzzy matching would catch legitimate name variants, and it would also occasionally surface a different company's prior screening as though it were this one. In a sanctions context, surfacing the wrong company's history is worse than surfacing nothing.

A stored conclusion is context, never a substitute for a re-check. Sanctions lists move, so every query pays for a live screening regardless of what recall found. Caching the verdict would have been the obvious optimisation and would have been wrong.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

LangGraph + a deep-agent framework: The orchestrator and the five specialists, reached through a delegation tool.The graph model gives durable checkpointing and interruption for free, which is what makes a twelve-minute investigation resumable and a human-approval gate possible.
Gemini with search grounding, and Claude, both on Vertex: Grounded adverse-media research, and reasoning-heavy synthesis.Two families for two jobs, with fallback that never crosses between them because their tool-calling behaviour differs enough that a cross-provider fallback is a different system rather than a degraded one.
Blob storage as the agent filesystem: Sessions, checkpoints, evidence, run control and exports.Implementing the framework's filesystem protocol over object storage with per-user roots makes tenant isolation structural, and compare-and-swap on object generation gives a control channel without new infrastructure.
Cloud Run: The API and agent runtime, scaling between two and twenty instances.The instance count is the reason run control is external and the request timeout is the reason the hard deadline exists. Both constraints shaped the architecture more than the framework did.

6 more sections are written and hidden: constraints, evaluation, safety, leadership, deeper, stack.