Architecture
These are not unrelated projects
Seven ideas recur across everything on this site. Some I chose deliberately; two of them I only noticed after they had turned up independently in three systems. Each is stated as a claim, argued for once, and then evidenced by the systems that instantiate it.
Cheap deterministic checks before expensive judgement
Run the free check first. Invoke the model only where judgement is actually required.
A model-based decision costs money, latency and variance. Most of the inputs it is handed do not need any of the three. They are resolvable by a comparison, a threshold or a string distance. Putting a deterministic stage in front does more than save cost: it shrinks the surface where non-determinism can enter, so the cases that do reach the model are the ones where a human would also have had to think. The sequencing is the same everywhere it appears, and it is the first thing I reach for when a system is too slow or too expensive.
4 systems that do this
- AI Benchmarking & Measurement Platform
Deterministic assertions (required content, forbidden content, bounds, resolvable citations) run first and cost nothing. The judge is invoked only for scenarios that declare a rubric.
- Sanctions Screening Automation
A fuzzy matcher generates candidates cheaply across the customer-by-list cross-join; the model adjudicates only the ambiguous band where the answer is genuinely in doubt.
- Carrier Root-Cause Validation Agent
A deterministic router and pre-check classify what structured fields already settle, before anything reaches the agent.
- What a Production Agent Fleet Actually Costs
Routing signals available before the model call, the ones that cost nothing to read, carry the routing decision; anything requiring inference to obtain was rejected as a routing input.
Fail closed on ambiguity, fail open on infrastructure
The two kinds of uncertainty deserve opposite defaults, and conflating them is how systems become either dangerous or brittle.
When the system is unsure what is true, guessing is the failure, so it must refuse, quarantine, or state the gap. When the system's own supporting infrastructure is unavailable, refusing is the failure, so it must degrade to a known-safe behaviour and make the degradation visible. Getting these the wrong way round produces the two classic outcomes: a confident fabrication, or a middleware outage that takes the product with it. Every system I build states its direction explicitly for each boundary.
3 systems that do this
- AI Compliance Investigation Platform
Fact polarity quarantines anything not explicitly affirmed or negated and never renders it into a prompt; the risk rubric treats unknown as a coverage gap rather than as low risk. Meanwhile authorization fails static, serving last-known-good grants within a bounded age when the warehouse is down.
- What a Production Agent Fleet Actually Costs
The routing middleware returns today's behaviour on every error path. A cost optimisation that can take the product down is not an optimisation, and a fail-open default is what makes it deployable.
- AI Benchmarking & Measurement Platform
A skipped or refused arm is scored as a failure, not excluded from the comparison, because excluding it makes the arm that refused to answer look like the fastest one.
Move domain knowledge out of prompts into reviewable artefacts
The people who own a rule should be able to read it, and changing it should not require an engineer.
Domain logic buried in a system prompt has two problems. It is invisible to the specialist who actually owns it, and every change to it is an engineering ticket on an engineering timeline. Moving it into structured files (skill packs, knowledge models, closed vocabularies, semantic layers) makes the logic auditable by the person best placed to audit it, and makes a rule change an edit rather than a release. It also, incidentally, makes the prompt shorter, which is its own benefit. This instinct showed up independently in three of my systems before I recognised it as a principle.
3 systems that do this
- Carrier Root-Cause Validation Agent
Eighteen structured knowledge files, including a typed decision tree with an entry-point filter, reason-code branching and declared output columns. The agent follows the tree; an analyst who cannot read Python can review it.
- Enterprise NL-to-SQL Agent Platform
Thirty-seven markdown skill packs loaded per domain by manifest, and a semantic-layer direction that replaces both raw schema dumps and hand-written capability code with one declarative description.
- AI Compliance Investigation Platform
Domain skill packs for export control, sanctions programmes, adverse-media methodology and beneficial ownership, plus a risk rubric built on a closed factor list rather than described criteria.
Guard at the boundary, not in the prompt
An instruction to a model is a strong prior. For a property that must hold every time, use a control.
Prompts are honoured most of the time, which is a different thing from being enforced. Anywhere a property has to hold every single time (no writes, no cross-tenant reads, no query text in an answer, no unbounded spend), the enforcement belongs at the lowest layer that can express it. That layer is usually the connection, the filesystem root, the middleware seam or the tool-call boundary, and putting it there means code paths that do not exist yet inherit the property automatically. The prompt still says it; the prompt is just no longer the thing relied upon.
3 systems that do this
- Enterprise NL-to-SQL Agent Platform
A listener on the database engine rejects any non-read statement, an allow-list constrains the agent filesystem so credentials and system prompts are unreachable, and two security checks share one parsing module so they cannot drift apart.
- Shipment Milestone Tracking Agent
An after-agent middleware strips query text from the final answer even though the prompt already forbids it, with bounded regex repetition so a long answer cannot cause catastrophic backtracking.
- AI Compliance Investigation Platform
Four independent budgets at the tool-call boundary plus a shared wall-clock deadline, and per-user storage roots that make cross-tenant access inexpressible rather than merely forbidden.
Measure the thing you are about to trust
An instrument you have not audited is confidence you have not earned.
The most expensive mistakes I have found were not in systems. They were in the things measuring the systems. A test suite that returns green regardless of behaviour. A gold set recording the model's own predictions as ground truth. A baseline nobody can run. A join that silently multiplied the training rows. None of these announce themselves; all of them read as evidence. So before a measurement is allowed to gate a decision, it gets measured itself, and the result gets published whether or not it is flattering.
4 systems that do this
- Enterprise NL-to-SQL Agent Platform
Audited the platform's own evaluation suite before letting it gate a cost programme: 265 of 271 legacy assertions asserted nothing. The published conclusion was to not gate on it.
- Freight Pricing Anomaly Detection
The gold set was recording the model's predictions as reviewer ground truth because the review form pre-ticked its anomaly box. Corrected precision came out several times lower, and the corrected figure is the one the programme used.
- Warehouse Pallet Estimation Model
Implemented the incumbent rule of thumb in code so the improvement was a comparison on identical inputs rather than a claim against a remembered number.
- Transit Time Prediction Service
A grain-and-fanout gate between extraction and modelling fails the run if a join has multiplied the rows, the failure mode that produces plausible metrics on duplicated evidence.
Say what you cannot say
A number without its validity condition attached will be reused where it means nothing.
Every measurement has a boundary beyond which it does not support a claim. Most reporting drops that boundary on the way to the summary, and the number then travels into a slide, into a decision, into someone else's plan, with its caveats stripped. So I attach the limit to the number mechanically rather than editorially: gates that derive a do-not-quote list, a retraction policy with a named owner, an open-questions section that stays open, a metric labelled untrustworthy in the report that contains it. This makes reports less impressive and considerably more useful.
3 systems that do this
- AI Benchmarking & Measurement Platform
Twenty-five validity gates, each declaring which claims it invalidates when it fails. The report's supported and do-not-quote lists are derived from gate state rather than written by hand, and a written retraction policy covers what happens when a published result turns out to be wrong.
- Freight Pricing Anomaly Detection
Recall is reported and explicitly labelled untrustworthy, because reviewers only ever worked the top of a ranked queue, so unflagged transactions are unlabeled rather than negative.
- What a Production Agent Fleet Actually Costs
The study ends with the questions it could not close and a taxonomy separating what was verified in code from what was observed, calculated, or modelled, with modelled figures published as ranges, never as results.
Pipeline first, then the agent on top of it
Build the offline computation that makes the question cheap, then put a conversational surface on it as a domain in a shared platform.
An agent that computes at query time pays twice: once for the query, and again for the tokens spent reasoning over its result. Doing the expensive computation offline collapses the question to a small lookup, and the agent becomes a thin, fast, cheap surface over something already correct. The second half matters as much as the first: each of these was built standalone and self-contained, then absorbed into the shared platform as a domain. Designing for absorption rather than for territory is what turned four separate analytics assets into one conversational product.
4 systems that do this
- Inventory Intelligence Platform
A weekly pipeline collapses a multi-million-row history into one row per item and site; the agent answers any history question with a single lookup. Later absorbed into the platform as a domain.
- Freight Mode-Shift Optimization Engine
A weekly optimisation engine, then a mode-shift agent over its output.
- Transit Time Prediction Service
A trained transit-time model exposed as an additional data source inside the agent platform: a model an agent can reason over rather than a report someone has to read.
- Enterprise NL-to-SQL Agent Platform
The excess-and-obsolete domain agent was built on top of my own prior analysis of that domain, which is the same pattern arriving from the other direction.