Evaluation infrastructure · claim validity as the product

AI Benchmarking & Measurement Platform

You pick arms to compare (a deployment, a model, a provider, or a mix), pick metrics, pick a workload, and run. The platform performs the measurement, the paired statistics, and 25 validity gates that decide which claims the result can support. Each gate declares which claims it kills when it fails, so the report's supported and do-not-quote lists are derived from the data rather than written by hand. A gate that could not run is treated as a failure for its claims: not checked must never read as checked and fine.

2026evaluationplatformeconomicsleadership

What I owned

Sole author. Design, implementation, statistics, governance policy and deployment.

open the diagram →

Validity gates: 25codeValidity gates
Statistical tests in the paired battery: 9codeStatistical tests in the paired battery
Test suite: ~6,000 linescodeTest suite

12components
2request paths
3load stages
6decisions, argued
2failures, owned

Built onPython + FastAPICloud RunServer-sent events

01Problem 02My role 03Architecture 04Decisions 05What broke 06Impact

The problem

What was actually wrong

Teams were making platform-migration and model-selection decisions from benchmark numbers, and the numbers were not trustworthy. The existing internal tool had three structural defects: it never produced a real measurement, because the only persisted results were local probes that had all failed with a 503 and the interface did not distinguish that from success; it issued verdicts from a single observation per platform and printed conclusions like 'migration looks low-risk on latency'; and it could not reach a deployed agent at all, so the comparison that actually mattered was impossible to run.

Measuring latency is the easy part. The interesting problem is that a benchmark result stops being a database row the moment somebody quotes it. Someone is sent a number in March. In April the run turns out to be wrong. In May they are still quoting it, and nothing has ever told them otherwise.

Before this existed

Each probe computed its own metrics with its own definitions, so numbers from different arms were not comparable even when they looked like the same metric.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

Heavy-tailed latency distributions: Unpaired means and percentiles over agent latency are dominated by a handful of slow requests, and routinely produce large wins that are pure noise.
Small samples are the normal case: Real comparisons run on tens of tasks, not thousands. Any honest system has to be able to say that eight paired observations support a direction but not a precise number.
The subjects are non-deterministic: The thing being measured returns different text each time it is asked, and so does the judge grading it.
Results outlive their runs: Numbers get pasted into decks and quoted for months. Deleting a bad run does not reach the person holding it, and looks, from outside, indistinguishable from a cover-up.

My role

Sole author. Design, implementation, statistics, governance policy and deployment.

Designed

The claim-validity model: gates that declare which claims they invalidate, so report language is derived rather than written
The paired-statistics battery and its sample-size floors
The judge design, including reliability sampling at non-zero temperature
The retraction policy and the results changelog, including its machine-readable twin

Built

25 validity gates, the paired-statistics module, the analysis layer as a pure function of the database
Three drivers: deployed agent over its native streaming API, direct model probe, in-process agent runtime
A dynamic model catalogue with entitlement probing
Roughly 6,000 lines of tests, including a test that fails the build if a version ships without a changelog entry

Led

Set the organisational standard for what counts as a valid AI comparison
Introduced a retraction policy for published numbers: a governance mechanism, not a code change

Architecture

How it is put together

A run executes a workload against every arm through a driver, stores raw executions, and then analysis (a pure function of the database) produces the report. Gates run over the same data and determine which claims the report is permitted to make.

The same architecture, in prose

A run is defined by arms, a workload and a metric set. The wizard intersects the metric sets each selected arm can actually produce, so an impossible comparison cannot be configured.

The engine sends the same task to every arm, which is what makes the analysis paired.

Three drivers sit behind one interface: a deployed-agent driver speaking the platform's native streaming protocol, a direct model-probe driver that isolates the serving layer by removing tools and data access, and an in-process agent runtime.

Executions are stored raw. Analysis reads only from the database and computes the report payload as a pure function of it, so re-running analysis on an old run is safe and free.

The statistics module runs a paired battery: Wilcoxon signed-rank, sign test, geometric ratio, bootstrap confidence intervals, trimmed means, Holm correction across the family, plus order-effect and repeat-spread checks.

The judge grades answers blind and pointwise, several times per answer at non-zero temperature, and reports the spread.

25 gates run over everything. Each declares which claims it invalidates. The report's supported and do-not-quote lists are derived from the gate outcomes, never hand-written.

Every path, written out (2)· the walkthroughs above, as text

Running a comparison

Someone needs to know whether a candidate deployment is faster than the incumbent.

01
Define the run
Two arms are selected. The metric picker offers only metrics both arms can produce.
Intersecting the metric sets up front prevents a comparison that looks valid on screen and is meaningless in the report.
02
Same task, every arm
Each workload task is sent to both arms.
This is the property that licences paired analysis. Pairing removes task difficulty, which dominates the variance.
Fails by: If one arm rate-limits, the completed set differs between arms. The survivorship gate catches exactly this.
03
Store raw
Every execution is persisted in full rather than summarised.
Analysis stays a pure function of the database, which means a corrected price table reaches an old report without re-running anything.
04
Paired battery
Wilcoxon, sign test, geometric ratio, bootstrap CI, trimmed mean, all Holm-corrected across the family.
Holm matters more than it sounds: left alone, a nightly comparison manufactures a regression roughly every third night from noise.
Fails by: A percentile with fewer than about ten observations beyond it is an artefact of the worst two samples, and the sample-size floor blocks the claim.
05
Grade quality, blind
The judge scores each answer without being told which arm produced it, several times, at non-zero temperature.
Without a quality number, the arm that refused to answer wins on latency and cost. The repeat spread tells you whether the quality number is worth anything.
Fails by: A judge that scores the same answer 45 and 80 fails the reliability gate and the quality claim is withheld.
06
Decide what may be claimed
All 25 gates run. Each one that fails removes the specific claims it declares.
The supported and do-not-quote lists are derived here. Nobody writes them, so they cannot drift from the numbers.
07
Publish with its limits attached
The report renders the numbers alongside the claims the run does and does not support, stamped with the analysis version.
Eight paired observations supports a direction but not a precise number. Saying so on the report is the product.

Failure path: a published number turns out to be wrong

A run was quoted in a decision meeting six weeks ago. A defect is found that invalidates it.

01
Retract, do not delete
The run is marked retracted with a reason and a date. Nothing is removed.
Deleting the evidence is indistinguishable from a cover-up, and it breaks the only channel that still reaches the person holding the number.
02
One gate invalidates everything
The retraction gate fails, and because it declares the entire claim vocabulary, every claim in the report is withdrawn at once.
The withdrawal is the headline. There is no partial retraction where some numbers survive because nobody thought about them.
03
The old link still works
The share link resolves. The report renders in full, with the retraction reason and date on top.
The person quoting the number will click their own link before they check a registry. That link has to be the thing that tells them.

What changes as load grows· the scale stages, as text

One comparison

code

2 arms · a handful of paired cases

A run is defined, the engine drives both arms over the same cases, results land in the execution store, and the paired statistics run. At this size the gates almost always fail: a handful of cases cannot support a claim, and the platform says so rather than reporting a winner.

Gives first: Sample size, and that is the point. The honest output of a small run is 'no claim', which is the behaviour most benchmarking tools get wrong.

A release gate

modelled

several arms · a full case set · judged outputs

The judge becomes the expensive component: every output in every arm needs scoring, so judge cost scales with arms × cases while the thing being measured scales with cases alone. Conformance checks matter more, because at this volume a driver that silently misconfigures one arm produces a clean-looking result. All 25 validity gates are load-bearing. This is the run someone will quote in a decision meeting.

Gives first: The judge. It is both the cost driver and the correlated error source: one judge scoring both arms means a judge bias does not cancel out, it just fails to show up as noise.

Continuous measurement

modelled

runs on every deploy · results accumulating over months

Old results become the liability. A report published under an assumption that later turns out to be wrong is still being cited, so retraction stops being a nicety and becomes infrastructure: a report needs to be able to un-say itself, and anything that cited it needs to know. The execution store grows without bound unless runs carry an expiry, and re-running an old comparison against a since-updated model is a different experiment wearing the same name.

Gives first: Claim provenance rather than compute. The hard problem at this scale is knowing which published claims are still true, not producing more of them.

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

Every gate declares which claims it invalidates. The report's supported and do-not-quote lists are computed from gate outcomes.

Context

Caveats written by hand go stale the moment the numbers change, and the caveat is the first thing dropped when a chart is pasted into a deck.

Alternatives, and why not

Hand-written caveats per reportThey drift from the numbers above them, and nothing detects the drift.
A single overall confidence scoreCollapses independent failures into one number and tells nobody which specific claim is unsafe.

Rationale

Validity is not a property of a run, it is a property of a claim about a run. The same data can perfectly well support 'A is directionally faster' and not support 'A is 12% faster'.

What it cost

Requires a claim vocabulary to be defined up front and maintained. Adding a new kind of claim means deciding which gates govern it.

Outcome

Reports say what they cannot support, in the same breath as what they can, and the two can never disagree.

What it moved in the diagram

25 validity gates
Report

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

The tool it replaced had never produced a real measurement

What happened: The predecessor's only persisted results were local probes that had all failed with a 503, and the interface displayed them indistinguishably from successes. It compared one observation per platform and printed conclusions about migration risk. It could not reach a deployed agent at all.
Root cause: Nothing in the design distinguished 'the measurement failed' from 'the measurement succeeded and the answer is this'. With no notion of validity, an error is just another value.
What I did: Rebuilt around the three defects rather than patching them: a success gate that separates failure from result, sample-size floors that block verdicts from n=1, and a deployed-agent driver so the comparison that mattered became possible.
What changed in the architecture: Validity became a first-class object in the data model rather than a property of whoever reads the chart.
What it taught me: A measurement tool with no concept of an invalid measurement will confidently report its own failures. That is worse than having no tool, because it is trusted.

The fastest arm is usually the one that refused to answer

What happened: Latency and cost comparisons systematically favour arms that produce short, evasive or refused answers.
Root cause: Speed and cost are trivially measurable; usefulness is not. Any battery that omits quality has a built-in bias toward not answering.
What I did: Made quality a required part of the battery, with a judge whose independence and reliability are themselves gated.
What changed in the architecture: Conformance checks added alongside quality: schema validity, response format, expected tool calls, and determinism, following the practice of treating every ground-truth argument value as a list of acceptable values.
What it taught me: A model that returns two paragraphs of apology where the caller's code called json.loads scores well on every number in the report. Conformance is the check that catches it.

Evaluation

How I knew whether it worked

The platform is evaluation infrastructure, so its own tests are about whether its measurements can be trusted rather than whether its code runs.

The changelog is enforced by a test

The results changelog documents how a report is computed, with entries classified additive or can-contradict. A test fails the suite if a version ships without an entry, so the governance artifact cannot silently fall behind the code.

Conformance expectations are citable

Expected schema and expected tool calls live on the task, versioned and pinned at run creation and reviewed by a subject expert, not typed into a wizard by whoever ran it. Verification is pure, so the same answer and the same task always produce the same verdict, which is what lets a stored run be re-checked after the rules change.

One canonical definition per metric

Time-to-first-token means the same thing for a raw model call and for a streaming agent response. In the tool this replaced, each probe computed its own and the results were not comparable.

Entitlement probing, not a hard-coded list

The model catalogue discovers what the project may actually call, and gates models it is refused. A definitive refusal is never retried. Distinguishing a permanent policy refusal from a transient error is the difference between a clean catalogue and a retry storm.

Security and safety

What the system refuses to do

Single sign-on with an explicit allow-list, and a deliberate escape hatch for local development.

SSO with an allow-list

Access is by enterprise single sign-on against a seeded allow-list rather than by domain membership.

A command-line escape hatch, on purpose

Local development has no SSO app to redirect to, so grants can be managed from a CLI. Naming the escape hatch and scoping it beats having people disable auth to get work done.

Retraction as a governance control

The ability to withdraw a published number is deliberately a human decision, not an automated one. Code enforces what a retraction means; people decide when one happens.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

25code

Validity gatesEach declaring the claims it invalidates

9code

Statistical tests in the paired batteryWilcoxon, sign test, geometric ratio, bootstrap CI, trimmed mean, Holm, order-effect, repeat-spread, parity

~6,000 linescode

Test suiteIncluding conformance, changelog and measurement-loop tests

0code

Predecessor's real measurementsEvery persisted result was a failed local probe displayed as a success

Business: Platform-migration and model-selection decisions moved from single-observation verdicts to paired comparisons with stated validity.
Engineering: Comparisons that were previously unrunnable (deployed agent against deployed agent, over the real streaming API) became routine, and their results carry their own limits.
People using it: The people quoting benchmark numbers now have a mechanism that tells them when one has been withdrawn, through the link they already have.

Leadership and hindsight

What I influenced, and what I would change

Technical leadership

Defined what counts as a valid AI comparison in the organisation, and encoded it so the standard applies without me in the room.
Wrote a retraction policy, explicitly a decision that has to be agreed by people rather than by code, for numbers that have already left the building.
Rebuilt an inherited tool by diagnosing why it was structurally incapable of being right, and documented the three defects as design requirements without ever making it about the people who wrote it.
Generalised the model from 'site' to 'arm' so one statistical battery serves deployments, models and providers, rather than three near-duplicate tools.

What I would do differently

The claim vocabulary should have come first. I built gates and then discovered I needed a shared language for what they invalidate; doing it in that order meant one revision of every gate.
Reliability sampling should be on by default with a low k rather than off by default with a good k. The people most likely to leave it off are the people most likely to be misled.
I would add a cost-per-valid-claim view. The platform can tell you a run was too small to support a claim, but not yet what it would cost to make it large enough.
Dropping the charting dependency and hand-building the visuals was the right call for bundle size, and I would do it again.

Go deeper

The detail, for people who want it

Collapsed by default. The case study stands without any of it.

What the 25 gates actually check

Run integrity: retracted, synthetic, success, sample_size, percentiles, work_parity, token_accounting.

Comparison fairness: subject_equivalence, comparison_axis, runtime_equivalence, cache_capability, effort_cache, client_path, vantage_agreement, order_effect, survivorship.

Measurement conditions: latency_decomposition, rate_limit, throughput, pricing, determinism.

Quality trustworthiness: quality, judge_independence, judge_reliability, position_bias, conformance.

The grouping matters less than the property they share: each one names the claims it kills. A cache_capability failure does not invalidate a latency claim, and the report reflects that precisely rather than adding a general warning banner.

Why percentiles need a sample-size floor

A p95 estimated from twelve observations is determined by the single worst one. It is not a percentile, it is a maximum with a decimal point.

The rule enforced is roughly ten observations beyond the percentile before the estimate stops being an artefact of the two or three worst samples, applied per percentile, so a run can support a p50 claim and not a p95 claim from the same data.

This is the same discipline as the effect-size floor in regression detection, from the other direction: one stops you claiming precision you do not have, the other stops you claiming significance that does not matter.

Graceful degradation of the statistics

The statistics module treats its scientific-computing dependency as optional. Every function degrades to a documented approximation when it is absent rather than failing or silently changing method.

Sign convention is fixed globally: the difference is always base minus other, so no reader ever has to work out which direction a positive number means.

Baselines store the workload hash of the run they came from, and a comparison refuses to proceed when it does not match. A regression check against a different workload is not a regression check.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

Python + FastAPI: The API, the run engine, the gates and the statistics.The statistical work and the driver work belong in the same process as the data model; splitting them would have meant serialising executions across a boundary for no benefit.
Cloud Run: Deployment, with Cloud Build and a drift pipeline.Benchmark runs are bursty and long. A request-scaled container with a generous timeout fits that shape without maintaining a scheduler.
Server-sent events: The deployed-agent driver speaks the platform's native streaming protocol.Measuring time-to-first-token honestly requires reading the stream the way a real client does, not waiting for a completed response.

6 more sections are written and hidden: constraints, evaluation, safety, leadership, deeper, stack.