Decision
The existing manual process was replicated as a pipeline stage so both methods run on the same inputs.
Context
The incumbent was a practice, not a system, so there was nothing to compare against directly.
Alternatives, and why not
- Compare against remembered or reported figures for the old methodDifferent inputs, different periods, and a comparison nobody can reproduce. It is the easiest way to produce an improvement that evaporates under scrutiny.
Rationale
An improvement claim is a comparison. If the thing being compared against cannot be run, the claim cannot be checked.
What it cost
Building a version of something you intend to replace.
Outcome
Mean absolute percentage error of 31.9% for the replicated incumbent against 6.0% for the model, on identical inputs.