Notes · Method · 7 August 2026
A holdout score tells you a model is consistent with its own test data. It tells you nothing about whether the model beats the spreadsheet it was built to replace. Finding that out means rebuilding the spreadsheet.
Every business we work with is already predicting the thing we have been asked to predict. Somebody is estimating the delivery date, or forecasting next month’s demand, and the estimate comes out of a rule: a lookup table, a formula in a spreadsheet, three days a planner adds on because six years on the job says add three days.
That rule is what we are competing with. It has no accuracy figure attached, because until now nobody has needed one, so before there is anything to compare a model against we have to go and build the comparison.
Finding the rule is fieldwork. It is almost never written down. It has exceptions added years ago for reasons nobody now remembers, it changes by site and by product line, and sometimes by who is on shift. We usually end up sitting with the planners and watching them do it, because watching is quicker than asking. We have also taken the documented version at face value and had to go back and do it again.
Then the rule has to be replayed across history under the same constraints the model will face. Apply a planner’s rule to an order from March using a field that was not filled in until April, and the rule comes out looking worse than it was. The error runs in our favour, which is the reason to be careful with it.
This is leakage. It is the most common reason a promising result does not survive production, and the rule for avoiding it is easy enough to state: at any point in the training history, a feature may only contain what the source system knew at that point. Not the value corrected the following week. Not a field that gets written when the truck leaves the yard, which predicts the delivery date almost perfectly and can never be used.
Enforcing it is slow. You need to know, field by field, when each system learned each value, and build features against that timeline rather than against the table as it looks now. A lot of source systems overwrite in place and keep no history at all, so sometimes the answer is that a feature cannot be used.
There is no clever way around this part.
Once the baseline exists there is one more thing to agree, in writing, before any modelling code gets written: the evaluation dataset, how the baseline is calculated, and the smallest improvement that would justify carrying on.
This is mostly about the conversation it avoids. A result lands, and the room starts working out what standard it should be judged against, which is very hard to do fairly once everyone can see where the ball landed.
It also makes a negative result worth paying for. If the model does not beat the planner by the agreed margin, that is an answer, measured against a test both sides signed, and the spending stops there.
More notes