AI systems
Ship the rule, not the model
Roughly half of what arrives described as an AI problem is a rules problem with bad documentation. Telling the difference early is the cheapest thing we do.
A client came to us with a classification problem. Incoming requests had to be routed to one of eleven teams, and the existing process was a person reading each one and deciding. They wanted a model.
We asked for six months of historical routing decisions before quoting anything. Two weeks later we could tell them that nine of the eleven teams were determined entirely by two fields already present on the form. A lookup table handled 94% of volume. The model only had to exist for the remaining 6%.
Why this keeps happening
The rules are usually real, they are just undocumented. They live in the head of whoever has done the job longest, and nobody has had a reason to write them down. When that person is asked how they decide, they describe the hard cases, because the hard cases are the ones they remember. The easy 94% is invisible to them precisely because it is easy.
So the brief arrives describing an inference problem, and it is genuinely how the work feels from the inside. It is just not what the data says.
What we do instead
- Ask for the historical decisions before the requirements. What was decided is more reliable than how people describe deciding.
- Fit the dumbest possible baseline first. A lookup, a decision tree, a regex. Measure it honestly.
- Only then ask what a model adds over that baseline, on the same holdout, in the same units.
- If the answer is a few points, ship the baseline and say so.
The part that is uncomfortable
This costs us revenue. A lookup table is a fortnight; a model with an evaluation harness, a fallback path and a monitoring story is a quarter. We have talked ourselves out of the larger engagement more than once.
It is still the right call. A deterministic rule is cheaper to run, faster to execute, testable in CI, explainable to an auditor and debuggable at three in the morning. A model is none of those things for free. It has to earn the difference, and when it does we will happily build it.
Where the model did earn it
The remaining 6% turned out to be the requests where the form fields contradicted the free text. Those genuinely needed reading. We built a small classifier for exactly that slice, with the lookup as the fallback whenever confidence dropped below threshold.
It runs on about one twentieth of the volume the original brief assumed, which means it costs about one twentieth as much to operate. That is the whole argument.