AI systems
What an evaluation set costs, and what it saves
Nobody budgets for the evaluation set. It is the single highest leverage artefact in an AI engagement, and the only reason you can change anything later without fear.
An evaluation set is a few hundred real inputs with the answer you would accept written next to each one. It takes a domain expert somewhere between three days and three weeks to build properly. That is the cost, and it is almost always the item a client tries to cut.
What you cannot do without one
- Change the prompt. Any change is a coin flip you cannot score.
- Change the model. A cheaper one might be fine, but you will never dare find out.
- Change the retrieval strategy, the chunk size, the reranker, the temperature.
- Tell a regulator what the system does, in numbers, on the record.
- Know that last month’s fix did not break something else.
Without an evaluation set, every subsequent change is argued from vibes and anecdote. Someone tries a new prompt, pastes three examples into a channel, and everyone agrees it looks better. Six weeks later a different three examples look worse and nobody can say when it regressed.
Building one that is actually useful
Sample from real traffic, not from imagination. Weight it toward the cases that matter rather than the cases that are common, then keep the common ones too so you notice if the easy path breaks. Include the inputs where the right answer is "I do not know", because a system that never abstains is a system that will confidently invent things.
Write the expected answer as the thing you would accept, not the thing you would love. A grader that demands perfection reports noise.
What it buys
On one engagement the evaluation set cost eleven working days of a senior clinician’s time. It then let us drop from a frontier model to a much smaller one on 70% of traffic, with a measured quality difference inside the noise band. The saving paid for those eleven days in the first six weeks of production, and has been paying since.
More importantly, it meant we could say what the system did. Not "it works well", but a number, against a set, with the failures listed. That is the difference between a demo and something you can put into a regulated process.
A system you cannot measure is a system you cannot change. In practice that means it is a system that will slowly rot while everyone is too nervous to touch it.