Write the eval before you write the prompt

Almost every engagement starts the same way. Someone demos a retrieval system that answers three questions beautifully, everyone agrees it is nearly done, and then it spends four months not shipping. The demo was never the hard part. The hard part is that nobody can say whether a change made it better.

The argument you are actually having

Without an eval set, every proposed change becomes a matter of taste. Someone swaps the embedding model and the team spends a fortnight deciding whether the results “feel” worse. Someone rewrites a prompt and the regression only shows up three weeks later, in a customer complaint about a case nobody thought to retry.

This is not a tooling problem. It is that the system has no definition of correct, so correctness defaults to whoever is most senior in the room.

What a useful eval set looks like

It does not need to be big. Our rule of thumb is about 200 cases to start, drawn from three places:

• Real queries from real users. Pull them from logs, not from imagination. The questions people actually ask are shaped differently from the ones you invent at a whiteboard.
• The cases that have already burned you. Every production incident becomes a permanent test. This is the cheapest institutional memory you will ever buy.
• Cases you expect to fail. Out-of-scope questions, ambiguous phrasing, queries that should return nothing. A system that never says “I don’t know” is not calibrated, it is confident.

Label the expected answer, or at minimum the documents that must appear in the retrieved set. Partial labels still beat no labels.

Wire it into the deploy

An eval that lives in a notebook will be run once and then quietly abandoned. The version that changes behaviour is the one that blocks a merge request:

npm run eval -- --threshold 0.82

Fail the build when recall drops. Print the diff of which cases changed direction, because the aggregate number hides the interesting part — a change that fixes thirty cases and breaks five is a different decision from one that fixes thirty and breaks nothing.

The payoff is speed, not rigour

The usual objection is that this is heavy process for a small team. It is the opposite. Once a model change is a number rather than a debate, upgrades land in an afternoon instead of a fortnight. The teams we have seen move fastest are the ones that made it boring to be wrong.