Why every AI feature I ship now starts with the eval set, not the prompt. A short brief on what changes when you reverse the order.
There is a version of AI product development that goes like this: write a prompt, run it against a few examples, ship it when it looks good. I did this for the first eighteen months. It works until it does not, and when it stops working you have no idea why.
The version I use now goes in the opposite order.
Start with the eval, not the prompt
Before I write a single line of a system prompt, I define what correct looks like. That means writing a set of test cases — inputs I expect the system to handle, outputs I expect it to produce, and criteria for judging whether a response passes or fails.
This is not a novel idea. It is how software testing has worked for decades. What is novel is how consistently the AI industry ignores it.
The reason teams skip eval-first is that it feels slow. You have something running in five minutes with a prompt. Writing a proper eval set takes a day or two. The five-minute version wins every sprint planning meeting until someone asks why the feature behaves differently in production than in the demo.
What a minimal eval set looks like
I am not describing something elaborate. A working eval set for most AI features has three parts:
Golden cases. Ten to twenty inputs where you know exactly what the right answer is. Not approximate — exactly. These are your regression tests. If a prompt change breaks a golden case, you catch it before it ships.
Adversarial cases. Inputs designed to break the system. Ambiguous phrasing, edge cases, content the model should decline, inputs that previously caused problems in production. These grow over time. Every incident adds a case.
Calibration cases. Inputs where reasonable people might disagree on the right answer. These are not pass/fail. They exist to expose where your judgment criteria are vague, so you can tighten them.
The eval set is also your specification. If you cannot write a test case for a behavior, you do not actually know what you want the system to do. The act of writing tests forces precision that prompt writing does not.
What changes when you reverse the order
When evaluation comes first, three things shift.
Iteration becomes measurable. Instead of "does this feel better," you have a score. Prompt changes that improve the score ship. Changes that do not improve the score get discarded, regardless of how clever they look.
Regressions are visible immediately. I have caught changes that improved the primary metric while quietly breaking three adjacent behaviors. Without a broad eval set, those regressions reach production.
Confidence in models changes is grounded. When Anthropic or OpenAI releases a new model version, I run the eval set against both models and look at the delta. The answer is data, not intuition.
The cost of doing it the other way
I can be specific about what the alternative looks like. At Warner Bros. Discovery, before we formalized evaluation for the personalization features, we shipped a prompt change that improved click-through rate in manual testing by what felt like a significant margin. In production, it degraded performance for a specific content category that was underrepresented in our manual checks. We caught it in monitoring two weeks later.
After we built a proper eval harness, that class of error stopped happening. The eval set included representative examples from every content category. A change that looked good in spot checks but failed on the underrepresented categories would not have passed.
The practical starting point
If you are building an AI feature and you do not have an eval set, stop writing the prompt and spend the next four hours building one instead. Twenty golden cases, five adversarial cases. Score them manually. That is your baseline.
Everything you build on top of that baseline is now something you can measure. The prompt is not the product. The evaluation framework is the product. The prompt is just the current best answer to a question the eval set defines.
That distinction — between the thing you ship and the system that tells you whether the thing you shipped is working — is what separates AI features that hold up from AI features that require constant supervision.
