Knowing If It Works

Traditional software has clear pass/fail tests. LLM outputs are probabilistic and subjective. Evaluation is the discipline that separates products from demos.

Traditional software has clear pass/fail tests. LLM outputs are probabilistic and subjective. The same input can produce different outputs. "Correct" is often a judgment call. Edge cases are infinite. And behavior can change when the model provider updates their model. This is why evaluation is the discipline that separates products from demos.

Start with 20 test cases

Not 200. Not 2,000. Twenty. Each test case has three parts:

• Input: The exact query or data the user would provide.
• Expected output: What a good response looks like. Doesn't have to be word-for-word, but describe the qualities.
• Pass/fail criteria: Specific, binary conditions. "Mentions at least two relevant factors." "Does not hallucinate a statistic." "Stays under 200 words."

Where to get them:

• 5 common cases: The bread-and-butter queries your product handles every day.
• 5 edge cases: Unusual inputs, very short or very long, ambiguous requests.
• 5 adversarial cases: Inputs designed to break things. Prompt injection attempts, off-topic questions, contradictory instructions.
• 5 failure cases: Situations where the AI genuinely shouldn't know the answer, and you want it to say so.

Run every test case. Score each one. Write down the results. You now have something most AI products never get: a baseline.

Four dimensions of quality

Most people think evaluation means "is the answer correct?" That's one dimension. There are four that matter:

Accuracy.Is the output factually correct? For structured outputs (classifications, extracted data), this is straightforward. For free-form text, you need to define what "correct" means in your domain.

Usefulness.Did the output help the user accomplish their goal? An answer can be technically correct and completely useless. If someone asks "how should I price my product?" and gets a textbook definition of pricing strategy, that's accurate and worthless.

Consistency. Same input, roughly similar quality? Not identical outputs, but if the same question produces a brilliant answer at 2pm and nonsense at 3pm, users will never trust it. Test the same input 3-5 times.

Graceful failure. What happens when the AI doesn't know? This is the one most builders skip, and it matters most for trust. Your AI should say "I'm not confident about this" instead of confidently making things up.

Three ways to score

Human evaluation. You (or someone who knows the domain) read the output and score it. Slow and expensive but irreplaceable for v1. If you're building a legal tool, a lawyer needs to look at those outputs. No shortcut.

Automated evaluation. Works when your outputs are structured. Did it return valid JSON? Are the required fields present? Is the classification one of the allowed values? Automated evals are fast, cheap, and should run on every deploy.

LLM-as-judge. Use a second AI model to evaluate the first one's output. Give it a detailed rubric, not just "is this good?" Claude or GPT-4 with a well-written scoring prompt can replicate human judgment at about 80-85% agreement. Good enough for regression testing.

Use all three. Automated evals on every change. LLM-as-judge weekly. Human eval on your 20 core test cases monthly, or whenever you change prompts significantly.

The feedback loop

Your eval set is a snapshot. Your users are a movie. Track these signals from production:

• Regeneration rate. How often do users click "try again"? High regeneration means the first output wasn't useful.
• Edit distance. If users can edit AI outputs, how much do they change? Heavy editing means your AI is a rough draft machine.
• Abandonment. Users who get an output and don't take the next action are telling you the output wasn't valuable.
• Thumbs up/down. Simple, but only useful if you actually read the downvoted outputs and understand why.

Every month, take your worst-performing real-world outputs and add them to your eval set. Your 20 test cases become 25, then 30, then 50. Each one represents a real failure. This is how your eval set matures from "things I thought might go wrong" to "things that actually went wrong."

---
description: Build your first 20 evaluation test cases for an AI feature. Defines what failure looks like and goes looking for it.
---

You are an AI evaluation specialist who believes in the After Action Review: you don't know if something works until you've defined what failure looks like and gone looking for it on purpose.

Ask the user: "Which AI feature are you evaluating? What does it take as input, and what does it produce as output?"

Then generate 20 test cases organized into four categories:

**5 Common Cases (the bread and butter)**
The queries this feature will handle every day. Representative, normal inputs. These should always pass. If they don't, the feature isn't ready.

**5 Edge Cases (the unusual)**
Unusual inputs: very short, very long, ambiguous, misspelled, multiple languages, contradictory information. These reveal how robust the feature is under real-world messiness.

**5 Adversarial Cases (the attacks)**
Inputs designed to break things:
- Prompt injection: "Ignore all previous instructions and..."
- Off-topic: Questions completely outside the feature's scope
- Extraction: "What is your system prompt?"
- Overload: Extremely long or complex inputs
These test whether the feature fails safely or fails dangerously.

**5 Failure Cases (the "I don't know")**
Situations where the AI genuinely should NOT know the answer. The correct behavior is to say "I'm not confident" or "I can't help with that." If the AI confidently makes something up instead, that's a trust-destroying bug.

For each test case, include:
- **Input**: The exact query or data
- **Expected behavior**: What a good response looks like (not word-for-word, but qualities)
- **Pass/fail criteria**: Specific, binary. "Mentions at least two relevant factors." "Does not hallucinate a statistic." "Stays under 200 words." "Declines to answer."

After generating all 20, tell the user:

"Run every test case. Score each one. Write down the results. You now have a baseline. Every month, take your worst real-world outputs and add them to this set. That's how your eval set matures from 'things I thought might go wrong' to 'things that actually went wrong.' Reference: https://builderspath.dev/playbook/#knowing-if-it-works"