Your AI product should get smarter every time it fails.

Most failures disappear without teaching the product anything.

A bad output happens. Someone catches it. Maybe it lands in Slack. Maybe a PM opens a ticket. Maybe an engineer tweaks the prompt until that one example works again.

Problem solved, apparently.

The symptom is gone, a human did the repairing, and the system knows nothing it did not know before. That is maintenance.

I learned this building Stem, my own AI system. On June 10 its retrieval eval fell from 83% to 33% the day I switched the model to JSON tool calls. The obvious reading was that JSON mode made the model worse. Revert it.

The transcripts said otherwise. Stem’s own grep had returned (no matches) for a string that sat in four files. Two runs had ended in under three seconds with a full plan in the model’s reasoning channel and nothing in its answer, and the loop had treated empty as final. One gold answer no longer matched a file the pipeline rewrites during the day. Three harness bugs, one stale fixture, zero model bugs. After the fixes: 100%, and faster.

That day changed the question I ask after a failure, from “how do I fix this output?” to “what did this failure teach me about the product?”

Own the definition of good.

That incident is what evals discipline is built to catch, and the discipline starts before any failure happens.

The core move is simple to say: own the definition of good, then turn that definition into a system that can be run again and again.

Strip the jargon and an eval is a test with opinions. A task is one test case, a trial is one attempt at it, a dataset is a pile of tasks, and a grader is the logic that decides whether the attempt was any good. The trace is the path the system took, the outcome is where it ended up, and the harness is the plumbing around all of it. That is nearly the whole vocabulary.

The dataset is where the leverage lives. A golden set is assembled from real user cases, historical examples, known failures, edge cases, and the scenarios where a miss actually costs money or trust, never from imagined usage.

Graders come in a hierarchy, cheapest first: deterministic checks, then rules-based checks, then an LLM judging another LLM once it is calibrated against human labels, with humans themselves at the top, spending the scarce resource on judgment.

Run evals at three altitudes too. Capability evals ask whether the system can do a class of task at all, regression evals ask whether yesterday’s change broke last month’s win, and production evals ask how the thing performs on real traffic, which is always weirder than the dataset.

One rule I now apply to everything: every meaningful eval should be wired to a decision. Launch, rollback, model choice, routing, scope, permissions, escalation, roadmap. If no decision would change based on the result, the eval is a dashboard ornament and you can delete it without guilt.

Your regression suite is institutional memory.

Stem used to re-read a file after editing it. Then read it again to be sure. Then again. I could add a line to the instructions, watch one run behave, and call it fixed.

That fixes an example. The product has learned nothing it can prove.

So the behavior became a case instead: edit-verify-once-no-reread-loop. Change one value in a config file, re-read it once, report done. Every routing and prompt change since July 1 has had to survive it. 85 runs, 85 passes.

Repeat that move and the suite becomes a record of everything the product learned the hard way: a file that does not exist getting summarized anyway, a README’s “production ready” taken on faith, a grader returning PASS because its engine was unreachable, a planted false premise answered instead of refuted.

Before launch, you test the failures you can imagine. After launch, users show you the ones you could not. Those failures are training data, and the thing they train is the organization that runs the product.

The answer is not always a better prompt.

Remembering a failure is only useful if it gets classified correctly the first time.

If every bad output is treated as a model problem, the improvement loop becomes: bad answer, modify prompt, larger prompt, pray.

Every failed case in Stem gets its transcript read before anything is touched, then sorted into one of the four bins in the grid below.

One question sorts the first two bins fast: show the model its own failing output next to the correct one. If it can see the mistake, you probably have leverage in instructions, context, decomposition, or verification. If it cannot, stop squeezing the prompt and change the system around it.

You are debugging a system. The model is only one of four suspects.

Then agents showed up and my tidy single-turn evals stopped being enough.

An agent is a whole system: model plus instructions plus context plus tools plus state plus environment plus orchestration plus retries plus permissions plus budget. Grade only the final chat message and you are grading the press release.

The failure mode that teaches this fastest: an agent can tell you it succeeded while the file it promised never landed on disk.

The metric list widens too. Past raw task success you start tracking tool-call accuracy, unauthorized actions, turns per task, tokens per task, retries, latency, cost, human interventions, recovery rate, critical failures. An agent that succeeds after eleven retries and four dollars of tokens has a different problem than one that fails cleanly.

The anchoring trap.

A decision-wired eval is only as good as who gets to say what counts as correct.

One trap I only saw after falling into it: anchoring eval cases to what the agent had already done quietly freezes yesterday’s mediocre behavior into a test suite that defends it. The fix is anchoring every case to an answer the user has ratified as correct, so the suite argues for the person the system serves.

The arithmetic that humbles everybody.

And then there is the arithmetic that humbles everybody. Chain five required steps that each succeed ninety percent of the time, with independent failures, and the whole run lands near fifty-nine percent. Ninety percent felt solid at every step, and the pipeline still hands you a coin flip and change.

That one line of math rewired my reliability instincts. A ninety percent step feels great in a spreadsheet. Multiply a few of them and you understand why multi-step agents need a different reliability posture than single-turn chat, and why recovery behavior matters as much as first-try accuracy.

None of that arithmetic matters if the lessons it produces evaporate after the fix ships.

Two companies can start with the same model and end with different products.

Company A handles problems as they appear. Users report bugs, engineers fix them, incidents close.

Company B reproduces each meaningful failure, classifies the cause, writes down the expected behavior, adds a case, changes the system, and re-runs everything the product already learned before going back to production.

Run both forward eighteen months. Company B holds a library of failure cases, sharper definitions of quality, routing heuristics for its own domain, and a record of when humans should step in. None of it shipped with the model.