A search over my own notes returned a confident wrong answer in June, and my first move was to go shopping for a better model. The document that would have answered the question was in the index the whole time. It came back ranked twelfth, and the assembler that builds the window took the top eight. The model I was about to replace had never seen the evidence I was blaming it for missing.
Finding that took an afternoon. Fixing it took ten minutes and one reranker setting. Swapping models would have cost a week and repaired nothing, because none of what failed was happening at the weights.
You never change the weights. You change what the model sees, and everything it sees was assembled by code you wrote, which puts most of the failures worth finding somewhere in the assembly.
Two ideas make that assembly legible: a context window read as a budget, and a retrieval chain read as six separate places to lose evidence. Both come from an AI product management playbook I worked through this summer, and what follows is those two ideas checked against a system that runs in my house. Two machines, a local model on a card in my office, a frontier model in the cloud for the hard thinking. Markdown files carry the state across session restarts, so nothing has to be re-explained.
Start with the budget, since it is the cheaper lesson and I paid full price for it anyway. Every token you put in the window costs money and latency, widens the surface a hallucination can start on, and competes with every other token for attention that already fades with distance. The ends of the window carry the most weight. Irrelevant context does not sit there harmlessly; it pushes the evidence you need toward the middle, where models walk past it.
That last part took me the longest to believe, because shoveling in everything that might help feels like insurance. There is a paper called Lost in the Middle that measures what the insurance actually buys, and I watched my own system reproduce the result for free before I acted on it.
Treat the window as a budget and everything competes for one line. System prompt, examples, retrieved documents, the conversation so far, tool results, whatever memory you inject. When I finally audited a month of my own sessions I expected retrieved documents to be the expensive item. The biggest line was me: about a million and a half tokens across one month, roughly nine thousand a session, spent re-reading files that were already sitting in the window. One file carried most of it, opened about twice a session because opening it felt like diligence.
One rule fixed it. Read a file once per session, then trust that it is still there, because the window remembers mid-session and I was the one forgetting. If you instrument a single thing before you tune anything else, instrument what you are paying to send, since a window nobody has measured is a bill nobody has read.
Measured that way, the job stops being collection and becomes selection. Freshness, ordering, trust, permissions: each one decides what earns a place in the window. More context stops being better context surprisingly early.
Retrieval is where selection gets industrial. RAG is sold as a checkbox on a pricing page. Underneath it is a full information system: parse the corpus, cut it into chunks, index them, rewrite the query, retrieve, filter, rerank, assemble the window, generate. Every stage can fail on its own and every stage fails quietly, which is why "the model hallucinated" is the diagnosis you reach for when you cannot see any of them.
Ask six questions in order before you ask anything about the model. Did the evidence exist? Was it indexed? Did the query find it? Was it ranked high enough to survive the cut? Did assembly keep it intact? Did the model use what it was handed?
Run against my bad answer, the evidence cleared the first three questions and died on the fourth, sitting at position twelve while the assembler took eight.
Evaluate retrieval separately from generation. A single score over the whole chain tells you that something is broken and never which thing. That gap is the difference between a ten-minute setting change and a week spent replacing the one component that was doing its job.
The second idea is about how much machinery any of this deserves, and it reduces to one rule: choose the minimum architecture and autonomy that solves the task. The ladder under that rule runs six rungs, from plain code with no model anywhere up to an agent running its own strategy. Nearly everything useful I run sits on the third, a model doing bounded work inside a pipeline that deterministic code drives.
Demos live at five, and so does every impressive clip on your feed, along with the unbounded loops, the actions nobody approved and the bills nobody predicted. Every level you climb should be a level you can defend on a whiteboard, capability by capability. This piece is retrieval, this piece is a tool, this piece stays deterministic, this piece stays human. When you cannot say which of those a component is, you have already lost the ability to trace a failure to it the way I traced that ranking cut.
Routing is the architecture pattern I lean on hardest, because my system handles two kinds of content and only one kind is allowed to leave the house. Some of it is public: code, docs, research. Some of it is personal: journals, finances, family. So a small classifier reads each turn before any model does. External work goes to the cloud model and personal content stays on the machine in my office. When the classifier cannot decide, it defaults to local.
That default is the design decision I am proudest of, because it fails in the direction that costs me money instead of the direction that costs somebody trust. Nobody notices a fallback until the day it fires, so choose its direction while you are calm and write down why you chose it. System design sounds like architecture diagrams, and a surprising amount of it turns out to be ethics with an if statement.
Both ideas sit inside one loop the playbook draws end to end: problem, value hypothesis, task definition, system design, evaluation, product experience, production, outcomes, failures, new evals, improvement, then around again. The stretch that changed how I work is the last third of that list. Every failure becomes a permanent eval case before anyone is allowed to fix it. My system has a script named `ratchet` for exactly that, because a lesson, once paid for, should never be unlearned.
What this does not show
Both numbers are one operator measuring one system. The million and a half tokens is a month of my own transcripts on my own machines. Your biggest line item will be something else, and the part that transfers is that it was measured at all. The window bar above is a single session drawn roughly to scale, and it is not a monthly average.
The retrieval trace is one incident, and one incident cannot tell you where retrieval usually dies. A reranker setting fixed mine. The six questions are the durable part here, and my answer to question four is local to my index.
Level three describes my constraints, which are small systems with one user and a real cost of being wrong. Read it as a ceiling and you will build less machinery than your problem needs.
You never change the weights. You change what the model sees.
So measure context the way you would measure UX: systematically, on a schedule, against real data. I used to think this job was mostly picking the smartest model. A few months of running one of these taught me it is mostly deciding what to leave out of the window. Then you build the plumbing that makes that decision the same way every time.
Edition 02 goes at model selection and evals, where the scar tissue is thicker.