What the Model Sees

I have spent the past few months building an agent system that lives in my house. Two machines, a local model on a graphics card in my office, a frontier model in the cloud for the hard thinking, and a pile of markdown files that survive session restarts so nothing has to be re-explained.

While building it I have been working through an AI product management playbook, and two chapters landed harder than the rest because I had already paid for their lessons in tokens. Context engineering and system design. This is what they teach, checked against what actually happened on my machines.

Start with the sentence the context chapter is built on. You never change the weights. You change what the model sees.

Everything else in the chapter is the fine print on that sentence.

Every token costs money.

Every token costs latency.

A longer window grows the surface area for hallucination.

Attention fades with distance.

The start and the end of the window carry the most weight.

Irrelevant context actively hurts.

That last line took me the longest to believe. Shoveling in everything that might help feels safe. It is the opposite of safe. There is a paper called Lost in the Middle showing models walk right past evidence buried halfway through a long window, and I have watched my own system reproduce the result for free.

So the window is a budget, and everything competes for it: the system prompt, the examples, the retrieved documents, the conversation so far, tool results, whatever memory you inject. When I finally audited my own spend, the biggest line item turned out to be me re-reading files that were already sitting in the window, about a million and a half wasted tokens across one month.

One rule fixed it. Read a file once per session, then trust that it is still there. The window remembers mid-session. I was the one forgetting.

The playbook frames all of this as a selection problem. Which information belongs where. Freshness, ordering, trust, permissions. More context stops being better context surprisingly early, and the actual job becomes deciding what to leave out.

Retrieval is where that job gets industrial. RAG gets sold as a checkbox on a pricing page, and underneath it is a full information system: parse the corpus, cut it into chunks, index them, rewrite the query, retrieve, filter, rerank, assemble the window, generate. Every stage can fail on its own, quietly.

Which means when a retrieval-backed answer comes out wrong, "the model hallucinated" is usually the lazy diagnosis. The chapter hands you six questions to ask in order instead. Did the evidence exist? Was it indexed? Did the query find it? Was it ranked high enough? Did assembly keep it intact? Did the model actually use it?

I ran that checklist on a bad answer from my own vault search, and the evidence had survived four stages before dying at ranking, sitting at position twelve when the assembler only took the top eight.

The fix took ten minutes. I had been about to swap models, which would have cost a week and repaired nothing. Evaluate retrieval separately from generation. That one habit pays for the whole chapter.

The system design chapter is about how much machinery any of this deserves, and it opens with the rule I now say out loud in every design conversation: choose the minimum architecture and autonomy that solves the task.

It gives you a ladder to stand on while you argue.

Level zero, plain code, no AI anywhere.

Level one, a single model call.

Level two, a call augmented with retrieval or tools.

Level three, AI steps inside a workflow that code controls.

Level four, an agent making bounded dynamic decisions.

Level five, an autonomous agent running its own strategy.

Nearly everything useful I run sits at level three, a model doing bounded work inside a pipeline that ordinary deterministic code drives.

The pull toward five is real. Demos live up there, and so does every impressive clip on your feed. The failure modes live up there too: unbounded loops, actions nobody approved, bills nobody predicted. Every level you climb should be a level you can defend on a whiteboard, capability by capability. This piece is retrieval, this piece is a tool, this piece stays deterministic, this piece stays human.

Of the architecture patterns the chapter catalogs, the one I lean on hardest is routing. My system handles two kinds of content. Some of it is public, code and docs and research. Some of it is personal, journals and finances and family. So a small classifier reads each turn before any model does, sends external work to the cloud model, keeps personal content on the machine in my office, and when it cannot decide, it defaults to local.

That default is the design decision I am proudest of, because it fails in the safe direction. Nobody notices a fallback until the day it fires, and by then you have already chosen who gets to see what. System design sounds like architecture diagrams. A lot of it turns out to be ethics with an if statement.

The playbook wraps both chapters in one loop: problem, value hypothesis, task definition, system design, evaluation, product experience, production, outcomes, failures, new evals, improvement, then around again. The stretch that changed how I work is the end of it. Every failure becomes a permanent eval case before anyone is allowed to fix it. My system has a script literally named ratchet, because the rule is that a lesson, once paid for, never gets unlearned. The net only tightens.

The PM implication the context chapter closes on is blunt, and it holds up. Two products built on the identical model diverge entirely on context quality.

The weights are rented.

Everyone rents the same weights.

What the model sees is yours.

You never change the weights. You change what the model sees.

So measure context the way you would measure UX. Systematically, on a schedule, with real data. I used to think this job was mostly picking the smartest model. A few months of running one of these systems taught me it is mostly deciding what to leave out of the window, and building the plumbing that makes that decision the same way every time.

Next up in the playbook: model selection and evals. I have some scar tissue there too.