Work / Local-first AI

A Local-First AI System With Zero Confirmed Leaks Across 19,124 Production Events · Stem

A daily AI practice that mixed sensitive documents with ordinary questions defaulted every request to the same cloud model, or slowed down to sort them by hand. A fail-to-local router and a deterministic redaction gate now sit as the one code path a request has to clear before it can reach a cloud model, checked against a 96/96 self-test gate.

The product page shows Stem running.

Identifying details anonymized. Results and measurements are unchanged.

The user and their week

The user is one person, running sensitive, document-heavy work daily: notes, transcripts, documents, reference libraries, and a growing trail of prior decisions, scattered, and some of it not safe to hand to a cloud model. Every day mixed plain general-knowledge questions with proprietary and confidential material of his own.

Before a router existed, every request faced the same expensive choice: run it through the easy default, a frontier cloud model, and accept the risk that one careless prompt leaks sensitive material to a third-party API, or stop and manually decide, case by case, what was safe to send off the machine. That manual sorting cost real hours on its own: re-explaining context every session, re-deciding questions that were already settled. Across the life of the system, one demanding daily user has put 30,230 stored turns through it so far.

Why it mattered

Treating every request the same way had two costly failure modes: overpay for cloud inference on questions a cheap local model handles fine, or build a system where one careless prompt leaks sensitive data to a third party. The system had to solve five connected problems at once: keep sensitive information local by default, search and reason across a large fragmented knowledge base, preserve context across sessions without refilling every prompt with history, execute multi-step tasks reliably across different models and machines, and improve over time so the same failures stop repeating. The requirement was an operating environment around the model, memory, control, evidence, and learning, with private data confined to controlled infrastructure.

The bet, and what was rejected

The product bet, in its plainest form: sensitive work should remain private without giving up access to frontier AI. Two easier paths were rejected on purpose. A single cloud subscription would have been simpler to run and would have given every task access to the strongest available reasoning by default; that convenience was given up. A policy document telling the system not to send sensitive data to the cloud was rejected too, on the grounds that a policy gets read once and violated under deadline pressure.

What replaced both: a tiered architecture, sensitive inference kept on controlled local infrastructure, cloud models used only for tasks carrying no private content or under tightly controlled read-only review, with the boundary enforced through code paths, exercised by dedicated privacy and sovereignty tests.

The quality bar

What the sources do show is the standard the system was actually held to. On routing: any low classifier confidence, any classifier error, any attachment present, or any ambiguity at all defaults to local, never to cloud, and confidence has to clear a deliberately high bar before a request is ever allowed out. On shipping: nothing ships unless the full self-test suite passes; the gate stands at 96/96 today, and the suite only grows stricter, never looser. On the newest safeguard, a detector for degenerating output: it had to catch every real repetition loop without flagging a single healthy turn, checked against the full stored history, 30,230 turns, three loops caught, zero healthy turns flagged.

What shipped

A router that classifies every incoming request, before it goes anywhere, as general-knowledge or sensitive/proprietary; the classifier runs locally, on the request itself, so nothing crosses the boundary just to decide whether it should.
A deterministic redaction gate as the single code path in the system allowed to emit anything externally: every field in every outbound payload passes an allowlist, anything unrecognized gets tagged and stripped, and the gate runs a self-test on its own output before emitting, refusing to emit at all if any substantive source content survived.
A human approval gate, named plainly: the system drafts, builds, and reconciles; a person approves anything that ships, spends, or leaves the machine. Nothing that touches money, external delivery, or an outbound request passes without that approval.

Every request in this system passes through one gate before it can leave the machine. If the classifier is unsure, the request stays local. Nothing skips the line.

What it takes to make an agent dependable

The router is one gate. Under it is the runtime that every turn, tool call, and long task runs on, and dependability is a property of that runtime, not of any one model. Seven concerns, each described as the mechanism exists in the code, at the width it actually has. A runtime page that rounds up is the failure it claims to prevent.

The spine every turn runs on. Controls and human authority attach at execution, where a call can spend, write, or leave the machine. The suite feeds the next release.

01Tool execution

Declared schemas · dispatch validation · model-call retries · host breaker · timeouts · read dedup

Every tool carries a declared schema, and a call missing a required argument errors at dispatch instead of reaching the tool. Retries are narrow on purpose: a model call gets three attempts inside a 600-second wall-clock budget, behind a per-host breaker that stands a dead endpoint down for two minutes. A tool call is never retried on the runtime's behalf; a failed write comes back as an error the model has to reason about, never a silent second attempt, which is what keeps one write one write. Shell calls hold a 120-second timeout. Repeat read-only calls inside a turn are served from a dedup cache; anything that writes or runs a shell always executes.

02State

Append-only transcripts · progress sidecar · all-or-nothing multi-file edits · grants that die with the session

Every session is an append-only transcript on disk: each tool call with its inputs and outputs, and the token usage of every model turn. A long task rewrites a progress sidecar at every step (the task, the engine, the todo list, the files touched), so a stopped run shows where it got to. That is a progress record, not a resumable state machine; a person reads it and decides whether to continue. A multi-file edit applies all of its edits or none of them. A session's approval grants live in memory and are never written to disk.

03Context

Per-turn assembly · token budget from measured usage · lazy attachments · local retrieval · recall gate

Context is assembled per turn against a budget. The estimate starts as characters per token and is refined by the real usage the last ten turns reported, so the window is trimmed before the model sees it rather than after it overflows. An attachment over 8,000 characters is outlined and paged on demand instead of pasted whole. Retrieval runs against a local corpus index, and chat memory enters through a recall gate that keeps the turn on a local model, so remembering never becomes a leak path.

04Control

Permission gate · empty destructive grant · session always-allow · step limits · cancel · approval pause

Only tools that write, run a shell, or destroy hit the approval gate; read-only tools never do. The destructive set starts with an empty grant, and an always-allow lasts one session and dies with it. A run stops at 20 steps by default and says so in the log rather than trailing off. Any turn can be cancelled from the interface, which trips a per-turn signal the runtime watches, and a long task pauses on an approval request until a person answers it.

05Evaluation

Self-test gate · behavioral cases · tool-use checks · step-count bounds · six failure classes

Nothing ships past the 96/96 self-test gate. Evaluation cases check the run, not just the answer: which tools were called, how many steps it took, what the log recorded, and string or pattern checks on the output. Those are deterministic rule checks rather than a learned judge, on purpose; a rule cannot drift. Transcript mining sorts failures into six classes (a missed edit, a malformed tool call, a path or schema miss, a transient error, a denied approval, a tool error), so a recurring failure has a name before it has a fix.

06Learning

Production traces → failure classes → regression cases → next release · append-only ratchet · nightly critic

The loop is traces to failures to cases to releases. A production miss is reproduced from the transcript, classified, and appended to the regression suite by a ratchet that only adds; the suite can tighten and never loosen. A nightly critic reads the day's transcripts for the same classes. The two incidents below went in as cases before their fixes shipped.

07Economics

Token usage per turn · context fill · model choice by measured score · cloud only for external content · cheap models for recon

Every model turn records its token usage and how full the context was. Local engines are ranked by their measured score on the eval suite, and the winner takes the next turn; cloud models are reserved for content classified external, so the expensive tier is spent only on work that can leave the machine. Recon fans out to parallel read-only scouts on cheap local models, and only findings return to the main context. Latency is measured where a person feels it: 115 milliseconds to first audio on the voice loop, warm.

What failed, and what changed

Two production incidents traced to the same root cause: infrastructure edges the system had never been tested against under real conditions. First, a nightly evaluation process kept generating reports that a silent downstream failure left unread; review freshness was added after the fact, once the gap surfaced. Second, a shell-specific argument-handling issue stripped quotation marks during model handoff; the fix moved prompt delivery from command arguments to standard input.

Each incident became a permanent regression test before its fix shipped, per the standing rule that a failure has to convert into a test before it counts as closed.

Result

The audited number is the router's: zero confirmed leaks, validated against the full production transcript history, 1,821 transcripts and 19,124 individual events. Escalation runs through exactly one code path with no parallel route around it, so there is nothing else to audit; the gate either holds for every request or it does not run at all.

The self-test gate that has to pass before anything ships stands at 96/96. The degeneration detector, checked against the same discipline, ran against 30,230 stored turns and caught three real loops while flagging zero healthy turns.

The honest cost of this result: erring conservative on every borderline case means real cloud-tier speed and quality get left on the table any time the classifier is not confident enough to escalate. That is the accepted price of a boundary with no bypass to audit.

Adoption, and what carries forward

The adoption evidence on this page is one person: the builder, running the system daily against work he could not risk handing to a cloud model outright, across the same 30,230 stored turns cited above.

What carries to the next engagement is the pattern: classify before routing, using a model cheap and local enough that classification itself never becomes a new leak surface; default to the safe side on any doubt; enforce the boundary in exactly one code path, with a fail-closed allowlist and a self-check, so the guarantee never depends on a future contributor remembering a rule. Engineering and construction organizations carry the same fragmented-knowledge, unclear-boundary problem this system was built to solve; the transferable version connects fragmented records with provenance intact, defines the data and approval boundary in code, and turns incidents into regression tests and standards, cutting repeat costs, while the people with technical and commercial authority keep the approval.