Anonymized portfolio piece. All system names, internal identifiers, and usage details are removed. The mechanism and the measured throughput numbers are real.
The problem
Most AI systems that run a task through a language model serialize everything: one model, one growing context, one step after another until the task is done. That is slow when a task is naturally decomposable, and it is also lossy: as the context window fills with the transcript of everything the model has already done, quality degrades, especially on local models with smaller context budgets. A task that is really three independent research steps followed by one combine step gets run as four steps stacked in a single, ever-longer conversation, when three of those steps have no dependency on each other at all.
What I built
An orchestration layer for a local-first AI operating system that treats a task as a dependency graph instead of a list:
- Decompose. A planning pass splits an incoming task into the fewest self-contained sub-tasks, each tagged with a kind (code, research, synthesis, classification) and with which earlier sub-tasks, if any, it actually depends on.
- Route by specialty. Each sub-task's kind routes it to the model best suited for that kind of work, rather than running every sub-task through one generalist model.
- Fresh-context firewall. Every sub-task runs in its own clean context. Only a compact digest of the specific prior results a sub-task actually depends on crosses that boundary, never a full transcript. This is what prevents the quality decay that comes from one long, accumulating chain.
- Wave-based concurrent scheduling. A topological scheduler groups sub-tasks into dependency waves. A fully independent plan collapses into one wave and fans out concurrently against a batched inference endpoint. A linear chain expands into one sub-task per wave and pipelines. Mixed plans get both in the same run.
- Cost-aware fallback. A task the planner cannot usefully split (one sub-task or zero) falls straight through to a single plain run. Orchestration overhead is never paid on work that does not need it.
- Independent verify-and-resynthesize gate. After the sub-results are merged into one answer, a separate grading pass checks the merged result against a rubric. On failure, it re-synthesizes, carrying the rejected draft and every prior failure report forward so the retry revises instead of rediscovering the same problem.
The result
Measured 2.3x realized end-to-end throughput running real decomposable work through the full pipeline, against 2.86x-3.06x raw endpoint batching at 4 concurrent slots. The gap between the two numbers is expected and instructive: the raw batching ceiling is the hardware limit, and the realized figure is what the orchestration layer actually delivers once per-request overhead is included, which is the number that matters for a production system.
An illustrative example
This case study itself is a small proof by demonstration: the underlying system it describes takes on a writing task like this one by decomposing it into research, drafting, and review steps handled by different specialist passes, then verifying the merged result before it ships. Same pattern, applied to prose instead of code.
Why it transfers
The pattern here is not tied to any one model, task type, or hardware tier: decompose into a dependency graph, fan out what is truly independent, keep every sub-agent's context clean, and gate the merged result behind an independent check before it goes out the door. That is the difference between an AI system that feels slow and fragile on real work and one that scales its speed to the shape of the task, whether it is running on a single local GPU or a fleet of cloud endpoints.
That systems thinking is what I bring to consulting engagements: the same decomposition, verification, and orchestration discipline applied to document intelligence, estimating workflows, and business process automation.
Josiah Bujanda, AI & Construction Technology Consultant. Mechanical engineer with more than ten years in commercial lighting controls and Division 26 electrical, plus hands-on architecture of production AI and automation systems.