Work / Orchestration

A Dependency-Aware Orchestrator That Delivered 2.25× Realized Throughput

Every task ran through one model in one growing context, one step at a time, whether or not the steps depended on each other. A dependency-aware orchestrator decomposes each task into a graph, fans out the independent pieces, and grades the merged result before it counts as finished.

Anonymized. Client, employer, project, manufacturer, part, and pricing identifiers are removed; the mechanism and the numbers are real.

The user and their week

I run real decomposable work, research, drafting, and review, through Stem, my local-first AI operating system. Before this build, every task ran the default way most AI systems run a task: one model, one growing context, one step after another until the task finished. A task that was actually three independent research steps followed by one combine step ran as four steps stacked into a single, ever-longer conversation, even though three of those steps had no dependency on each other at all.

Why it mattered

That default cost me twice over. It was slow, because steps that could have run at once instead ran one after another. It was also lossy: as the context window filled with the transcript of everything the model had already done, quality degraded, especially on local models with smaller context budgets. What the source does establish is the scale of what serializing was leaving on the table: the orchestrator's realized 2.25x throughput sits against a raw batching ceiling of 2.86x-3.06x at 4 concurrent slots.

The bet, and what was rejected

I bet on treating a task as a dependency graph: decompose first, then route and fan out only what a planning pass actually proves is independent. That bet costs a planning pass on every task, and it only pays off on work that is genuinely decomposable, so the design has to know when to get out of its own way. A task the planner cannot usefully split, one sub-task or zero, falls straight through to a single plain run instead of paying orchestration overhead it would never earn back.

I designed the orchestrator around measured throughput. I could have chased maximum agent autonomy instead: let each sub-agent decide its own scope, decomposition, and pacing with no shared plan or grading step. I rejected that path because an autonomous agent working inside a growing context degrades the same way a serialized chain does, and because a system built around performance needed a plan I could measure against a number.

Acceptance criteria, set before build

What the record does show is the constraint that shaped the bet before any scheduling code existed: orchestration overhead has to be earned, never assumed. A task the planner cannot usefully decompose runs through as a single plain execution instead of paying for a planning pass it would never recoup. That bar, together with the requirement that every sub-task run in its own clean context so only a compact digest of the specific prior results it actually depends on crosses the boundary, is what the system had to hold up under before it counted as shipped.

What shipped

An orchestration layer that decomposes an incoming task into the fewest self-contained sub-tasks, each tagged with a kind (code, research, synthesis, classification) and its dependencies on earlier sub-tasks
Kind-based routing to the model best suited for that sub-task, with each sub-task run in its own clean context that receives only a compact digest of the specific prior results it depends on
A topological scheduler that groups sub-tasks into dependency waves: a fully independent plan collapses into one wave and fans out concurrently against a batched inference endpoint; a linear chain expands into one sub-task per wave and pipelines; mixed plans get both in the same run
A grading pass that checks the merged result against a rubric after sub-results are combined; on failure it re-synthesizes, carrying the rejected draft and every prior failure report forward so the retry revises instead of rediscovering the same problem

The grading pass is the deterministic gate: a merged result that fails the rubric never ships as-is, it goes back for a revision that carries its own failure history forward. The human gate sits above that. I am the tool's only user, and nothing the orchestrator produces reaches a downstream decision, a client-facing document, a committed line of code, without me reading the graded result first.

The old way ran every step in one growing conversation. The orchestrator groups the independent steps into one wave and only the combine step waits on them.

What failed, and what changed

The gap between the realized 2.25x and the 2.86x-3.06x raw batching ceiling is real per-request overhead: planning, digesting, and grading that the design has not closed. That gap is the system's own evaluation of itself; this is a single-user tool, so there was no beta population to report failures back. I would want to shrink that per-request overhead further before calling the orchestration layer optimized.

Result

Measured 2.25x realized end-to-end throughput running real decomposable work through the full pipeline, against a raw batching ceiling of 2.86x-3.06x at 4 concurrent slots. The realized figure counts per-request overhead that the raw batching ceiling does not. It is the number that describes what the orchestrator actually delivers when it runs production work.

Adoption, and what carries forward

I am the tool's user, and the orchestrator now runs my real decomposable work: research, drafting, and review passes handled by different specialist steps, merged, and graded before I use the result. This page itself went through that pipeline: the underlying system took on writing it by decomposing it into research, drafting, and review steps handled by different specialist passes, then verifying the merged result before it shipped.

What carries forward is the pattern: decompose into a dependency graph, fan out what a planning pass actually proves is independent, keep every sub-agent's context clean, and gate the merged result behind an independent check before it goes out the door. That is the same decomposition, verification, and orchestration discipline I would bring to document intelligence, estimating workflows, and business process automation engagements.