Cost-efficient hierarchical agent orchestration

2026-09-30

When to delegate coding work from a scarce premium model to cheaper ones — and when not to. A multi-file benchmark where hierarchical orchestration matched solo pass rate, burned similar premium tokens, and was ~2.8× slower.

agents · orchestration · llm · benchmarks · workflows

I keep running into the same bad default: put one expensive model on everything and call it an agent. That is like asking a principal engineer to write every line, review every diff, and still own the product brief. It works on a tiny task. It wastes the scarce thing.

The scarce thing is not “tokens” in the abstract. On subscription tools the premium third-party models sit behind a tight quota; in-house or cheaper frontier models often have a much looser one. Delegated work can be near-free in quota terms even when it burns more total tokens. The useful optimization is weighted cost and attention, not the raw sum.

That does not mean “always delegate.” Sometimes the premium model should just do the work. This note is about when the split helps and when it does not.

The shape

A software team already has the pattern. Someone owns intent, decomposition, acceptance, and the consequential calls. Other people do variable-complexity implementation. Someone else checks the work against the criteria without inheriting the author’s story about why it is fine.

Agent stacks can mirror that:

Premium orchestrator contracts a cheap implementor, which contracts workers. Summaries return upward. An independent verifier checks artifacts and reports to the orchestrator.

I wrote that down as a working rule (trimmed here to the parts that matter for the argument):

Cost-Efficient Hierarchical Orchestration

Objective

Minimize expensive-model execution and context exposure while preserving strategic control, correctness, and user intent. Optimize weighted cost, not aggregate tokens: cheap delegated work may use more total tokens to protect the main orchestrator’s attention and context.

Roles and routing

The user-selected main model is the strategic orchestrator. It owns intent interpretation, decomposition, acceptance criteria, consequential decisions and trade-offs, cross-feature synthesis, and concise user communication.

Route by information containment and coordination needs: Small: main executes context-light work. Medium: delegate a coherent slice to one implementor. Large: assign independent features or subsystems to multiple implementor-orchestrators, which may use nested workers to isolate low-level context.

Independent verifiers check explicit acceptance criteria without inheriting the implementor’s reasoning; implementor self-checks do not replace required independent verification.

Delegation and reporting

Every delegated task must specify: objective and scope; owned artifacts; constraints and approved decisions; acceptance criteria; required return format.

Never forward raw tool output, tool-call metadata, logs, search traces, repeated reasoning, full transcripts, or successful intermediate steps.

Verification

Prefer cheap independent verification whenever correctness can be checked cheaply.

The “Small: main executes context-light work” clause already predicts the result below. Models are examples, not the thesis. In this run the premium seat was claude-opus-5-5-low; the hierarchical implementor was grok-4.7-medium-fast. The point is the role split, not a vendor scoreboard.

Benchmark (run 2)

I wanted a check that does not need a human judge: hard enough to write, cheap to score.

Three multi-file Python tasks, hidden pytest suites, n = 2 per task per arm (6 runs per arm):

taskwhat it asks for
10_todo_climulti-module CLI todo app (20 tests)
11_kv_serviceclient/server key-value service (12 tests)
12_refactor_reportrefactor a legacy report script into a package (8 tests)

Two conditions:

conditionwho does what
Soloone premium model implements end-to-end
Hierarchicalpremium orchestrator writes a delegation.json contract; cheap implementor writes the code; pytest is the pass/fail ground truth

Harness and raw artifacts live in agent-orchestration-bench (results/run2/). Token totals are input + output + cache_read + cache_write from the agent JSON usage fields. Cursor reports uncached input separately from cache.

Results

Both arms passed every run: 6/6.

Per-arm totals (results/run2/summary.json headline):

metricSoloHierarchical
Pass rate6/66/6
Wall-clock (sum)497.5 s1,380.7 s
Premium tokens460,525452,086
Cheap tokens01,626,201
Premium model calls66
Cheap model calls06

Hierarchical is about 2.8× slower on the wall (1,380.7 s vs 497.5 s). Premium token totals are roughly equal. Hierarchical spent 1.63M cheap tokens on coding.

Most of the premium total is cache traffic on fixed session setup context, not fresh reasoning:

arminputoutputcache_readcache_writetotal
Solo5214,015357,43389,025460,525
Hierarchical5218,517343,04190,476452,086

Per task (both replicates):

taskrepsolo wallhier wallsolo premiumhier premiumhier cheap
10_todo_cli191.1248.169,18267,281252,279
10_todo_cli293.6240.692,94567,751222,877
11_kv_service192.7287.596,41668,654256,430
11_kv_service290.2153.670,70768,935294,245
12_refactor_report189.7301.865,81289,676331,749
12_refactor_report240.2149.065,46389,789268,621

Solo versus hierarchical bar charts for premium tokens, cheap tokens, and wall-clock. Both arms passed 6/6.

On these minutes-scale tasks, delegation did not save premium quota. The premium model still paid the fixed per-session context cost either way. Hierarchical added orchestration latency and a large cheap-token bill. Correctness was the same.

That is the honest result. It is also the expected one for work that is still “small” in the routing sense: the premium model is not spending most of its tokens on long tool loops through a messy repo.

Rule of thumb

Delegate when the premium model would otherwise spend most of its tokens executing — reading many files, long tool loops, retries, parallel slices — and a written contract can carry the decisions that matter. Then cheap models absorb the bulk tokens; the premium seat stays on intent and acceptance; subscription quota on the scarce tier is what you protect.

Do not delegate when the task is minutes-scale and context-light enough that fixed setup / cache cost already dominates the premium bill. Orchestration then buys equal (or worse) premium spend, extra wall time, and a large cheap-token total for no quota win. Just let the main model do the work.

The thesis still holds for the large case. It does not claim a free lunch on every “three-module” feature.

What is still awkward

Delegation contracts cost tokens and time. Writing a good delegation.json is work. On a short task it can dominate — here, premium output alone was higher under hierarchical (18,517) than solo (14,015).

Handoffs lose context. The worker does not see the orchestrator’s private chain of thought. That is intentional (isolation), and it means the contract has to carry the decisions that matter. None of the six hierarchical runs needed a repair pass.

Quota and pricing move. Today’s “almost free” tier can become metered tomorrow. Cache may or may not count against the scarce quota the same way uncached input does — I only report what the CLI returned.

Limits of this note

Three multi-file tasks, n = 2 per task per arm (6 runs each), minutes-scale walls. Premium totals are dominated by cache_read / cache_write of fixed setup context, so “premium tokens” here are a poor proxy for “premium thinking.” Billing weights for cache are unknown. I am not claiming a universal speedup or a universal savings. I am claiming the team-shaped split is the right default when premium quota is the bottleneck and the work is large enough that the premium model would otherwise burn most tokens executing.

Run 1 footnote: four single-file toy tasks with no token telemetry; both arms 4/4, hierarchical wave wall worse — same directional lesson, weaker evidence.

What I would do next

Repeat with longer tasks where solo premium spend is dominated by tool-loop execution rather than session cache. That is where premium context protection should show up as a real quota gap, not a near-tie on cache-heavy totals.