Coding Agent Harness Cost: 3x Spend, No Accuracy Gain

By Lazar Milicevic · Published October 24, 2026 · 10 min read

You picked a coding agent, pointed it at a real repo, and the token bill is larger than the value of the pull requests it produced. Before you switch models or pay for a bigger plan, look at the layer people rarely measure: the harness wrapped around the model. A new study suggests that layer can drive cost up by as much as 3x without moving accuracy in any way you can distinguish from noise.

This post walks through what the study found, what it does not prove, why the cost gap exists mechanically, and how to measure your own harness so you stop paying for tokens that buy nothing.

What the study actually tested

The paper, "What Does a Harness Buy? Tokens, Mostly." by Yangze Liu and Zhongyi Han of Shandong University (arXiv 2610.04433), compared five models across three production harnesses: Claude Code, mini-SWE-agent, and OpenCode. The benchmark was SWE-bench Verified. Per the authors' repository, they also reran identical configurations to measure how much a score moves when nothing changes except the run itself.

That last part is the methodological point worth stealing. Most harness comparisons report a single pass rate per configuration and treat any gap as signal. Agents are stochastic, so some gap is always noise. The authors measured the noise floor first, then asked whether harness differences cleared it.

Scope matters, and secondary coverage blurs it:

• The large pool was 447 tasks, and only two models were run there, comparing Claude Code and mini-SWE-agent.
• A separate 45-task hard subset carried the five-model comparison across all three harnesses.
• Cost figures come from API-served models priced at vendor list prices, not from Claude or OpenAI API dollars.

If you read a headline saying "nearly 500 tasks, five models, three harnesses," that is a compression of two different experiments. Keep them separate when you reason about what the data supports.

The headline results, stated carefully

Here is what the sources support, with the qualifiers attached:

The 13%-versus-13% result is the one I would pin to a wall. If changing your entire harness shifts task outcomes by the same amount as pressing "run" a second time, then a lot of "we improved the agent" claims are measuring dice.

"Triples the cost" is a ceiling, not a rule

The headline framing oversimplifies, and you should not take it to your CFO as is. The study reports up to 3x across its whole design. On GLM-5.3-Flash, Claude Code cost more than three times as much as OpenCode: mean cost per trial over 45 tasks was 1.81 CNY for Claude Code, 0.97 CNY for mini-SWE-agent, and 0.56 CNY for OpenCode (paper). OpenCode was the cheapest and also the only harness that lost accuracy, and part of its low cost came from stopping after fewer model calls (VentureBeat).

The ranking is also model-dependent. On DeepSeek-V4-Flash, the lightest harness did not win on cost: mini-SWE-agent took 141 model calls per trial against Claude Code's 104, and cost slightly more, about 15.2 cents versus 14.6 cents per trial (VentureBeat).

Two caveats before you extrapolate dollars:

1. Costs are in CNY and cents at vendor list prices, with cache-hit pricing that varies by vendor. They do not translate to Claude or OpenAI API pricing.
2. The locally served Qwen models and the subscription-only Opus runs have no priced cost at all (alphaXiv summary).

So the defensible claim is: harness choice can swing cost by large multiples at equal accuracy, and which direction depends on the model. It is not "lean harness always wins."

Why a harness costs what it costs

The paper's cost model is simple enough to put in code (alphaXiv): every model call resends a fixed harness preamble, the conversation context grows across calls, and the number of calls multiplies the per-step input.

A back-of-envelope version, with round numbers that are illustrative only, not from the study:

def input_tokens(preamble, task_text, per_step_growth, calls):
    """Total input tokens billed across one agent trial.

    Each call resends the preamble plus the accumulated context.
    """
    total = 0
    context = task_text
    for _ in range(calls):
        total += preamble + context
        context += per_step_growth  # tool output + model reply appended
    return total

# Illustrative round numbers, NOT measurements
lean  = input_tokens(preamble=1_000,  task_text=2_000, per_step_growth=1_500, calls=40)
heavy = input_tokens(preamble=15_000, task_text=2_000, per_step_growth=1_500, calls=40)

print(lean, heavy, round(heavy / lean, 2))

Run that and you will see the preamble is a flat tax paid on every call, so it matters most when the agent takes many steps. It also shows why call count is a lever as strong as preamble size: more steps means more resends and a longer context each time. That is how a "lighter" harness can still cost more if it takes more turns, as the DeepSeek result shows.

For scale, the paper reconstructs the first-call preamble on GLM at roughly 16,581 tokens for Claude Code, 7,025 for OpenCode, and 829 for mini-SWE-agent (alphaXiv). Treat these as estimates: the authors reconstructed them rather than observing them directly.

Where the one real accuracy gap came from

The only harness effect that beat the noise was OpenCode's loss, and the paper traces about half of it to something mundane. On Qwen3.8-27B, roughly half of OpenCode's gap was associated with runs cut short at the 32,000-token output limit. Claude Code hit the same cap but automatically prompted the model to continue (VentureBeat).

This is the useful kind of harness feature: not a bigger system prompt, but a recovery behavior. When the model is truncated mid-task, the harness continues instead of scoring a failure. That is cheap, targeted, and plausibly accuracy-relevant. Compare it with a 16k-token preamble that the data say buys no measurable pass-rate lift.

The lesson for your own agent: the harness features worth paying for are the ones that fix specific, observable failure modes. Generic "more instructions, more tools, more context rules" has not shown up as an accuracy gain here.

Vendor harness on vendor model: no free lunch either

A natural objection: surely a vendor's own harness is tuned for its own model. The data do not support that. DeepSeek's harness on DeepSeek-V4-Flash passed 21 of 45 tasks, the same as Claude Code on that model (paper).

A separate study points the same direction on the cost side. An arXiv analysis of the Qwen Code CLI found that across 35 releases, token consumption nearly doubled while agent effectiveness showed no overall improvement (arXiv 2607.03691). Harness complexity tends to accumulate across versions, and unless someone measures cost against quality at each release, the bill grows quietly.

One honest counterpoint: another recent paper (UMass/Emory) reports that context management, planning, and tool choice have conditional effects that depend on the model and the context budget. That is a different study with a different conclusion shape, not a confirmation of the Shandong result. Read the two together as "harness effects are real in some conditions and invisible in others, and you can't assume which you're in." That is exactly why you measure.

What this does not tell you

Be disciplined about transfer:

• Domain. The evidence is SWE-bench Verified coding tasks (The Decoder also mentions a small Terminal-Bench cross-check). I found no evidence that these results carry over to non-coding business agents such as invoice handling, email triage, or lead follow-up. The mechanics (preamble resend, context growth, call count) are general, but whether extra harness machinery buys accuracy in your workflow is an empirical question for your workflow.
• Pricing. Do not convert the CNY and cent figures into dollar savings for a Claude- or OpenAI-priced setup.
• Causality of features. The study shows harnesses as bundles. It does not isolate which individual tool or rule is waste. Removing things one at a time is your job.

How to measure your own harness in an afternoon

You do not need a research lab. You need a fixed task set, a noise estimate, and a cost log. This is the protocol the paper's design implies, scaled down.

1. Build a task set you can score automatically. 20-40 real tasks from your own work, each with a pass/fail check (tests pass, expected file diff, schema-valid output). If you cannot score it automatically, you cannot compare harness variants reliably.

2. Measure the noise floor first. Run the identical configuration at least twice and compare task by task. The paper's recommendation is to publish a rerun of the same setup alongside any harness comparison so readers can see how much of a difference is noise (The Decoder).

def flip_rate(run_a: dict[str, bool], run_b: dict[str, bool]) -> float:
    """Share of tasks whose pass/fail outcome differs between two runs."""
    tasks = run_a.keys() & run_b.keys()
    flips = sum(run_a[t] != run_b[t] for t in tasks)
    return flips / len(tasks)

# noise = flip_rate(baseline_run_1, baseline_run_2)
# effect = flip_rate(baseline_run_1, variant_run_1)
# If effect is not clearly above noise, you have not shown the variant matters.

3. Log the three cost drivers per trial, not just the dollar total:

{
  "trial_id": "t-017",
  "harness_variant": "lean-v2",
  "preamble_tokens_est": 3200,
  "model_calls": 38,
  "input_tokens_total": 410000,
  "output_tokens_total": 21000,
  "passed": true
}

Preamble, call count, and total input tell you why a variant is expensive. A variant can get costlier through a fat preamble, through extra turns, or through context that balloons; the fix differs for each.

4. Ablate one thing at a time. Trim the system prompt. Drop the tools the agent rarely calls. Cap tool-output length before it enters context. Re-run each variant against the same task set and compare to your noise floor.

5. Add recovery behaviors before adding instructions. The truncation-continue example suggests the best return is in handling concrete failure modes: output-limit cutoffs, tool errors, stuck loops.

6. Re-test per model. Because cost rankings flip by model, a harness tuned on one model needs re-checking when you swap.

What to do with a harness that's already expensive

If your measurements show a heavy harness is not buying accuracy, work down this list in order of risk:

1. Cap tool-output size. Large tool results get resent on every subsequent call. Truncating or summarizing them attacks context growth directly.
2. Prune unused tools from the definition list. Tool schemas ride along in the preamble on every call.
3. Shorten the system prompt to the rules that correspond to failures you have actually seen.
4. Limit step count with a sensible ceiling and a graceful stop, since call count multiplies everything else.
5. Use prompt caching where your provider supports it, and check the current pricing page, since cache-hit pricing varies by vendor.

After each change, rerun the task set and compare against your noise floor. Keep a change only if accuracy stays inside the noise and cost drops.

Putting it into practice

The workflow above is deliberately plain: log preamble size, call count, and token totals per run from the start, set a noise baseline on a scored task set, and then remove harness weight step by step until accuracy actually moves. If you do that, you will see for yourself which parts of your harness earn their tokens. The study above is about coding agents; for business workflows, validate on your own tasks rather than assuming the result transfers.

If you already have an agent running and the token bill looks out of proportion to the work, start with the audit steps in this post. They are the cheapest way to find cost-versus-quality waste in your own setup.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.