Cut LLM Token Spend 40% Without Losing Accuracy

By Lazar Milicevic · Published July 21, 2026 · 10 min read

Your prototype worked. Then the bill arrived. If you're running a Claude or GPT pipeline in production and watching your token spend climb faster than your MRR, you're not alone — and you don't need a bigger budget, you need a harness around your model calls. Writer's research team recently published work on their "AI HP" harness showing that systematic optimization of the orchestration layer — not the model itself — cut token spend by close to 40% while holding accuracy steady. The interesting part isn't the number. It's that every technique they used is available to a solo engineer with a weekend and a decent eval set.

The ROI paradox nobody warned you about

Enterprise AI has a well-known ugly middle: the demo screams, the production bill sobs. Throwing your most expensive model at every request works beautifully when you're serving 30 hand-picked test queries. It falls apart when you're processing 30,000 real ones a day, most of which don't need a frontier model at all.

The reflex fix is to swap to a cheaper model and pray. That usually costs you accuracy on the 15% of queries that actually needed the smart model. What Writer's paper formalizes — and what teams shipping real workloads have been doing quietly for over a year — is that the orchestration layer around the model is where the wins live. Prompt structure, routing, retrieval design, retry policy, and caching. Individually, each is a 5-15% saving. Stacked, they compound to roughly the 40% Writer reports.

The rest of this post walks through the specific levers, in the order I'd apply them to an existing pipeline. Start at the top, measure after each step, stop when the ROI stops.

Step 1: Instrument before you optimize

Before you touch a prompt, log every model call with: input tokens, output tokens, model name, latency, cost, request category, and whether the output was accepted downstream (retried, edited, or shipped). If you can't answer "which 10% of my requests eat 60% of my spend?" in a SQL query, you're guessing.

A minimal logging wrapper looks like this:

import time, json, sqlite3
from anthropic import Anthropic

client = Anthropic()
db = sqlite3.connect("llm_calls.db")

def call_and_log(model, messages, category, **kwargs):
    t0 = time.time()
    resp = client.messages.create(
        model=model, messages=messages, **kwargs
    )
    db.execute(
        "INSERT INTO calls VALUES (?,?,?,?,?,?,?)",
        (
            time.time(),
            category,
            model,
            resp.usage.input_tokens,
            resp.usage.output_tokens,
            time.time() - t0,
            json.dumps(messages)[:2000],
        ),
    )
    db.commit()
    return resp

Run this for a week. You will find two things almost every time: a handful of prompt templates dominate spend, and a meaningful fraction of your calls are duplicates or near-duplicates. That's your target list.

Step 2: Route by difficulty, not by habit

The single biggest lever is model routing. Most pipelines default to the strongest model for every step "to be safe." In practice, 50-80% of requests in a typical support, extraction, or summarization workflow are handled just as well by a mid-tier model at a fraction of the cost.

A routing pattern that works in production:

def route(request):
    # Cheap classifier decides which tier
    tier = classify_difficulty(request)  # small model, ~200 tokens
    if tier == "trivial":
        return "haiku-class"      # extraction, formatting, yes/no
    if tier == "standard":
        return "sonnet-class"     # most reasoning, drafting, RAG
    return "opus-class"           # multi-step reasoning, edge cases

resp = call_and_log(route(req), build_messages(req), category=req.type)

The classifier itself can be a small model, a regex heuristic, or a tiny fine-tuned model — pick whichever your team can maintain. What matters is that you have an eval set with human-labeled expected outputs for each tier, and you measure accuracy per tier after routing goes live. The failure mode is a classifier that quietly demotes 8% of hard queries to the cheap model. Your eval set catches that.

Even a conservative implementation of this — no cache, no small-model tier, just splitting mid vs. frontier — usually saves 20-30% by itself.

Step 3: Compress the prompt without losing the plot

Every token in your system prompt gets billed on every single call. In pipelines with high request volume, a bloated system prompt is a recurring tax. The audit is unglamorous but reliably wins 10-20%:

1. Delete instructions the model already follows. Frontier models don't need "You are a helpful assistant." They don't need "Think step by step" if you're not actually using CoT output. Remove and re-run your eval — most of it doesn't move the needle.
2. Move stable context to prompt caching. Anthropic and OpenAI both offer prompt caching. If your system prompt or reference docs don't change per call, mark them cacheable. On repeated calls within the cache window, you pay a small fraction for the cached portion.
3. Trim few-shot examples. Three well-chosen examples usually beat eight mediocre ones. Test the drop, don't assume it.
4. Ship structured output requests as JSON schemas, not English paragraphs. A 40-line "please return JSON with these fields" instruction compresses into a 6-line schema block.

Here's what caching looks like in the Anthropic SDK:

resp = client.messages.create(
    model="claude-sonnet-class",
    system=[
        {
            "type": "text",
            "text": LONG_STABLE_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": user_input}],
)

The first call writes the cache (slightly more expensive), every subsequent call within the cache window reads it (much cheaper). For any workflow where the system prompt is > 1,000 tokens and reused across requests, this is close to free money.

Step 4: Fix retrieval before you fix the model

If you're running RAG, your retrieval quality determines your token spend more than your model choice does. Bad retrieval forces you to stuff more chunks into context "just in case," which balloons input tokens on every call.

Concrete fixes, in order of impact:

• Chunk on semantic boundaries, not fixed sizes. A 512-token chunk that splits mid-sentence forces the model to burn tokens reconstructing meaning. Split on headings, paragraphs, or logical units.
• Rerank aggressively. Retrieve top-20 with your embedding model, rerank with a cross-encoder, keep top-3 to top-5. You'll cut context tokens per call by 60-75% versus dumping top-10 raw hits.
• Deduplicate retrieved chunks. In real corpora, near-duplicate chunks appear constantly. Hash-based or embedding-based dedup before context assembly is 20 lines of code.
• Add a "no relevant context" path. If retrieval scores are all below a threshold, return an "I don't have information on that" response without calling the LLM at all. Free win.

The mental model: every extra chunk in context is a rounding-up bet against your retrieval quality. Improve retrieval, shrink context, save on every subsequent call forever.

Step 5: Cache, dedupe, and batch the boring stuff

Three cheap patterns that most teams underuse:

Semantic caching. For any workload with repetitive queries — support triage, FAQ answering, product Q&A — hash the normalized input and check for a prior identical or semantically similar answer before hitting the model. A simple embedding + cosine similarity check with a 0.95 threshold catches a surprising share of production traffic.

def maybe_cached(query):
    q_emb = embed(query)
    hit = vector_store.search(q_emb, k=1, min_score=0.95)
    return hit[0].answer if hit else None

answer = maybe_cached(query) or call_llm(query)

Request batching. If you're extracting the same fields from 500 documents, don't make 500 calls. Batch 5-20 documents per call where the model can handle it, and parse structured output. You pay one system-prompt overhead instead of hundreds. Watch output token limits and error-recovery carefully — one malformed doc shouldn't kill a batch of 20.

Async retry with backoff instead of blind retry. A hard-coded retry(3) on every failure doubles or triples spend during model outages. Retry only on transient errors (rate limits, timeouts), and use exponential backoff. Non-transient errors (400s, content policy) should fail fast, not retry.

Step 6: Measure accuracy honestly, or none of this matters

Every optimization above can silently degrade quality. The only defense is a real evaluation harness — not vibes, not spot-checking, not "it looked fine when I tried it."

The minimum viable eval:

1. 50-200 labeled examples per major workflow. Real production inputs, expected outputs graded by a human.
2. A grader. For structured output, exact or fuzzy match. For freeform, either human review of a sample or LLM-as-judge with a well-defined rubric. LLM-as-judge is fine if you validate the judge against human grades on a subset first.
3. A regression gate. Every prompt change, model swap, or routing change runs against the eval set. If accuracy drops more than a defined threshold, the change doesn't ship.

def evaluate(pipeline_fn, dataset):
    results = []
    for ex in dataset:
        pred = pipeline_fn(ex.input)
        results.append({
            "id": ex.id,
            "correct": grade(pred, ex.expected),
            "tokens": pred.usage,
        })
    accuracy = sum(r["correct"] for r in results) / len(results)
    avg_tokens = sum(r["tokens"] for r in results) / len(results)
    return accuracy, avg_tokens

You now have the two numbers every optimization decision hinges on: accuracy and tokens per request. Track both across every change. Writer's harness reports its 40% number precisely because it measures both — a token-cut result without an accuracy result is meaningless.

Step 7: Cap output length and structure it

Output tokens cost several times more than input tokens on most providers. Two-thirds of production spend I've audited on chatty pipelines is output tokens.

• Set explicit max_tokens to a realistic ceiling per task. If your extraction outputs are always under 300 tokens, cap at 400, not 4,000.
• Force JSON or structured output rather than prose. A 12-field JSON object is dramatically shorter than a paragraph describing the same data, and easier to parse downstream.
• Ban preambles. "Certainly! Here is the answer you requested..." is billable. A one-line system instruction to skip preamble and return only the answer typically saves 20-40 tokens per call — trivial per call, meaningful at 100k calls a week.

Where you need reasoning, use it deliberately. If you don't need visible chain-of-thought in the output, don't ask for it — do the reasoning in a separate cheap call, or use the model's internal reasoning feature and only bill for the final answer where the provider supports that.

How BizFlowAI approaches this

Most of the client audits we run at BizFlowAI start the same way: a founder shows us a Claude or GPT bill that has quietly 4x'd since launch, and asks whether they need to switch models. Usually they don't. What they need is the harness — routing, caching, prompt hygiene, retrieval cleanup, and a real eval set that makes each change safe to ship. The 30-40% range Writer reports lines up with what we consistently pull out of production pipelines in a week or two of work, without touching the underlying model.

If your token bill has outgrown your comfort and you're not sure which lever to pull first, book a discovery call and we'll audit the pipeline: where the spend is concentrated, which optimizations apply, and what the realistic savings floor looks like given your accuracy constraints. No template deck — just the same numbers you'd see if you instrumented it yourself.

What to actually do Monday morning

If you take one thing from Writer's paper and this post, it's the order of operations. In practice:

1. Log every call for a week. You cannot optimize what you can't see.
2. Pick the top three prompt templates by spend. These are your only targets.
3. Add prompt caching to any stable system prompt. Fastest single win.
4. Build a 100-example eval set for the workflow that matters most.
5. Introduce a routing tier — even a two-tier split moves the needle.
6. Trim retrieval and cap outputs. Boring, reliable, compounding.
7. Rerun the eval after every change. No accuracy regression, no rollback drama.

None of this requires a research team or a new vendor. It requires treating your LLM pipeline the way you'd treat any other production system: instrumented, measured, and iterated against a defined objective. The 40% isn't magic. It's the sum of seven unglamorous decisions made in the right order.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.