Cutting RAG Inference Costs: Decide What Never Hits the LLM

By Lazar Milicevic · Published October 29, 2026 · 13 min read

Your RAG pipeline works in the demo, then the invoice arrives, or a compliance person asks why the system classified one specific document the way it did and nobody can reconstruct the answer. Both problems have the same root cause: every ambiguous case goes straight to the language model, and the model is trusted to sort it out. The fix is architectural, and it starts before retrieval, not after.

The all-LLM architecture and why it breaks

The default design is easy to explain: take an incoming item (an invoice, a support ticket, a contract clause, a claim), retrieve the most relevant context, stuff it into a prompt, and let the LLM decide. For a demo of 50 examples, it looks great.

It breaks in three places at once.

Cost scales with volume, not difficulty. Every item pays the full price of retrieval plus a large-context LLM call, including the ones a regex or a lookup table could have resolved. Most real-world streams are lopsided: a big share of items are boring and unambiguous, and you are paying LLM prices for them.

Decisions are hard to explain. When a model reads retrieved chunks and returns "category: B," the reasoning lives inside a probabilistic process. You can log the prompt and the output, but you cannot say "rule 14 matched, therefore B." For low-stakes work that's fine. For high-stakes classification (credit, hiring, compliance flags, anything a regulator might ask about), "the model decided" is a weak answer.

Failures are invisible. If the retriever pulls the wrong chunk and the LLM confidently classifies from it, nothing in the system flags a problem. You find out when a customer complains or an audit samples your decisions.

A VentureBeat guest article by Vineet Vijay, a lead AI/ML engineer, describes exactly this pattern and the alternative: in one system he worked on, sending only the genuinely ambiguous 10 to 15% of cases to the LLM cut inference cost by roughly 6x versus an all-LLM baseline (source).

A caveat before going further. That 6x is a single-system, author-reported result. The article doesn't give the baseline model, volumes, dollar figures, or methodology, and the 10 to 15% share is the author's estimate for his own data. Treat both as an illustration of the shape of the win, not a number to promise yourself. Your ratio depends on how ambiguous your data actually is, which you can measure (more on that below).

The core idea: a three-stage cascade

The pattern is a cascade where each stage is more expensive and more flexible than the last, and each stage only sees what the previous one couldn't resolve:

1. Deterministic rules. Exact matches, regexes, lookups, schema checks, thresholds, allow/deny lists. Cheap, instant, fully explainable.
2. Retrieval. For items the rules couldn't settle, fetch relevant reference material (policies, past decisions, product catalog, contract templates) and try to resolve against it, using similarity scores or a small model.
3. LLM, for the residue only. The genuinely ambiguous remainder gets the expensive call, with focused context.

The key mental shift: the LLM is not the classifier. It is the escalation path for cases your cheaper machinery honestly cannot decide. That reframing changes what you build first.

Here's a minimal skeleton in Python. It's deliberately boring, because the boring parts are where the savings are:

from dataclasses import dataclass
from typing import Optional

@dataclass
class Decision:
    label: Optional[str]
    stage: str          # "rules" | "retrieval" | "llm" | "human"
    reason: str         # human-readable, stored for audit
    confidence: float

def classify(item: dict) -> Decision:
    # Stage 1: deterministic rules
    d = apply_rules(item)
    if d:
        return d

    # Stage 2: retrieval-backed resolution
    d = resolve_with_retrieval(item)
    if d and d.confidence >= RETRIEVAL_THRESHOLD:
        return d

    # Stage 3: LLM, only for the residue
    return llm_classify(item, context=d.context if d else None)

Every Decision carries a stage and a reason. That's not decoration. It is what lets you answer "why was this decided this way" later, and it's what lets you measure how much traffic each stage absorbs.

Stage 1: what deterministic rules should absorb

Rules should catch anything where the correct answer is derivable without judgment. Typical candidates in SMB document pipelines:

• Exact and near-exact identifiers. Known vendor names, invoice number formats, customer IDs, SKU patterns.
• Structural checks. Missing required fields, wrong file type, empty body, duplicate of something already processed.
• Threshold logic. Amounts under or over a fixed limit, dates outside a window.
• Known senders and templates. Mail from a supplier whose documents always follow one layout.
• Explicit overrides. Items a human has already decided about (a corrected prior decision should never be re-litigated by a model).

Build this stage from your own data, not from guesses. A practical way to start:

# Pull a sample of recent items with their final (human-confirmed) labels
# and see what fraction a simple rule set would have gotten right.
python analyze_rule_coverage.py \
  --input labeled_sample.jsonl \
  --rules rules_v1.yaml \
  --report coverage.json

A rules file can be as simple as:

rules:
  - id: known_vendor_invoice
    when:
      sender_domain_in: [acme-supplies.com, northwind-parts.com]
      has_pattern: "INV-\\d{6}"
    then:
      label: accounts_payable
      reason: "Known vendor domain + invoice number pattern"

  - id: empty_or_unreadable
    when:
      text_length_lt: 20
    then:
      label: needs_manual_review
      reason: "Document text too short to classify"

The author of the VentureBeat piece reports that a large share of the items in his system were cleared by the deterministic stage, but he also notes that this depends on data quality. Messy, inconsistent inputs mean fewer rule hits. So the honest first step isn't "write rules," it's "measure how much of my labeled history rules could have handled." If the answer is 5%, the cascade still helps but the economics are different than if it's 50%.

A side benefit: rules are the cheapest part of the system to test. You can write unit tests with exact expected outputs, which you cannot do for an LLM.

Stage 2: retrieval as a filter, not just context

Most teams use retrieval only to feed the LLM. In a cascade, retrieval does a second job: it resolves or narrows cases before any generation happens.

Three ways to use it as a filter:

Nearest-precedent matching. If a new item is very close to a past item that a human already labeled, reuse that label (with the precedent ID stored as the reason). A high similarity threshold keeps this conservative.

Metadata filtering. Narrow the candidate policies or categories by metadata before searching. AWS describes Ring using metadata-driven filtering to serve region-specific content from a centralized knowledge system, which, according to AWS, cut the cost of scaling to each additional locale by 21% (source). The takeaway for a small team: filter by region, product line, customer tier, or document type before vector search, so you retrieve less, search less, and prompt less.

Confidence gating. Retrieval returns scores. If the top match is clearly dominant, resolve; if the top two are close or the best score is weak, that's your signal that the case is ambiguous and should escalate.

def resolve_with_retrieval(item):
    hits = vector_search(
        query=item["text"],
        filters={"doc_type": item["doc_type"], "region": item.get("region")},
        top_k=5,
    )
    if not hits:
        return None

    top, runner_up = hits[0], hits[1] if len(hits) > 1 else None
    margin = top.score - (runner_up.score if runner_up else 0.0)

    if top.score >= HIGH_SIM and margin >= MIN_MARGIN:
        return Decision(
            label=top.label,
            stage="retrieval",
            reason=f"Matched precedent {top.id} (score {top.score:.2f}, margin {margin:.2f})",
            confidence=top.score,
        )
    return Decision(None, "retrieval", "No dominant match", top.score)

The thresholds (HIGH_SIM, MIN_MARGIN) are the knobs that trade cost against risk. Set them from a held-out labeled sample, not intuition: run the stage on historical data, see where it's wrong, and place the threshold where its error rate is acceptable for your use case.

Stage 3: the LLM gets the residue, with a cheaper-first ladder

What reaches the LLM should be a small, genuinely hard slice. That's also where model choice matters, because you don't have to send the whole residue to your most expensive model.

Anthropic positions Claude Haiku 5.5 for high-volume work like classification and request routing (source). Its official pricing is $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens (above that, $0.50 input and $2.50 output). Claude Sonnet 5.5 is priced at $2 per million input tokens and $10 per million output tokens, with a cache read at $0.10 per million and a 1M-token context window (Haiku/Sonnet pricing, Sonnet 5.5 overview). Check the current pricing page before you budget, since list prices change.

Using those list prices, input tokens on Sonnet 5.5 cost 20x what they do on Haiku 5.5, and output tokens cost 20x as well. So a second-level cascade inside the LLM stage is worth considering: try the cheaper model first, escalate to the stronger one only when the cheaper one reports low confidence or disagrees with a check.

Here's a back-of-envelope example. These numbers are illustrative round figures, not measurements:

For the first row, input is 100,000 × 3,000 = 300M tokens, about $600 at $2 per million, and output is 20M tokens, about $200 at $10 per million, so roughly $800. The second row is 15% of that, roughly $120. Run the same 15,000 calls on Haiku 5.5 and input costs about $4.50 and output about $1.50, roughly $6, before any escalations. Your real numbers will differ with prompt size and retry rates, but the structure holds: call count and model tier are two independent multipliers, and the cascade attacks the first one while model tiering attacks the second.

Two more levers on the LLM stage itself:

• Prompt caching. Sonnet 5.5's cache read price is $0.10 per million tokens against $2 for regular input. If your prompt has a large stable prefix (policy text, instructions, few-shot examples), caching it changes the math significantly.
• Batch processing for anything non-urgent. Anthropic's Message Batches API charges 50% of standard API prices for asynchronous workloads; most batches finish in under an hour, with a 24-hour processing window (docs). Overnight document classification, backlog processing, and weekly audits are natural fits.

Don't assume you have to pick one. A reasonable production setup: real-time items go through the cascade with synchronous calls; the long tail of "can wait until tomorrow" goes through batch.

The audit trail is a feature of the cascade, not a bolt-on

The cost argument gets the attention, but the explainability argument is what makes this architecture survivable in front of a compliance officer.

In a cascade, every decision has a recorded path: which stage resolved it, which rule or precedent fired, what the confidence was, and, for LLM decisions, the exact prompt, retrieved context, model version, and output. A rule-resolved decision can be defended with "this rule, this input, this output." That's a much stronger position than "the model said so."

A minimal audit record:

{
  "item_id": "doc_20261029_00481",
  "timestamp": "2026-10-29T14:02:11Z",
  "stage": "retrieval",
  "label": "accounts_payable",
  "reason": "Matched precedent doc_20260811_00212 (score 0.94, margin 0.18)",
  "model": null,
  "rules_version": "rules_v14",
  "retrieval_index_version": "idx_2026_10_20",
  "input_hash": "sha256:9f2c...",
  "human_override": null
}

Version everything that can change an outcome: rule set, index snapshot, prompt template, model identifier. Without that, you can store a log and still be unable to reproduce a decision.

On regulation, be precise about scope. The EU AI Act's record-keeping rule, Article 12, requires high-risk AI systems to technically allow automatic recording of events (logs) over the system's lifetime (Article 12). The logging must enable recording of events relevant to identifying risk situations or substantial modifications, post-market monitoring, and monitoring of operation by deployers (Article 12(2)). Those obligations apply to systems that fall into the high-risk categories, such as certain employment, credit-scoring, education, and essential-services use cases. A generic document-sorting or support-triage pipeline is not automatically high-risk, so don't assume the Act covers every RAG system you build. If you think yours might qualify, that's a question for a lawyer, not a blog post.

Timing also moved recently. The Digital Omnibus on AI amends the AI Act and deferred some of the high-risk obligations (DLA Piper). According to law-firm trackers, obligations for stand-alone high-risk systems under Article 6(2) and Annex III now apply later than originally planned (Orrick), and high-risk systems under Article 6(1) and Annex I, meaning AI in products covered by EU harmonisation legislation, have a separate and later date (FPF). Some pre-Omnibus articles still quote the original dates, so check the date on any source you rely on and confirm the exact deadlines against the official text on EUR-Lex. Penalty tiers and log retention periods also vary by role and article, so read the primary text rather than a summary before you design to them.

Separately, Article 50 transparency duties, including machine-readable marking for generative AI systems, have their own timeline (Technology Org). That one may touch you even if you're nowhere near high-risk, if you ship generated content to EU users. Verify the current dates on the official source before you plan around them.

Even if no regulator ever asks, the audit trail pays for itself the first time a customer disputes a decision and you can resolve it in five minutes instead of five days.

Measuring it: the metrics that tell you if the cascade works

You can't tune what you don't count. Instrument these from day one:

• Stage distribution. Percent of items resolved at rules, retrieval, LLM, and human review. This is your cost lever. If the LLM share creeps up, something upstream degraded.
• Per-stage accuracy against a labeled sample. A cheap stage that's fast and wrong is worse than an expensive one. Keep a rolling gold set and re-score weekly.
• Escalation reasons. Why did items fall through? "No rule matched" and "retrieval margin too small" point to different fixes.
• Cost per resolved item, by stage. Compute it from actual token usage, not list-price assumptions.
• Override rate. How often humans change a decision, broken down by stage. A high override rate at the rules stage means a rule is wrong. At the LLM stage it means the residue is hard, which is expected.

A simple weekly report from your decision log:

SELECT
  stage,
  COUNT(*) AS items,
  ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER (), 1) AS pct,
  ROUND(AVG(CASE WHEN human_override IS NOT NULL THEN 1.0 ELSE 0 END), 3) AS override_rate
FROM decisions
WHERE timestamp >= NOW() - INTERVAL '7 days'
GROUP BY stage
ORDER BY items DESC;

Run this before you build anything. If you already have labeled history, you can simulate the cascade offline against it and see the projected stage split and error rate without touching production. That simulation is the honest way to estimate your own version of the "6x" before committing to it.

Where cascades go wrong

A few failure modes worth designing against:

Rules rot. A rule written in March encodes assumptions about vendor formats that changed in September. Give every rule an owner, a creation date, and a hit-rate metric; review rules whose hit rate drops or whose override rate rises.

Overconfident retrieval. Similarity scores aren't calibrated probabilities. Two documents can be semantically near and categorically different (a contract and its termination notice). Always test thresholds against labeled errors, and keep a margin check, not just a top score.

The LLM stage becomes a dumping ground. If every rule/retrieval failure silently flows to the LLM, you've rebuilt the all-LLM system with extra steps. Watch the LLM share; treat growth as a bug to investigate, not a cost to absorb.

No human lane. Some items should never be auto-decided, whatever the stage says. Define explicit conditions (high value, low confidence, flagged entity) that route to a person, and log that routing like any other decision.

Optimizing cost before correctness. Get accuracy right on the all-LLM baseline first so you have a reference. Then check that the cascade matches it on your labeled set before you celebrate savings.

Putting it into practice

The document pipelines I build for small teams follow this shape: deterministic checks first, retrieval as a resolver and filter, and an LLM only for what's left, with a decision record for every item. The practical reason is the same as the argument above. Filtering before the model keeps costs tied to how hard your data is rather than how much of it there is, and it leaves a trail you can defend.

If you're already running a RAG pipeline and the bill or the audit story worries you, the useful first step is to measure how your traffic would split across stages using your own labeled history, then redesign around that split. I can't promise a specific multiple in advance, since the 6x in the source article was one system and your ratio depends on your data. If you'd like to talk through your own case, you can contact me.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.