The Co-Failure Ceiling Breaks Multi-Model Routing

By Lazar Milicevic · Published July 11, 2026 · 9 min read

You built a router. Coding questions go to a code specialist, math and logic go to a reasoning model, everything else falls back to a generalist. On paper, each model covers the others' blind spots. In production, your error rate is roughly double what your spreadsheet predicted, customers keep hitting the same three failure classes, and you can't figure out why the "diversification" isn't working.

The math you used assumes model failures are independent. They almost never are. There's a name for the wall you keep hitting: the co-failure ceiling.

What the co-failure ceiling actually is

The co-failure ceiling is the upper bound on how much reliability you can gain by routing across multiple models, given how correlated their failures are on your prompt distribution. If two models fail on overlapping subsets of prompts, stacking them doesn't multiply your reliability — it barely improves it.

Recent evaluations of frontier models across dozens of providers keep landing on the same result: models trained on similar public data, similar synthetic pipelines, and similar RLHF preferences fail on the same categories of prompts. Tokenizer edge cases, ambiguous multi-hop reasoning, obscure API versions, adversarial phrasing — the failures cluster. When teams assume independence and multiply out expected reliability, they routinely underestimate real-world failure rates by roughly 2x. That gap is not a rounding error. It is the difference between a 2% and a 4.5% agent failure rate, which is the difference between "acceptable" and "your support inbox is on fire."

The core mistake is treating "different model, different lab" as evidence of statistical independence. Different weights are not different failure modes.

The math your assumption depends on

Most multi-model routing designs implicitly use this formula:

P(system fails) = P(A fails) × P(B fails)

If model A fails 5% of the time and model B fails 5% of the time, you conclude the fallback pair fails 0.25% of the time. That is a 20x reliability improvement for the cost of one extra API call. It sounds free.

The correct formula includes correlation:

P(A ∩ B fails) = P(A fails) × P(B fails | A failed)

That conditional term is the entire problem. When A fails, what is the probability B fails on the same prompt? For unrelated random events, it equals P(B fails) and you get the clean multiplicative gain. For models trained on overlapping data with overlapping objectives, it can be 5x to 10x higher than P(B fails).

A quick simulation makes it concrete:

import random

def simulate(n_prompts, p_a, p_b, correlation):
    """correlation = P(B fails | A failed) / P(B fails)"""
    both_fail = 0
    for _ in range(n_prompts):
        a_fails = random.random() < p_a
        # If A failed, B's failure probability is amplified
        p_b_conditional = min(1.0, p_b * correlation) if a_fails else p_b
        b_fails = random.random() < p_b_conditional
        if a_fails and b_fails:
            both_fail += 1
    return both_fail / n_prompts

# Assumed (independence): ~0.25%
print(simulate(100_000, 0.05, 0.05, correlation=1.0))
# Realistic (correlated failures): ~1-2%
print(simulate(100_000, 0.05, 0.05, correlation=5.0))

That factor-of-5 correlation is not pessimistic. On benchmarks probing multi-step arithmetic, tokenizer-boundary bugs, or code involving library versions past the training cutoff, correlations between "different" frontier models can be higher.

Why frontier models fail on the same prompts

Independence assumes the models were built independently. They weren't. Four structural reasons drive the correlation:

Training data overlap. Every major frontier model was pretrained on a very similar slice of the open web, GitHub, arXiv, Wikipedia, and Common Crawl. If a fact is wrong or missing on the internet, most models get it wrong the same way. If a Python library's docs are sparse, most models hallucinate the same plausible-looking API.

Tokenizer families. BPE tokenizers trained on similar corpora produce similar rare-token behavior. Prompts that hit weird tokenization — non-Latin scripts mixed with code, unusual whitespace, long numeric strings — degrade multiple models simultaneously.

RLHF convergence. Human preference tuning pushes different models toward the same "helpful, harmless, hedged" response style. When a prompt triggers an over-refusal or a confidently wrong answer, it tends to trigger it across labs.

Shared blind spots in reasoning. Chain-of-thought failures on prompts requiring bookkeeping across many entities, or arithmetic beyond a certain digit count, appear across model families because the underlying transformer architecture handles those tasks in similar ways.

The practical consequence: your "coding specialist + generalist" stack fails on the same broken StackOverflow answer, the same deprecated library, and the same off-by-one loop.

How to measure your real co-failure rate

You cannot fix what you have not measured. Before adding a third model to your router, spend one afternoon on this:

1. Pull 500-2000 real prompts from your production logs. Not synthetic ones. Real ones, with the actual distribution of your traffic.
2. Run all of them through every model in your stack independently, with the same system prompt.
3. Grade each response — pass, fail, or ambiguous. For coding tasks, run the output. For factual tasks, check against a source of truth. For summarization, spot-check.
4. Build a co-failure matrix.

The matrix is the entire deliverable:

Two things fall out of this table immediately. First, your co-failure rate on the classes that hurt most (deprecated libraries, multi-step math, ambiguous asks) is often 3-7x worse than the independence math predicts. Second, that ratio is not uniform — it varies by prompt class. Which means your router should not treat all prompts the same.

Save this matrix. Re-run it every time you swap a model or change a system prompt. This is the closest thing to a unit test that frontier LLM stacks have.

Routing patterns that survive the ceiling

If routing to "another big frontier model" is barely a fallback, what does work? Four patterns that hold up in production:

Diversify the mechanism, not just the vendor. Instead of Model A → Model B, use Model A → deterministic tool → Model B. For math, route failed responses through a calculator or SymPy call. For code, run the output against a linter or a test suite before returning it. Deterministic tools have failure modes uncorrelated with LLMs, which is what you actually wanted from "diversification."

def answer_math(prompt):
    result = model_a(prompt)
    if not verify_with_calculator(result):
        # Don't just retry with model B on the same prompt.
        # Reframe the problem with the calculator's partial results.
        result = model_b(prompt, hint=calculator_partial(prompt))
    return result

Route by prompt class, not by model preference. Your co-failure matrix tells you which prompt classes have low correlation between models. Use different fallback chains for different classes. Deprecated-library questions might benefit from a retrieval step before either model sees the prompt. Ambiguous asks might benefit from a clarification loop, not a second model.

Verification, not majority vote. Three models agreeing on a wrong answer is common, because they share the same wrong training signal. Instead of voting, verify. Does the code compile? Does the SQL parse? Does the cited source exist? A single model plus a verifier beats three models plus a vote on most real workloads.

Cheap generalist first, specialist on failure signal. Route the cheap model first. If the response has a low-confidence signal (hedging, refusal, self-contradiction, tool-call failure), escalate. This keeps costs down and only pays the specialist tax when the generalist's confidence drops.

Fallback logic that actually reduces failure rate

Here is a config for a router that respects the co-failure ceiling. It uses a verification step, prompt-class-aware routing, and a deterministic tool where possible:

router:
  classify:
    model: fast_generalist
    classes: [code, math, factual, ambiguous, other]

  routes:
    code:
      primary: code_specialist
      verify:
        - lint
        - run_tests_if_present
      fallback:
        - reframe_with_error_context: code_specialist_v2
        - deterministic: search_official_docs

    math:
      primary: reasoning_model
      verify:
        - symbolic_check
      fallback:
        - tool: sympy_evaluate
        - reasoning_model_with_calculator_hint

    ambiguous:
      primary: clarification_loop  # ask user, don't guess
      fallback: generalist

    factual:
      primary: retrieval_then_generalist
      verify:
        - source_exists
        - source_supports_claim
      fallback:
        - refuse_with_reason

  logging:
    log_every_verify_failure: true
    log_every_fallback_trigger: true
    log_prompt_class: true

Two things about this config matter more than the specific tools. First, verification is the gate — the router does not trust the model output, it checks it. Second, every fallback trigger and verification failure is logged so you can rebuild the co-failure matrix from real traffic every week.

Common mistakes teams make once they discover this

Adding a third model. If two correlated models did not save you, three correlated models will not either. You are paying 3x the token cost for a small marginal reliability gain. Add a verifier or a tool, not a third generalist.

Trusting benchmark scores. Public benchmarks tell you average pass rate, not failure correlation. A model that scores 89% on your benchmark of choice may fail on the same 11% of prompts as your existing model. The score is compatible with zero marginal reliability from routing.

Setting fallback triggers on latency or errors only. The most dangerous failures are confident wrong answers, not timeouts. If your fallback only fires on HTTP 500, you are missing the failure class that matters most.

Not versioning the co-failure matrix. Providers update models silently. A quiet checkpoint change on either side can shift your correlations without touching your code. Re-run the eval on a schedule.

Overfitting the router to eval prompts. If you tune routing rules on the same 500 prompts you used to measure, you will get great numbers on those prompts and worse numbers in production. Hold out a fresh sample.

How BizFlowAI approaches this

We build multi-model agent stacks for small teams and treat the co-failure ceiling as the default assumption, not the exception. Every routing design starts with a real eval on a client's production prompt distribution — not synthetic benchmarks — so we can see which failure classes actually correlate across the models under consideration. From there, the fallback logic is built around verifiers and deterministic tools first, and additional model calls only where the correlation numbers justify the token spend.

We also monitor these stacks after they ship. Model providers change checkpoints, prompt distributions drift as clients' businesses evolve, and a stack that hit its target failure rate in month one can quietly degrade by month four. If you're routing across multiple models and want to know your real co-failure rate before customers find it for you, book a discovery call and we'll walk through your logs together.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.