DeepSeek V4 Flash: Leaderboard Score vs. Real Agent Work

By Lazar Milicevic · Published October 15, 2026 · 10 min read

You picked a cheap model because it topped the leaderboards, wired it into an agent, and now it's quietly failing half your multi-step jobs. That gap between benchmark rank and finished work is the real cost of choosing a model, and it's measurable. Here's what one independent test showed, what happened to pricing since, and how to run your own check before you commit.

What Composio actually tested

The short version: Composio ran DeepSeek V4 Flash through multiple agent harnesses on deliberately hard, multi-step tasks against live tools, and it completed only a little over half of them. That is a long way from the leaderboard reputation the model earned.

The setup matters more than the headline number, so here are the details as reported:

• Scope: VentureBeat reports Composio ran V4 Flash through eight agent harnesses, including Claude Code, Codex and OpenCode, on 30 multi-step tasks spanning live tools like Gmail, GitHub, Slack and Google Sheets.
• Volume: Per Composio's write-up, across 240 runs the harnesses completed 129 successfully, an overall pass rate of roughly half.
• Consistency: Only six of the 30 workflows were completed successfully by every harness.
• Traps: Each harness got its own isolated fixture containing the real target data plus decoys, such as similarly named files and nearly identical keys, designed to catch overly broad actions.

That last point is the interesting design choice. Most public benchmarks reward getting to the right answer. This one also punished agents for touching the wrong thing. If you've ever had an agent update the wrong spreadsheet row because two names were close, you know why that matters more for a small business than a leaderboard delta.

A few caveats before you quote the pass rate anywhere:

• It applies to V4 Flash as tested, not to the model that replaced it (more on that below).
• The table below shows only six of the eight harnesses, and I haven't verified the remaining two from the sources, so the rows shown won't account for the full total. Treat the overall result as "roughly half," not a precise constant, and check Composio's page for the exact numbers.
• Whether every harness was configured like-for-like (reasoning effort, API provider routing) is something I couldn't confirm from Composio's own page, so I'm not building any argument on it. It's a reason to read the primary write-up yourself rather than rely on my summary.

Same model, very different results by harness

The most useful finding isn't the average. It's the spread. VentureBeat's takeaway is that orchestration, not raw model capability, may decide success: the same model produced substantially different results depending on harness, tool configuration, caching behavior, retries and provider stack.

Here are the pass rates Composio reported for six of the harnesses it broke out (two others were tested but are not shown here):

Same model, a wide gap between the best and worst harness. If you evaluate a model in one tool and deploy it in another, you haven't evaluated the thing you shipped.

Cost and speed varied too. Composio's cost per successful task was $0.195 for Claude Code, $0.081 for Codex and $0.045 for DeepAgents, with median completion times of 122.7s, 245.0s and 187.1s respectively. Notice what that does to the "cheapest model" intuition: if the per-token price is the same across those harnesses, the cost of a successful task still differs by more than 4x. Harnesses differ in how many tokens they burn on planning, retries and tool-call overhead, and in how much of their input is served from cache, and that, not the sticker price per million tokens, is what lands on your invoice.

One honest limitation: this is one vendor's test, on one model snapshot, at one point in time. It's strong evidence that harness choice matters. It is not a ranking you should hard-code into your stack.

Why leaderboard scores overstate agent performance

A leaderboard measures a model on a fixed, usually single-turn or tightly scaffolded task. A real agent run is a chain: read an email, find the right record, call an API, interpret the response, handle an error, decide what to do next. Reliability compounds. If each step succeeds 90% of the time, a ten-step workflow succeeds about 35% of the time (0.9^10 ≈ 0.35). That's an illustrative calculation with round numbers, not a measurement of any specific model, but it shows why a model that looks excellent per step can look mediocre per workflow.

Three things leaderboards typically don't capture:

1. Decoys and ambiguity. Real data has near-duplicates. Composio's fixtures deliberately included them. A model that's "smart" can still pick the wrong file when two are named almost identically.
2. Tool-call mechanics. Malformed arguments, wrong parameter types, and mishandled error responses fail a task even when the reasoning was correct.
3. Harness behavior. Context management, caching, retry policy and system prompts all sit between the model and the outcome. Change them and you change the pass rate.

None of this means leaderboards are useless. They're a decent way to build a shortlist. They're a poor way to make the final call.

The pricing story changed twice, so check the current page

The "prices surge" part of the story needs careful handling because it has moved since the original coverage.

What's well sourced:

• DeepSeek's V4-Flash and V4-Pro price increase and new peak/off-peak billing went live at 16:00 UTC on August 16, 2026, per Yahoo Finance.
• Peak hours were defined as 01:00–04:00 and 06:00–10:00 UTC, with off-peak at half the peak rate.
• Before the change, V4-Flash was a flat $0.28 per million output tokens and $0.14 per million cache-miss input tokens.
• Per VentureBeat, the August V4 Flash rates were $0.22 input / $0.66 output per million tokens off-peak and $0.44 input / $1.32 output at peak, which it described as a 57% to 371% increase.

A correction worth making because it gets garbled: the "1,100%" figure that circulated refers to cache-hit prices (VentureBeat gives 52% to 1,100% for cache hits), not the headline Flash input or output rates. If you're running agents with long, repeated context, cache pricing matters a lot, so don't ignore it, but don't quote 1,100% as "Flash got 12x more expensive."

Then the model itself changed. A third-party tracker (benchlm.ai) reports that on September 10, 2026 DeepSeek launched V4.1 Flash as deepseek-flash and retired V4 Flash, routing its ID to V4.1 Flash, at $0.30 input / $1.20 output per million tokens at peak and $0.15 / $0.60 off-peak. That's a third-party tracker, not DeepSeek's own page, so verify it. The same tracker says V4 Pro pricing has been unchanged since August 16 ($1.32 input / $3.96 output at peak, $0.66 / $1.98 off-peak).

The practical implication: if you integrated "V4 Flash" in the summer, the endpoint you call today may be a different model with different behavior and different prices. Pin versions where the API allows it, and re-test when a provider reroutes an ID.

For current numbers, always check DeepSeek's own pricing page rather than any article, including this one.

What the Composio result does and doesn't say about V4.1 Flash

This is where I'd push back on both the pessimists and the optimists.

The Composio result is for V4 Flash as tested in August. V4.1 Flash launched in September. As far as I know, there is no independent agent-harness testing of V4.1 Flash comparable to Composio's, and any claims the vendor makes about the newer model's agentic ability come from its own chosen benchmarks. Check DeepSeek's official release notes for what it actually states. So:

• Don't assume V4.1 Flash lands at the same pass rate. Nobody has shown that.
• Don't assume it's fixed either. Vendor benchmarks are exactly the thing this whole post is cautioning about.
• Do treat the Composio result as a methodology to copy, not a verdict to cite.

Also useful context on the underlying model: per its Hugging Face model card, V4 Flash is a Mixture-of-Experts model with 284B total and 13B activated parameters and a one-million-token context length. A long context window is a capability, not a guarantee. Agents that stuff a million tokens of context often get worse at finding the one relevant record, which is the decoy problem again.

How to run your own agent-task benchmark in an afternoon

You don't need Composio's budget. You need 15–30 representative tasks, a safe sandbox, and a scoring rule. Here's a minimal pattern.

Step 1: Collect real tasks. Pull them from your own history: "find the invoice from vendor X and log it in the sheet," "triage these 20 emails and draft replies for the urgent ones," "open a GitHub issue from this support thread." Aim for multi-step tasks with at least one tool call each.

Step 2: Build a fixture with decoys. Copy realistic data into a sandbox account. Add near-duplicates on purpose: two vendors with similar names, two files with almost the same filename, a near-identical API key in a config. Never point a test run at production data.

Step 3: Define pass/fail per task, including collateral damage. A task passes only if the target change happened and nothing else changed.

# tasks/invoice_logging.yaml
id: invoice-log-01
prompt: "Find the March invoice from Acme Supply and add it to the Q1 sheet."
fixture: fixtures/finance_sandbox.json
checks:
  - type: row_exists
    sheet: Q1
    match: { vendor: "Acme Supply", month: "March" }
  - type: no_other_changes
    allowed_paths: ["sheets/Q1"]
  - type: not_touched
    ids: ["file:acme-supplies-march-DRAFT", "row:Q1:vendor=Acme Supplies"]  # decoys
max_steps: 25

Step 4: Run each task several times per model and harness. Agent runs are stochastic. One run tells you almost nothing; five runs per task start to show reliability.

import itertools, json, statistics

models = ["model-a", "model-b"]          # your shortlist
harnesses = ["harness-x", "harness-y"]   # the tool you'll actually deploy in
RUNS = 5

results = []
for model, harness, task in itertools.product(models, harnesses, load_tasks("tasks/")):
    for i in range(RUNS):
        outcome = run_agent(model=model, harness=harness, task=task)  # your runner
        results.append({
            "model": model, "harness": harness, "task": task.id,
            "passed": outcome.passed,
            "collateral": outcome.unexpected_changes,
            "cost_usd": outcome.cost_usd,
            "seconds": outcome.seconds,
        })

json.dump(results, open("results.json", "w"), indent=2)

Step 5: Report cost per successful task, not cost per token.

def cost_per_success(rows):
    wins = [r for r in rows if r["passed"] and not r["collateral"]]
    total_cost = sum(r["cost_usd"] for r in rows)  # failures cost money too
    return total_cost / len(wins) if wins else float("inf")

That function charges the failed runs against the successes, which is the honest accounting. A model with half the per-token price and half the pass rate costs the same per finished task, before you count your own time cleaning up.

Step 6: Re-run on every model or provider change. Given that a model ID can be rerouted under you, put this in a scheduled job and compare against your last good baseline.

Decision framework: which model for which job

Use this as a starting heuristic, then let your own results overrule it.

On scheduling: with DeepSeek's peak and off-peak billing, batch jobs that don't need real-time answers can run in the cheaper window. The sourced peak windows are given in UTC (01:00–04:00 and 06:00–10:00), so convert them to your own time zone to see where they fall in your working day. Confirm the current windows and rates on the provider's page before building a schedule around them.

Putting it into practice

The approach I'd recommend before shipping an agent to production: build a small fixture from your real task types, with realistic decoys, run your candidate models through the harness you'll actually deploy in, and compare cost per successful task and collateral-change rate. The leaderboard is how you build the shortlist. The fixture result is how you decide.

Whichever model you land on, keep a human approval step on any write action that's costly or hard to undo, and re-run your benchmark whenever a provider changes or reroutes a model.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.