Why Single-Trace Agent Evals Hide Broken Products

By Lazar Milicevic · Published July 22, 2026 · 9 min read

You ship a Claude agent to production. You spot-check ten conversations. Every one looks clean — tools called correctly, tone on-brand, answers grounded. Two weeks later, support tickets spike, retention on the cohort that touched the agent drops, and you have no idea which conversations caused it. This is the gap that Harrison Chase (LangChain), Hui Zhang (Conviva) and Emmanuel Turlay (CoreWeave) named on stage at VB Transform 2026: a single agent conversation can score perfect and still point at a broken product.

If you are a solo builder or a small ops team running agents against real customers, the fix isn't more prompt tweaking. It's changing the unit of evaluation from the trace to the cohort.

The single-trace illusion

A trace is one conversation: user turns, model turns, tool calls, retrieved context, final answer. Scoring a trace answers did this specific run behave. It does not answer is the product working.

Three failure modes get through single-trace evals every time:

1. Correct answer, wrong outcome. The agent answers the question the user typed, not the question they meant. The trace passes an LLM-judge rubric. The user churns.
2. Locally right, globally regressive. A prompt change fixes 5 traces you looked at and quietly breaks 300 you didn't — a class of users whose phrasing sits outside your sample.
3. Behaviorally fine, economically broken. Every trace resolves. Median resolution takes 14 turns instead of 3. Token spend triples. Users get tired and drop off before the "good" answer arrives.

None of those show up when a reviewer opens LangSmith or Braintrust and reads a conversation. They show up when you compare this week's cohort of users who touched the agent against last week's, or against a control cohort that didn't.

What "cohort evaluation" actually means

Cohort evaluation borrows from product analytics and applies it to agents. Instead of asking "was this trace good?" you ask "did the group of users that hit version B of the agent behave measurably differently — on outcomes we care about — than the group that hit version A, or than the pre-agent baseline?"

The unit is a user × session, not a trace. The metrics are product metrics, not rubric scores:

Trace evals stay. They are your unit tests. Cohort evals are your integration tests against reality — and they are the only layer that tells you whether the agent is helping the business.

Instrumenting for cohort evals: the minimum stack

You don't need a data platform to start. You need three things wired together: a session id on every trace, a cohort tag on every user, and an outcome event you can join back.

Here's the minimum I ship for clients running Claude agents in production:

# every agent turn carries the same session_id + user_id
import anthropic, uuid, time
from analytics import track  # your product analytics (PostHog, Segment, etc.)

client = anthropic.Anthropic()

def agent_turn(user_id: str, session_id: str, agent_version: str, message: str):
    t0 = time.time()
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        system="...",
        messages=[{"role": "user", "content": message}],
        metadata={"user_id": user_id},
    )
    latency_ms = int((time.time() - t0) * 1000)

    track(user_id, "agent_turn", {
        "session_id": session_id,
        "agent_version": agent_version,     # <-- the cohort key
        "input_tokens": resp.usage.input_tokens,
        "output_tokens": resp.usage.output_tokens,
        "latency_ms": latency_ms,
        "stop_reason": resp.stop_reason,
    })
    return resp

Every user is assigned agent_version at session start (deterministic hash of user_id, so a user stays in one cohort). That single field is what lets you run:

-- 7-day resolution rate by cohort
select
  agent_version,
  count(distinct session_id)                                as sessions,
  avg(case when outcome = 'resolved' then 1 else 0 end)     as resolve_rate,
  percentile_cont(0.5) within group (order by turns)        as median_turns,
  sum(input_tokens + output_tokens) / count(distinct session_id) as tokens_per_session
from agent_sessions
where started_at > now() - interval '7 days'
group by agent_version;

If agent_version = v3.2 shows resolve_rate 0.71 and v3.3 shows 0.68 with median_turns up from 4 to 7 — you have a regression, even though every trace you spot-checked looked fine.

Choosing the baseline (this is where most teams fail)

Cohort evals are only as good as the baseline. Three baselines that work:

• Time-shifted baseline. Same users, same funnel, previous agent version. Cheapest to set up. Weak against seasonality — Monday users behave differently than Saturday users.
• Concurrent A/B. Hash users into control and treatment at the moment of eligibility. Cleanest signal. Requires you to be willing to ship two versions in production.
• No-agent control. A slice of users routed to the pre-agent flow (human, form, or old rule-based system). Painful, but the only baseline that answers "is the agent net positive at all?"

For anything that touches revenue or retention, run a concurrent A/B for at least one full business cycle before declaring a version "shipped." Harrison Chase's point on stage was blunt: teams that skip this step end up debugging churn six weeks later with no idea which prompt change caused it.

Metrics that actually matter (and the ones that lie)

Not every product metric is a good agent metric. Some move too slowly to catch a regression. Some are dominated by factors the agent doesn't touch. A working shortlist:

Trust these:

• Task completion rate (did the user's stated goal get resolved, per session)
• Median turns to resolution
• Escalation-to-human rate
• Tokens per resolved session (cost signal — silent regressions live here)
• 7-day return rate for users whose first session used version X
• CSAT / thumbs-up rate conditional on the user leaving one (watch for response bias)

Be careful with:

• Raw thumbs-up count — biased toward angry users
• Average sentiment — one bad turn drags the mean
• LLM-judge "helpfulness" scores at the cohort level — the judge has the same blind spots as the agent

Ignore as primary signal:

• Session length in minutes (longer can mean engaged or frustrated)
• Total messages (same problem)

Zhang's example from Conviva was worth stealing: they watched a version where LLM-judge scores went up while return rate went down. The judge liked the more thorough answers. Users found them long and left.

A working eval loop, end to end

Here's the shape of what I run in production for clients. It's a scheduled job, not a dashboard people remember to open.

# eval-loop.yaml — runs nightly
schedule: "0 3 * * *"

steps:
  - name: pull-sessions
    query: |
      select session_id, user_id, agent_version, outcome,
             turns, tokens_total, resolved_at, started_at
      from agent_sessions
      where started_at between now() - interval '8 days'
                          and now() - interval '1 day'

  - name: cohort-metrics
    group_by: agent_version
    metrics:
      - resolve_rate
      - median_turns
      - p90_turns
      - tokens_per_session
      - escalation_rate
      - d7_return_rate

  - name: regression-check
    baseline: previous_version
    fail_if:
      - resolve_rate.delta < -0.03
      - median_turns.delta > 1.5
      - tokens_per_session.delta > 0.25   # 25% cost jump
      - escalation_rate.delta > 0.05

  - name: notify
    on_fail: slack://#agent-alerts
    payload: cohort_report.md

Two rules I hold to:

1. Fail loud, fail early. The alert names the metric, the delta, and links to a random sample of 20 sessions from the regressing cohort. Not 20 traces — 20 sessions, joined to outcome.
2. Don't roll back on trace vibes. If cohort metrics are green and one trace looks weird, that's a bug to file, not a rollback trigger. Rollbacks are triggered by cohort deltas.

What a real regression investigation looks like

You get the Slack alert: v3.3 shows tokens_per_session +34%, median_turns 4 → 6, resolve_rate flat. Same resolution, more work.

The investigation is not "read 50 traces." It is:

1. Segment the cohort. Split v3.3 sessions by user type, entry point, tool-call path. Which sub-segment carries the delta?
2. Compare paired sessions. Find users who had a session on both v3.2 and v3.3. Diff their tool-call sequences.
3. Only now, read traces. Pull 10 sessions from the regressing sub-segment where the delta is largest. Look for the shared pattern.

Nine times out of ten the culprit is boring: a system-prompt tweak made the agent double-check itself once more per turn, or a retrieval change added a step, or a tool description got vaguer and Claude now asks a clarifying question it used to skip. You would not have found it by reading traces top-down.

Anthropic's own guidance on building effective agents makes a related point: agent quality is a function of the loop, not any single call. Cohort evals are how you measure the loop.

Where BizFlowAI approaches this

Most of the Claude agents we take over from clients were built the way everyone builds them the first time: a prompt, a few tools, LangSmith open in a tab, ship when the last five traces looked good. They work — until a prompt change breaks a cohort nobody was watching. What we install is the plumbing described above: session-scoped tracing tied to the client's product analytics, a cohort key on every user, a nightly eval job with a real baseline, and Slack alerts that fire on metric deltas rather than trace vibes.

The heavy lift is usually the outcome join — most teams have traces in one system and business events in another, and nothing that ties a session_id to whether the user actually got what they came for. Once that pipe exists, prompt changes stop being scary and start being measurable. If your agent is already in production and you can't answer "did last week's version convert better than this week's" in a query, that's the discovery call we run.

Getting started this week

You don't need a platform. You need a week of plumbing. In order:

1. Day 1-2. Add session_id, user_id, agent_version to every trace. Emit an agent_turn event to your existing analytics tool. If you have no analytics tool, PostHog free tier is fine.
2. Day 3. Define one outcome event that matters — ticket_resolved, checkout_completed, answer_marked_helpful, whatever your product already tracks. Join it to session_id.
3. Day 4. Write the cohort SQL above. Run it against last month's data. You now have a baseline.
4. Day 5. Wire the nightly job. Ship the Slack alert with real thresholds — not perfect ones, real ones you can revise.
5. Ongoing. Every prompt or tool change gets a new agent_version. Wait one business cycle before calling it shipped.

The point Chase, Zhang and Turlay were making on stage isn't a philosophical one. It's operational. Agents fail at the cohort layer, not the trace layer, and if you're not measuring at the cohort layer you're flying on vibes. The teams shipping agents that stick are the ones that treated evaluation as a data problem the day they went to production — not the day the first churn report landed.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.