Why Your AI Agent Gives Different Answers on the Same Input

By Lazar Milicevic · Published October 22, 2026 · 10 min read

You ran the same document through your agent four times and got four different results. You checked the temperature, set it to 0, and it still happened. Before you blame the model, read what CTGT found when it hit this exact problem, because the cause was probably not where it looked first.

On October 21, 2026, VentureBeat reported that CTGT is releasing Concordance, an Apache 2.0 open-source tool for tracing why an AI application makes different decisions on the same task. This post uses that case as a worked example, then turns it into a debugging method you can apply to your own agents, with or without Concordance.

The core finding: the model was consistent, the pipeline wasn't

When an agent produces different outputs from identical input, the model itself is often not the source of the variation. CTGT's own experiment showed the pipeline was changing what it sent to the model, so the model faithfully answered different questions each time.

The setup, per VentureBeat's reporting: CTGT's compliance application processed the same nine-page retail banking manual four times, using Qwen3.8-27B (as named in the article; check an official Qwen source before citing the model name elsewhere) on SGLang's deterministic inference mode. The four runs extracted 32, 31, 33 and 33 obligations, and all four outputs differed. In one run, the pipeline omitted a requirement to escalate a serious issue to the compliance chief.

The natural assumption is "LLMs are non-deterministic, so this is expected." That assumption would have sent the team in the wrong direction. Under the deterministic configuration, the model returned consistent responses across all 74 identical requests that recurred during the runs. The model was fine. The application kept changing what it sent.

This matters for how you debug. If you assume the model is the problem, you reach for prompt tweaks, a different model, or lower temperature. If the pipeline is the problem, none of those help, and you waste days.

One caveat on scope, which VentureBeat also states: this is one application processing one document. It does not tell us how common this failure is across deployments. Treat it as a well-documented example of a failure class, not a statistic.

Three leaks that made identical inputs different

CTGT traced the divergence to three causes. All three are ordinary engineering mistakes that show up in almost any multi-step LLM pipeline.

The sharpest detail is how small the difference was. In one comparison, only four identifier lines differed within a 12,525-character request. That tiny change was enough for the model to classify five conflict-resolution rules as background information, which dropped an obligation that another run retained.

So four lines out of 12,525 characters produced a missing compliance requirement. You would never find that by reading final outputs. You find it by diffing the actual requests.

These patterns are worth checking in your own stack:

• Database IDs, UUIDs, timestamps, filenames or session tokens interpolated into prompts. If the model doesn't need them, they are noise that breaks reproducibility.
• Lists built from sets, dictionary iteration or unsorted query results. Ordering that "usually" looks the same is not guaranteed.
• Shared mutable state such as registries, caches, "known topics" lists or memory stores that grow across runs and feed back into later prompts.

Find the earliest divergence, not the final difference

The debugging principle in the Concordance approach is simple: when two runs disagree, stop comparing final outputs and find the earliest step where they diverged. Everything downstream of that point is a consequence, not a cause.

Per the article, Concordance captures exact model requests and responses, compares outputs at successive stages, and identifies the earliest divergence. It does not require access to model weights, so in principle it can be used even when you don't host the model yourself (check the tool's documentation for what it supports).

You can apply the same logic without any special tool. The method:

1. Log every model call as a full record: the exact request payload and the exact response.
2. Tag each call with a stable step name (for example extract_obligations, classify_rule, route_topic), not just a timestamp.
3. Run the same input N times.
4. Walk the steps in order and compare the requests first, then the responses.
5. Stop at the first step where anything differs.

Here is a minimal sketch of that comparison in Python. It assumes you've saved each run as a JSON list of {"step": ..., "request": ..., "response": ...} records.

import json
import difflib

def load_run(path):
    with open(path) as f:
        return json.load(f)

def first_divergence(run_a, run_b):
    """Return the first step where requests or responses differ."""
    for i, (a, b) in enumerate(zip(run_a, run_b)):
        if a["request"] != b["request"]:
            return i, a["step"], "request", a["request"], b["request"]
        if a["response"] != b["response"]:
            return i, a["step"], "response", a["response"], b["response"]
    if len(run_a) != len(run_b):
        return min(len(run_a), len(run_b)), "length", "steps", None, None
    return None

def show_diff(x, y):
    xs = json.dumps(x, indent=2, sort_keys=True).splitlines()
    ys = json.dumps(y, indent=2, sort_keys=True).splitlines()
    for line in difflib.unified_diff(xs, ys, lineterm="", n=1):
        print(line)

run1 = load_run("run_1.json")
run2 = load_run("run_2.json")

result = first_divergence(run1, run2)
if result is None:
    print("Runs are identical end to end.")
else:
    idx, step, kind, a, b = result
    print(f"First divergence at call #{idx} ({step}), in the {kind}:")
    show_diff(a, b)

The key detail is the order of checks. If the request differs at step N, your pipeline changed the input, and that's a code problem. If the request is identical but the response differs, then the model or serving layer is the source, and that's a different investigation.

That branching is exactly what CTGT's data showed: identical requests got identical responses, so the variation had to come from upstream.

"Temperature 0" is not a determinism guarantee

Setting temperature to 0 does not make an LLM deterministic in practice, even on self-hosted stacks. Thinking Machines Lab's write-up, Defeating Nondeterminism in LLM Inference, found that temperature 0 does not make LLM APIs deterministic, and that the same holds for self-hosted vLLM or SGLang.

So there are really two layers where variation can enter:

CTGT's experiment is useful precisely because it separated these. Using SGLang's deterministic inference mode, CTGT's separate serving test showed Qwen3.8-27B matched its control response in all 800 requests. That measures exact textual agreement, not correctness. Even with the serving layer locked down, the application layer still produced four different outputs.

I won't quote performance overhead numbers for deterministic modes here, because I haven't verified them against a primary source. If you plan to turn one on, benchmark throughput on your own workload and check the current documentation for your inference engine.

If you use a hosted API, you generally can't control the serving layer at all. That makes the application-layer discipline below even more important, because it's the only layer you own.

The fixes were boring, and that's the point

After stabilizing the inputs, CTGT ran five subsequent runs and each produced the same 34 obligations. The fixes, per the article:

• Replace identifiers with stable labels. Instead of upload_8f3a91c2, use document_1.
• Sort the list. Make the 30-item routing list order deterministic.
• Isolate each test in a fresh organizational environment. No state carried over from prior runs.

None of this is clever. Here's what the first two look like in practice:

import re

def stabilize_prompt_inputs(items, doc_id):
    # 1. Stable label instead of a per-upload identifier
    doc_label = "document_1"

    # 2. Deterministic ordering for anything list-shaped
    routing = sorted(items, key=lambda x: x["name"].lower())

    return doc_label, routing

def scrub_ids(text):
    # Replace UUID-shaped strings with a stable placeholder
    return re.sub(
        r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b",
        "<ID>",
        text,
    )

And for the third fix, the equivalent in a typical SMB stack is making each test run start from a clean slate: a fresh database schema or namespace, an emptied cache, and a reset memory store. If your agent writes to a "known topics" table during run 1 and reads from it in run 2, you're not testing the same system twice.

A word of caution, again from the article: stabilizing inputs fixed repeatability, not correctness. Which brings us to the most important limitation.

Repeatable is not the same as correct

Getting the same answer every time proves nothing about whether the answer is right. CTGT's data makes this concrete: replaying the request behind an incorrect classification produced the same mistake eight times.

That means a consistency check and a correctness check are two different tests, and you need both:

• Consistency test: run the same input N times, expect identical (or acceptably equivalent) outputs. Catches pipeline instability.
• Correctness test: run inputs with known expected results, compare against ground truth. Catches systematically wrong behavior.

A pipeline can pass the first and fail the second indefinitely. If you only measure consistency, you can end up with an agent that is reliably wrong, which is arguably worse because it looks trustworthy.

Also note what Concordance does not claim. Concordance is a diagnostic for finding where runs diverge. It doesn't tell you which run was right, and repeatability is not the same as accuracy.

A practical reliability checklist for agents that commit decisions

CTGT founder and CEO Cyril Gorlla recommends three practices, per VentureBeat. They translate directly into a checklist for any agent that commits a decision (approves, files, sends, classifies, routes):

1. Preserve complete requests and responses for model calls that commit decisions. Not summaries, not "the prompt template," but the exact payload and the exact reply. When something goes wrong three weeks later, this is your only evidence.
2. Repeat tests when models, prompts or source documents change. Any of those three can shift behavior. Treat a model upgrade or a prompt edit like a code deploy that needs a regression run.
3. Keep trusted examples with known expected results. A small set of inputs where you know the right answer. This is your correctness anchor, since consistency alone can't provide one.

Here's a lightweight way to wire the second and third together as a pre-deploy check:

# regression_suite.yaml
runs_per_case: 5
cases:
  - name: escalation_rule_present
    input: fixtures/banking_manual_excerpt.txt
    expect:
      must_include:
        - "escalate to compliance chief"
  - name: invoice_total_extraction
    input: fixtures/invoice_0042.pdf
    expect:
      equals:
        total: "1,240.00"
        currency: "USD"
consistency:
  max_distinct_outputs: 1   # all 5 runs must agree

If max_distinct_outputs exceeds 1, diff the logged requests with the divergence script above. If consistency passes but a must_include or equals check fails, you have a stable-but-wrong agent, which needs a prompt or logic fix rather than a determinism fix.

Where Concordance fits: it's positioned as an addition to existing tracing tools such as LangSmith and Braintrust, not a replacement, and CTGT supplied no head-to-head benchmark against them. If you already use a tracing tool, the question is whether its comparison features surface the earliest request-level divergence. If they don't, Concordance or a script like the one above fills that gap.

Be realistic about cost, too. The article notes that an enterprise adopting this tooling must still integrate it with its workflows and account for the model calls, storage and engineering work that testing involves. The release includes the testing protocol, reference implementation and experimental data, and businesses may use and modify the software commercially. CTGT says no paid or hosted edition is planned. But the proprietary compliance application used in the experiment is not part of the release, so you're getting the diagnostic method, not CTGT's pipeline.

Putting this into practice

The failure mode in CTGT's experiment, where identical inputs reach the model as subtly different requests, is an example of a class of errors that can show up in any multi-step LLM pipeline. If an agent of yours commits decisions, the approach I'd recommend is the one above: keep full request/response records for every decision-committing call, run regression tests against known-good examples before any prompt, model or document change ships, and put limits on the actions an agent can take, so a wrong-but-consistent answer can't silently trigger a real-world side effect.

Concordance is a separate open-source project from CTGT and is not related to BizFlowAI; the method it demonstrates (find the earliest divergence in the logged requests) is something you can do with or without it. If you have an agent that's about to touch real workflows like invoicing, lead follow-up or email triage, run the consistency and correctness checks above before it goes live, so it behaves on Tuesday the way it did in testing on Monday.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.