Most "AI Agents" in Production Are Just Chatbots

By Lazar Milicevic · Published July 16, 2026 · 11 min read

You've been told to "add agents" to your product or ops stack. You look around and everyone else claims they already did it. Then you actually inspect what they shipped and it's a chat window calling one API with a system prompt. That gap — between what leadership thinks was deployed and what actually runs — is the real story of enterprise AI right now.

The recent wave of enterprise agent surveys makes this obvious. Orchestration is consolidating onto model-provider platforms (Anthropic's Claude is the current front-runner by usage among enterprises building agent workloads), buyers pick a platform based on the gravity of the underlying model, and they judge success on one axis: reliable multi-step execution. But the ambition runs miles ahead of the deployments. Most "agents" in production are chatbot wrappers. The control plane most enterprises actually want is hybrid, not single-vendor. And nobody has fully solved the "who is liable when the agent takes an action" question.

This post is for the operator who has to ship something real, not a slide. Here's what's actually going on, and how to build past it.

The deployment problem, stated plainly

Enterprise AI is not stuck because the models are weak. GPT-class and Claude-class models can already reason across multiple tools, handle branching logic, and produce structured output that survives contact with a real system. The models are ready. What's not ready is the wiring around them.

The deployment problem has three parts:

1. Naming inflation. A chat UI over one LLM call gets shipped as "our AI agent." It has no tools, no memory beyond the session, and no ability to take a real action. It answers questions. That is a chatbot.
2. Missing control plane. Once an agent actually does things — writes to a CRM, moves money, files a ticket, sends an email to a customer — you need auth, audit, retry, rollback, human-in-the-loop, and observability. Most orgs have none of this and no owner assigned to build it.
3. Hybrid reality vs single-vendor pitch. The vendor pitch is "buy our platform and get agents." The real deployment is Claude for reasoning, a smaller model for classification, an internal API gateway for tools, a workflow engine for retries, and a homegrown eval harness. Every serious deployment is hybrid whether the vendor slide admits it or not.

If your "agent project" hasn't hit these three walls yet, you probably haven't shipped an agent yet. You've shipped a chatbot.

What actually qualifies as an agent

A useful working definition, one you can defend in a review meeting:

An agent is a system where an LLM decides, at runtime, which tools to call and in what order, executes those calls, observes the results, and loops until a goal state or stop condition is reached — with side effects on real systems.

Three things matter in that sentence: decides at runtime, executes with side effects, and loops. Strip any one out and you have something simpler.

Most enterprise "agents" you'll see in a demo are row 1 or row 3. Nothing wrong with either — a well-scoped RAG bot or a fixed workflow is often the correct answer. Just don't call it something it isn't, because your monitoring, cost model, and risk posture all differ based on which row you're on.

Why orchestration is consolidating on model providers

The interesting shift over the last 12–18 months is that orchestration — the layer that decides which tool to call, tracks state across steps, and manages retries — is moving into the model provider's stack. Anthropic's tool-use plus MCP (Model Context Protocol), OpenAI's Responses API and agent primitives, Google's agent tooling in Vertex — the providers are absorbing what used to be a LangChain-style middleware layer.

Why? Because the model itself is the state machine. When the reasoning quality of one model is meaningfully better than the next, you'd rather push more of the decision logic into that model's native format than translate through a third-party abstraction. The gravity is the model. The orchestration follows.

For the buyer, this has practical consequences:

• Lock-in is real but not fatal. You will write Claude tool schemas or OpenAI tool schemas, not portable ones. Plan a 1–2 week port cost per model swap, not zero.
• Middleware value is shrinking. LangChain, LlamaIndex, and CrewAI still solve problems, but the "orchestration framework" pitch is thinner than it was. Use them where they save real work; don't adopt them by default.
• MCP matters more than the average blog post admits. MCP is the closest thing to a portable interface between agents and tools. If you're building tools now, make them MCP servers. Your future self will not have to rewrite them.

The hybrid control plane enterprises actually want

Nobody serious is running production agents with "the model provider is our entire stack." The pattern in real deployments looks more like this:

                    ┌─────────────────────┐
                    │  Reasoning model    │  ← Claude / GPT
                    │  (agent loop)       │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │  Tool gateway       │  ← Auth, rate limits,
                    │  (MCP / internal)   │    audit log, PII scrub
                    └──────────┬──────────┘
                               │
        ┌──────────────┬───────┼───────┬──────────────┐
        ▼              ▼       ▼       ▼              ▼
    ┌───────┐    ┌────────┐ ┌─────┐ ┌──────┐    ┌──────────┐
    │  CRM  │    │ Billing│ │Email│ │Search│    │ Human    │
    │       │    │        │ │     │ │      │    │ approval │
    └───────┘    └────────┘ └─────┘ └──────┘    └──────────┘

The control plane sits between the model and your systems. It exists because your CFO, your CISO, and your on-call engineer all need it to exist. It handles:

• Auth and scope. The agent acts as a specific principal with specific permissions. Not as "the app."
• Rate limits and budget. Per-tenant, per-tool, per-day. Otherwise one runaway loop drains your OpenAI budget in an afternoon.
• Audit log. Every tool call, with inputs, outputs, model reasoning trace, and timestamp. Legal will ask for this.
• Human-in-the-loop. Actions above a risk threshold pause and wait for approval.
• Idempotency and rollback. Retries don't double-charge customers. Failed multi-step runs can be reversed or reconciled.

Here's a stripped-down example of what a real tool-call wrapper looks like — this is the kind of glue that separates a chatbot from an agent:

from typing import Any
import hashlib, json, time

def call_tool(tool_name: str, args: dict, principal: str,
              run_id: str) -> dict[str, Any]:
    # 1. Auth: does this principal have scope for this tool?
    if not authz.can(principal, tool_name):
        return {"error": "forbidden", "tool": tool_name}

    # 2. Budget: has this run exceeded its per-tool budget?
    if budget.exceeded(run_id, tool_name):
        return {"error": "budget_exceeded"}

    # 3. Idempotency: same call in this run returns cached result
    key = hashlib.sha256(
        f"{run_id}:{tool_name}:{json.dumps(args, sort_keys=True)}".encode()
    ).hexdigest()
    if cached := cache.get(key):
        return cached

    # 4. Human approval gate for high-risk tools
    if policy.requires_approval(tool_name, args):
        approval = approvals.wait(run_id, tool_name, args, timeout=3600)
        if not approval.granted:
            return {"error": "not_approved"}

    # 5. Execute + audit
    started = time.time()
    try:
        result = registry[tool_name](**args)
        audit.log(run_id, principal, tool_name, args, result,
                  duration_ms=(time.time()-started)*1000)
        cache.set(key, result, ttl=300)
        return result
    except Exception as e:
        audit.log(run_id, principal, tool_name, args,
                  error=str(e), duration_ms=(time.time()-started)*1000)
        raise

None of that is exotic. It's the plumbing every "we shipped an agent" story quietly required. Skip it and you don't have an agent, you have a liability.

Reliable multi-step execution is the only metric that matters

Ask a vendor how they measure agent quality and you'll get benchmark scores. Ask an operator running agents in production and you'll get one number: the fraction of runs that complete the full task without human rescue.

Call it end-to-end success rate. On a hard, multi-tool task with 5+ steps, a strong current-gen model with a well-designed tool set lands somewhere in the 60–85% range in real deployments I've seen. That's not a benchmark I'm making up — that's the range where teams start to trust the system enough to remove the human review step on low-risk actions. Anything below that and you're running a very expensive typing assistant.

The three things that move that number:

1. Fewer, sharper tools. Ten well-named tools with clear schemas beat forty overlapping ones. The model spends less reasoning budget picking, and picks correctly more often.
2. Structured output at every hop. If a tool returns a 4KB blob of prose, the next reasoning step degrades. Return typed, minimal fields. Push the prose to a "notes" field the model can ignore.
3. Explicit stop conditions. Every agent loop should have a max-step budget, a max-cost budget, and a goal predicate. Runs that hit the budget without hitting the goal get flagged, not silently swallowed.

Here's a minimal loop that respects all three:

def run_agent(goal: str, max_steps: int = 20, max_usd: float = 2.00):
    messages = [{"role": "system", "content": AGENT_SYSTEM},
                {"role": "user", "content": goal}]
    cost = 0.0

    for step in range(max_steps):
        resp = model.chat(messages, tools=TOOLS)
        cost += resp.usage.cost_usd

        if cost > max_usd:
            return {"status": "budget_exceeded", "step": step, "cost": cost}

        if resp.stop_reason == "end_turn":
            return {"status": "done", "step": step, "cost": cost,
                    "answer": resp.text}

        for call in resp.tool_calls:
            result = call_tool(call.name, call.args,
                               principal=USER, run_id=RUN_ID)
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": json.dumps(result)})

    return {"status": "max_steps", "cost": cost}

You can build fancier: parallel tool calls, planner/executor split, reflection steps. But if the basic loop above isn't instrumented and observable, none of the fancy stuff will save you.

Where most enterprise agent projects actually fail

Pattern I've seen enough times to call it a pattern:

• Month 1: Demo built in a week. Everyone's excited.
• Month 2–3: Someone asks about auth, PII, and audit. Progress slows.
• Month 4–5: Integration with the real CRM/ERP/ticketing system is 10x harder than the mocked-up demo. Model calls are correct; the CRM API rejects half of them because of field-level validation nobody knew about.
• Month 6: Pilot runs on a narrow use case. Success rate is 55%. Business owner says "call me when it's 90."
• Month 7: Project quietly rebranded from "agent" to "AI assistant." Chat UI ships. Nobody uses it.

The failure isn't the model. It's that the org treated the agent as a model project when it was actually a systems integration project with a smart component in the middle. The team composition was wrong. You needed backend engineers who know your internal APIs, not ML people who know prompting.

The fix is unglamorous: pick one workflow that's currently done by a human in 10–30 minutes, that touches 2–4 systems, and where a wrong action is recoverable. Build the tool gateway, wire in Claude or your model of choice, ship it with human approval on every action, measure the end-to-end success rate for two weeks, then start removing approvals on the categories that hit 95%+.

How BizFlowAI approaches this

We take the "chatbot called an agent" problem literally. When a client comes to us saying they want an agent, our first week is spent mapping the actual workflow, the systems it touches, and the failure modes a wrong action would create. Only then do we decide whether the answer is a real agent loop (Claude + MCP tools + the control-plane wrapper above), a fixed workflow with an LLM step, or a plain automation with no LLM at all. Often it's the third one, and we say so.

When it is a real agent, we build the tool gateway before we build the prompt. Auth, audit, idempotency, and human-in-the-loop are non-negotiable. The reasoning model is swappable; the control plane is what makes the system safe to run against a real CRM, a real billing system, or a real inbox. If you want to see the difference between a chatbot wrapper and a multi-step agent that actually completes work end-to-end, book a discovery call and we'll walk through one of yours.

What to do this quarter if you're on the hook for "agents"

Short list, in order:

1. Audit what you already call an agent. For each one, mark it chatbot, workflow, assistant, or agent using the table earlier in this post. Do not skip this. Most orgs discover they have zero agents and three chatbots with agent branding.
2. Pick one real agent candidate. Multi-step, cross-system, recoverable if wrong. Not customer-facing on day one.
3. Build the tool gateway first. Auth, audit, budget, approval, idempotency. This is 2–4 weeks of work for a competent backend engineer and it's the thing that lets you sleep.
4. Wire the reasoning model in second. Claude or GPT, tool-use format, MCP for anything you'll want to reuse. Keep the tool set small.
5. Measure end-to-end success rate. Not token cost, not latency, not benchmark scores. Fraction of runs that complete the goal without human rescue.
6. Decide what "good enough" means before you start. Write down the success rate at which you'll remove human approval, and for which action categories.

The organizations winning at agents right now are not the ones with the biggest model contracts. They're the ones treating agents as a systems problem with a smart component in the middle, and staffing accordingly. The model is table stakes. The rest is engineering you already know how to do.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.