Enterprise AI Agents Shipped Without Governance

By Lazar Milicevic · Published September 28, 2026 · 9 min read

Your agent has write access to Gmail, Stripe, and your CRM. It runs on a service account nobody audits. If it wire-transfers $40,000 to a spoofed vendor at 2 AM, you find out from your bank. This is the actual state of most agent deployments right now — not the ones on stage at re:Invent, the ones already touching production data at the company down the street.

VentureBeat Research fielded five parallel surveys in June covering the full agentic stack, and the pattern they surfaced isn't subtle: enterprises deployed AI agents before the controls needed to manage them existed, they did it on purpose, and they are now retrofitting governance while the agents run. Budgets in every one of the five control layers they measured are up. That is the tell. Nobody spends real money on a control layer they weren't already scared of.

If you're a solo founder or a small ops team, don't read that as "big-co problem." Read it as: the people with legal, security, and audit teams already skipped this step. You don't have those teams. You need the controls anyway.

What "governance didn't catch up" actually means

Governance for an AI agent isn't a policy PDF. It's the concrete set of runtime controls that decide what the agent can touch, what it must ask before touching it, what gets logged, and who is on the hook when it does something dumb. When VentureBeat's respondents say governance "hasn't caught up," they're describing a specific gap: agents were granted broad tool access under a single service account, without per-action authorization, without human-in-the-loop on irreversible steps, and without an audit trail a compliance team could actually read.

Concretely, the layers usually missing:

If you can't answer all five for an agent you've shipped, you're in the same retrofit bucket as the enterprises in the survey.

The five layers, in the order you should fix them

Fix in this order because each one makes the next one meaningful. Logging without identity gives you noise. Approval gates without authorization scopes give you a rubber stamp.

1. Identity. Every agent action should be attributable to (a) the human who triggered it, and (b) the specific agent version that ran. Not "our automation." A user ID and an agent build hash.

2. Authorization. Scope tokens down to the minimum. If the agent drafts email replies, it does not need gmail.send — it needs gmail.drafts.create. If it reads invoices, it does not need write access to the accounting ledger.

3. Approval / human-in-the-loop. For any action that (a) spends money, (b) sends an external message, (c) writes to a system of record, or (d) can't be undone in one click — a human confirms.

4. Observability. Structured logs with a correlation ID that ties: user request → LLM prompt → tool calls → tool outputs → final response. Not "we log to CloudWatch."

5. Recovery. A kill switch you can flip in under 60 seconds and a way to see what the agent did in the last N minutes so you can undo it.

This is the order. Ship them out of order and you'll write the same controls twice.

The concrete authorization pattern

The single highest-leverage fix is to stop giving your agent one giant token. Wrap every tool in a permission check that runs on the actual user's identity, not the agent's.

from dataclasses import dataclass

@dataclass
class AgentContext:
    user_id: str           # the human who initiated
    agent_run_id: str      # unique per invocation
    agent_version: str     # git sha of the agent

def send_invoice(ctx: AgentContext, invoice_id: str, amount_cents: int):
    # 1. Authorization: does THIS user have permission?
    if not permissions.can(ctx.user_id, "invoice.send", invoice_id):
        raise PermissionDenied(ctx.user_id, "invoice.send")

    # 2. Policy: does this action need a human?
    if amount_cents > 100_000:  # $1,000
        approval = approvals.require(
            ctx, action="invoice.send",
            resource=invoice_id, amount_cents=amount_cents,
        )
        if not approval.granted:
            return {"status": "pending_approval", "id": approval.id}

    # 3. Execute with full attribution
    result = billing.send_invoice(invoice_id, actor=ctx.user_id)

    # 4. Audit
    audit.log(
        run_id=ctx.agent_run_id, user_id=ctx.user_id,
        agent_version=ctx.agent_version, action="invoice.send",
        resource=invoice_id, amount_cents=amount_cents,
        result=result.status,
    )
    return result

Three things worth noting. The permission check runs against the user, not the agent — so if a user shouldn't be able to send invoices manually, the agent can't do it on their behalf. The approval gate is inside the tool, not in the prompt — the LLM cannot talk its way past it. And the audit call is unconditional, on every path.

Human-in-the-loop without killing the point of automation

The reflex when you hear "governance" is to route everything through a human. That kills the ROI. The pattern that actually works is a tiered approval policy — cheap, reversible actions go straight through; expensive or irreversible ones stop for confirmation.

A workable default policy for a small business agent:

policies:
  - action: "email.draft"
    approval: none
  - action: "email.send"
    approval: required
    channel: slack
    timeout_minutes: 60
  - action: "calendar.create_event"
    approval: none
  - action: "invoice.send"
    approval: required
    when: "amount_cents > 100000"  # $1,000
  - action: "payment.transfer"
    approval: required
    channel: slack
    require_two_approvers: true
  - action: "crm.update_contact"
    approval: none
    reversible: true

Approvals live in the tool the human already uses. Slack for most teams. An interactive message with "Approve / Reject / See details," a 60-minute timeout, and an audit entry either way. If the approver is idle, the agent doesn't send. That's the point.

Two failure modes to design against. First, approval fatigue — if a human clicks Approve on 40 things a day, they aren't reading. Keep the required-approval list short and raise the bar. Second, prompt injection tricking the agent into skipping the gate. It can't, because the gate is in the tool code, not the prompt. The LLM can decide to call payment.transfer; it cannot decide whether the transfer executes.

What to log, and how to actually read it

"We log everything" is not observability. What you need is a per-run trace that a non-engineer can follow. One correlation ID from user request to final result, with every LLM call and tool call in between.

Minimum structured schema per event:

{
  "run_id": "run_01HZ...",
  "user_id": "u_842",
  "agent_version": "v0.14.2-a1b2c3d",
  "timestamp": "2026-09-28T14:22:11Z",
  "event_type": "tool_call",
  "tool": "invoice.send",
  "input": {"invoice_id": "inv_7781", "amount_cents": 245000},
  "output": {"status": "pending_approval", "approval_id": "apr_..."},
  "latency_ms": 312,
  "cost_usd": 0.0041,
  "policy_hits": ["amount_gt_1000_requires_approval"]
}

Ship these to whatever log store you already have — CloudWatch, BigQuery, a Postgres table with an index on run_id. The requirement is that when someone asks "what did the agent do for customer X last Tuesday?" you can answer in one query, not by grepping five services.

For LLM-specific tracing, tools like Langfuse, LangSmith, or OpenTelemetry with the GenAI semantic conventions give you the prompt/response side of the trace. Wire them to the same run_id and you can see the model's reasoning alongside the tool calls it made — which is where most bugs actually hide.

The retrofit path when you already shipped an agent

Most readers here aren't building from scratch. You've got an agent in production talking to Gmail or HubSpot or Stripe on a service account, and now you want to bring it under control without breaking it. Order of operations:

Week 1: Read-only audit. Turn on full logging. Do not change behavior yet. For one week, capture every tool call the agent makes. You want to know what it actually does, not what you think it does.

Week 2: Scope the token. Look at the audit. If the agent has gmail.modify but only ever calls drafts.create and messages.get, downgrade the scope. This one step closes most of the blast radius.

Week 3: Add the approval gate on the top three risks. Not everything. The three actions that cost money or send external messages. Route them through Slack. Watch the approval rate — if humans approve 100%, you've picked the wrong gate; loosen it. If they reject 20%, you found real bugs.

Week 4: Kill switch and runbook. A single feature flag that stops the agent from calling any tool. A one-page runbook: how to flip it, how to query the last hour of activity, who to notify. Test it. Actually flip it in staging.

After that, you're in the same posture the retrofitting enterprises are aiming for. You'll iterate on policies for months, but the acute risk is gone.

What Anthropic and OpenAI actually give you

Both major model providers now ship primitives that make the above easier, and both have obvious gaps. Use what's there; don't assume it's enough.

Anthropic's Claude API supports tool use with explicit input schemas, and you can inspect the tool call before executing it. That's the hook where your authorization check goes. Claude also supports prompt caching, which matters for governance because it lets you pin a long system prompt (containing your policy rules) at low incremental cost, so you're not tempted to shorten it.

OpenAI's function calling and the Assistants API give similar hooks. Both providers stop short of shipping the approval, audit, and recovery layers — those are yours to build. Don't expect the vendor to enforce your policy.

Where you shouldn't rely on the model: content filtering as a security boundary. If your prompt says "never send more than $10,000 without approval," and the enforcement is only in the prompt, you have no enforcement. Every real control has to be in code the model cannot see or bypass.

Common retrofit mistakes I keep seeing

A short list, from actual client codebases:

• Approval prompt inside the LLM system prompt. "Ask the user before spending money." The model complies 95% of the time, which means it fails 1 in 20. Move the check to the tool.
• Logs without a correlation ID. You have logs from the LLM, logs from the tools, logs from the queue. None share an ID. Debugging takes hours instead of minutes.
• Service account with owner-level access. Because it was easier to set up. It's always easier to set up. Scope it down before you scale up.
• No agent version in the audit. Something broke last Thursday. Which prompt was live? Which tool schema? Nobody knows. Tag every deployment.
• Human-in-the-loop over email. Approvals sit in an inbox for three days. Use a channel with a timeout.
• Testing only the happy path. The interesting failures are ambiguous inputs, partial tool errors, and injection attempts. Build a small eval set of nasty cases and run it on every deploy.

How BizFlowAI approaches this

We build agents on Claude for small teams — most of them 2 to 15 people — and every deployment ships with the five control layers wired in from day one. Identity per user, tool-level authorization, tiered approval gates in Slack, structured audit logs with correlation IDs, and a kill switch. Not because clients ask for it; because we've watched what happens without it.

If you already have an agent running and you're looking at this list wondering which of the five you're missing, that's exactly the discovery call we do. We'll go through your current tool scopes, your logging, and the top three actions that could hurt if they went wrong, and give you a concrete retrofit plan you could hand to your own developer or have us ship. No slide decks.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.