Why Legacy Infra, Not Models, Slows Your Agents

By Lazar Milicevic · Published July 18, 2026 · 10 min read

Your agent decides what to do in 40 milliseconds. Then it waits 4 seconds for a mainframe SOAP call to return a customer record. That gap — between how fast the model reasons and how slow the systems around it respond — is where most agent projects quietly die. It's also what LinkedIn, Walmart, and Zendesk spent an hour dissecting at VB Transform 2026.

The panel — Animesh Singh (LinkedIn), Desiree Gosby (Walmart), and Zendesk's infra lead — kept returning to the same point: the models are fine. The plumbing isn't. If you're a solo builder or a small ops team trying to ship an agent into a real business, this is the part nobody warned you about.

The bottleneck isn't the model, it's everything around it

The direct answer: LLM inference is now the fastest link in the chain. A GPT-class model streams first tokens in tens of milliseconds. But an agent that does actual work — reads a ticket, checks inventory, updates a CRM — inherits every millisecond of latency from every system it touches. Legacy CRMs, batch ETL, ORM-heavy microservices, and SOAP endpoints add seconds, not milliseconds.

The Transform panel framed it simply. Singh pointed out that at LinkedIn, model latency stopped being the dominant cost inside their agent stack once they moved to streaming and speculative decoding. The dominant cost became fetching authoritative user, connection, and content signals from systems that were designed for page renders, not sub-second agent loops.

Concretely, here's what "the gap" looks like when you profile a real agent turn:

Same model. Same prompt. Different infrastructure. The agent that answers in half a second feels intelligent. The one that answers in six seconds feels broken.

What LinkedIn, Walmart, and Zendesk actually changed

The direct answer: none of the three rebuilt their monoliths. They wrapped them. Each company kept the systems of record and put a fast, agent-shaped layer in front — event streams, materialized read models, and a thin protocol layer (MCP or an internal equivalent) that agents call instead of the raw backend.

Singh described LinkedIn's approach as "agents don't talk to services, agents talk to a curated context surface." That surface is built by upstream jobs that pre-compute what an agent will likely need — member profile snapshot, recent activity, permissions — and keep it warm in a low-latency store. When the agent asks, the answer is already sitting there.

Gosby's version at Walmart was similar but framed around inventory and order data. They exposed a small number of high-quality, agent-consumable APIs on top of dozens of legacy systems, and pushed authoritative writes back through the old paths asynchronously. The agent sees a fast, consistent world. The old systems catch up when they catch up.

Zendesk's angle was on the customer support side: streaming ticket context to the agent as it's being composed by the user, not after submit. Same pattern — remove the request/response round trip from the hot path.

Three different businesses, one shared shape:

1. Read from a fast, curated projection. Never the source system directly during an agent turn.
2. Write through a durable event bus. The agent emits an intent; a worker reconciles it with the legacy system.
3. Expose everything to the agent through one protocol (MCP-shaped tool calls) so the model doesn't care what's behind them.

The pattern: event streams and read models in front of legacy

The direct answer: put Kafka (or Pub/Sub, or Kinesis, or Redpanda — pick your poison) between your agents and your systems of record. Materialize the shapes your agents need into a fast store. Treat the legacy database as the eventually-consistent tail, not the hot path.

Here's a minimal version of the pattern that a small team can actually run:

# Read path (agent -> curated projection)
agent_tool: get_customer_context
  reads_from: redis://ctx-cache
  fallback: postgres://read-model
  never_reads: legacy-crm-soap-endpoint

# Write path (agent -> event -> legacy)
agent_tool: update_customer_note
  publishes_to: kafka://customer.notes.v1
  ack_semantics: at-least-once
  reconciler: legacy-crm-writer-worker
  agent_sees: optimistic_write_confirmation

The agent never blocks on the legacy CRM. It publishes an event and gets an immediate confirmation from the read model. A background worker — which can retry, batch, and rate-limit against the SOAP endpoint — handles the actual write. If the write fails, it emits a compensating event and the read model rolls back.

For a solo builder, the smallest useful version is:

# Fast read model backed by Postgres logical replication
async def get_customer_context(customer_id: str) -> dict:
    # <20ms p99 from a denormalized table
    row = await pg_read.fetchrow(
        "SELECT ctx FROM customer_context WHERE id = $1",
        customer_id,
    )
    return row["ctx"]

# Agent write that doesn't block on legacy
async def add_note(customer_id: str, note: str) -> dict:
    event_id = uuid4().hex
    await bus.publish("customer.notes.v1", {
        "event_id": event_id,
        "customer_id": customer_id,
        "note": note,
        "ts": time.time(),
    })
    # update read model synchronously so agent sees its own write
    await pg_write.execute(
        "UPDATE customer_context SET ctx = jsonb_set(ctx, '{notes}', ctx->'notes' || $2) WHERE id = $1",
        customer_id, json.dumps([{"id": event_id, "text": note}]),
    )
    return {"status": "queued", "event_id": event_id}

That's ~30 lines. It gives you sub-second agent turns against a legacy CRM that takes 3 seconds to acknowledge a write. The reconciler catches up in the background.

MCP as the connective tissue (and why it matters here)

The direct answer: Model Context Protocol standardizes how agents call tools, which lets you swap the implementation behind a tool from "direct API call" to "read-model + event bus" without changing the agent's prompt or reasoning loop. That's the piece that makes the wrap-your-legacy strategy actually work in practice.

Before MCP-shaped tool interfaces, every backend change meant re-prompting, re-testing, and often re-fine-tuning. The agent knew too much about its tools. Now the contract is: the tool takes a JSON schema in, returns a JSON schema out. Whether the tool is a 40 ms Redis read or a 4 second SOAP call is invisible to the model.

This matters because you almost never migrate a legacy system in one shot. You migrate one endpoint at a time. MCP lets you do that without the agent noticing:

{
  "tool": "get_customer_context",
  "input_schema": { "customer_id": "string" },
  "output_schema": {
    "customer_id": "string",
    "recent_orders": "array",
    "notes": "array",
    "loyalty_tier": "string"
  }
}

Week 1, that tool hits the legacy CRM directly and takes 3 seconds. Week 2, you materialize a read model and the same tool takes 30 ms. The agent's behavior improves without a single prompt change. That's the leverage.

The Transform panel didn't spend much time on MCP by name — LinkedIn and Walmart both have internal equivalents that predate it — but the shape is identical. Standardize the contract, then optimize behind it.

Where solo builders and SMBs get this wrong

The direct answer: they either wire the agent directly into the raw backend and ship something painfully slow, or they over-engineer the read-model layer before knowing which fields the agent actually uses. Both are avoidable if you profile first and materialize second.

The common failure modes I see on client audits:

• Wiring the agent to a Rails ORM in production. The ORM was fine for page renders. The agent calls it 8 times per turn and hits N+1 queries every time. p95 goes to 12 seconds.
• Using a vector DB as a substitute for a read model. RAG doesn't fix latency on transactional lookups. If the agent needs "current order status," it needs a fresh projection, not a semantic search.
• One giant MCP tool that returns everything. The model doesn't need the customer's entire history on every turn. Split the tool. Let the agent ask for what it needs.
• No caching layer between the agent and anything. Even a 60-second Redis cache in front of a slow endpoint often cuts p95 in half, because agents re-fetch the same context across steps of the same task.
• Synchronous writes to slow systems. If the write takes 4 seconds and you're awaiting it in the tool call, you've just made every agent turn 4 seconds long. Publish an event, return optimistic ack, reconcile in background.

The fix is nearly always the same: profile one agent turn end-to-end, find the two or three tools that dominate p95, and put a projection or a cache in front of them. You don't need Kafka on day one. A Postgres table refreshed by a cron job is a valid read model.

A concrete migration path you can start this week

The direct answer: pick one agent workflow, instrument it, materialize the top two slow reads, move one write to async, and measure again. That's a one-week sprint, not a quarterly project.

Here's the sequence I use on audits:

1. Instrument the agent turn. Log every tool call with duration. One day of real traffic tells you where the time actually goes. Assumptions about where it goes are wrong roughly half the time.
2. Rank tools by contribution to p95. Usually two or three tools cause 80% of the pain. Ignore the rest for now.
3. Materialize the top slow read. Build one denormalized table with exactly the fields the agent uses. Populate it from CDC (Debezium, Postgres logical replication, or a simple polling job if volume is low).
4. Point the MCP tool at the projection. Keep the tool contract identical. The agent notices nothing except that it's faster.
5. Move one write to async. Pick the write that's blocking the longest. Publish an event, update the read model optimistically, reconcile in background with a worker that can retry.
6. Re-measure. If p95 didn't drop meaningfully, you materialized the wrong thing. Go back to step 2.

A useful outbound reference here: Martin Kleppmann's writing on turning the database inside out is still the clearest explanation of why event streams + materialized views beat direct-to-legacy reads for latency-sensitive systems. It's a decade old and still correct.

What "good" looks like when you're done

The direct answer: p95 agent turn under 1.5 seconds, no direct reads from systems of record during an agent turn, and writes that return optimistic acks within 100 ms. If you hit those three, you're inside the same performance envelope LinkedIn and Walmart described at Transform — at a fraction of the scale, but with the same architecture.

A rough checklist:

• Every agent tool has an MCP-shaped contract independent of its implementation
• No agent tool call blocks on a system with p95 > 500 ms
• Every write path has a durable event log the agent publishes to
• The read model is rebuildable from the event log (this is your escape hatch when it gets corrupted, and it will)
• You have one dashboard showing agent turn p50 / p95 / p99, broken down by tool
• You have a kill switch that can disable any tool without breaking the agent (the model should degrade gracefully, not crash)

Miss any of these and you have a demo, not a production agent.

How BizFlowAI approaches this

Most of our client work starts with exactly the audit above: instrument one agent workflow, find the two or three tools that dominate p95, and design the read models and event pipelines that pull the agent turn back under 1.5 seconds. We build the MCP tool layer, the reconcilers, and the observability so the model can keep improving without the plumbing regressing.

We don't rip out legacy systems. We wrap them — because that's what actually ships inside a small team's budget and calendar. If your agent works in a demo but times out in production, or if you're staring at a 6-second p95 and don't know which tool call to blame, an audit is the fastest way to find out. Book a discovery call and we'll walk through your traces together.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.