What Expedia learned from billions of AI predictions

By Lazar Milicevic · Published July 8, 2026 · 9 min read

Your agent works. It handled the last 200 tickets, drafted the invoices, replied to leads. Then on a Tuesday it starts hallucinating shipping addresses, and you find out three days later because a customer complained on X. This is the gap between AI that works today and AI that survives contact with real traffic — and it's the gap most solo builders and small teams fall into after their first working prototype.

Expedia has been shipping ML into production since long before "AI agent" was a phrase anyone used. They serve billions of predictions across search ranking, pricing, fraud detection, and personalization. What their engineering org has been saying publicly for years — long before the current agent hype cycle — is that the hardest part of ML in production isn't getting a model to work once. It's building systems that keep working when the world shifts underneath them.

That lesson didn't age. It got sharper. Agents amplify every discipline gap a static model exposed.

Working once vs. working at scale — the actual difference

A model that "works" is one that produced a reasonable answer on your test inputs. A system that lasts at scale keeps producing reasonable answers as inputs drift, dependencies change, edge cases stack up, and the cost per call adds up to a real bill.

The prototype path looks like this: prompt an LLM, wire it to Gmail or Stripe, get a green light on ten test cases, ship. Six weeks later the failure modes look like:

• The model started returning JSON with a trailing comma because a vendor updated the underlying weights.
• A single customer with an unusual email signature blows up your parser.
• Token costs tripled because someone added a long system prompt and nobody noticed.
• You have no idea which conversations went sideways because nothing was logged.

Expedia's playbook — public in their engineering blog and conference talks — treats each of these as a first-class problem. Data quality, feature drift, model monitoring, and rollback procedures get roughly the same engineering weight as the model itself. That's the discipline gap. Small teams skip it because it feels like overhead. Then it becomes the reason the automation gets ripped out.

Evals are the deployment gate, not a nice-to-have

An eval is a repeatable test that scores your agent's output against known-correct behavior. If you don't have one, you can't ship a prompt change without gambling.

For a solo operator, this doesn't need to be a research-grade eval framework. It needs to be a file of real examples with the expected outcome, and a script that runs the agent against them and diffs the result.

# evals/invoice_extractor.py
import json
from pathlib import Path
from my_agent import extract_invoice

CASES = json.loads(Path("evals/cases.json").read_text())

def score(expected, actual):
    fields = ["vendor", "total", "due_date", "invoice_number"]
    hits = sum(1 for f in fields if str(expected[f]) == str(actual.get(f)))
    return hits / len(fields)

results = []
for case in CASES:
    actual = extract_invoice(case["input_path"])
    results.append({
        "id": case["id"],
        "score": score(case["expected"], actual),
        "actual": actual,
    })

avg = sum(r["score"] for r in results) / len(results)
print(f"Avg accuracy: {avg:.2%}")
print(f"Regressions: {[r['id'] for r in results if r['score'] < 1.0]}")

Start with 20 real cases from your own data. Add every bug report as a new case. When accuracy on the eval set drops below your threshold, you don't ship. That single rule saves more agent deployments than any prompt engineering trick.

The Expedia principle underneath: you cannot improve what you cannot measure, and you cannot ship safely what you have not measured against a baseline.

Observability: know what your agent did, on which input, at what cost

Observability for an agent means every call is logged with its input, output, model version, latency, and token cost — and you can search those logs by customer, error type, or time window in under a minute.

If a customer calls tomorrow saying "your bot told me the wrong price on Monday," what's your answer? For most solo builders, the honest answer is "I have no idea what it told them." That's not sustainable past ten users.

Minimum viable logging schema:

{
  "trace_id": "uuid",
  "timestamp": "2026-07-08T14:22:11Z",
  "customer_id": "cust_8821",
  "agent": "invoice_parser_v3",
  "model": "claude-sonnet-4.5",
  "prompt_version": "sha256:a3f...",
  "input_tokens": 1842,
  "output_tokens": 311,
  "latency_ms": 2140,
  "tool_calls": ["fetch_pdf", "extract_lineitems"],
  "output": {...},
  "error": null
}

Ship this to any queryable store — Postgres works fine at small scale, ClickHouse or BigQuery when volumes grow. The rule is: before you ship an agent, you must be able to answer "what did the agent do for customer X yesterday" in one query.

Google's SRE guidance on the four golden signals — latency, traffic, errors, saturation — maps cleanly to agents. Add token cost and eval score to that list. That's your dashboard.

Drift is not a future problem

Drift is when the distribution of your inputs, or the behavior of your model, changes over time — so an agent that had 95% accuracy last month has 78% this month and nobody noticed.

Three kinds of drift will hit a small-business agent:

The Expedia-scale version of this involves feature stores and automated retraining pipelines. Your version is a weekly cron that reruns your eval set, plus a Slack alert when scores drop more than 5 points. That's it. It costs an afternoon to set up and it's the difference between finding a regression on Monday morning versus finding it after three weeks of bad outputs went to customers.

Integrations are where agents actually die

Most agent failures in production aren't model failures — they're integration failures. The API changed, the auth token expired, the schema added a required field, the rate limit tightened.

A model that generates a beautiful email draft is useless if the send call fails silently. This is why serious teams are moving toward MCP (Model Context Protocol) — a standard way for agents to talk to tools with typed schemas, versioning, and predictable error surfaces. Instead of every integration being a bespoke wrapper around a REST API that breaks when Stripe changes a field, MCP servers give you a contract.

The practical rule for small teams:

• Every external call is wrapped with retries, timeouts, and structured error logging.
• Every integration has a health check that runs independently of the agent.
• Every credential has an expiry alert set 14 days before it dies.
• Every schema change on your own side is versioned so old traces stay parseable.

None of this is glamorous. All of it is what separates an agent that survives six months from one that gets quietly turned off after two.

Rollback is a feature, not an emergency

If you can't roll back a prompt, a model version, or an integration change in under five minutes, you don't actually control your production system.

Treat prompts like code. Every prompt lives in version control with a semantic hash. Every deployment records which prompt version is live. When something goes wrong at 11pm, the answer isn't "let me figure out what I changed" — it's git revert and redeploy.

# config/agents/invoice_parser.yaml
version: 3.2.1
model: claude-sonnet-4.5
prompt_ref: prompts/invoice_parser_v3.md
prompt_sha: a3f9c2e8b1d4
temperature: 0
tools:
  - fetch_pdf@1.2
  - extract_lineitems@2.0
eval_threshold: 0.92
rollback_to: 3.2.0

The rollback_to field is not optional. Ship it with your first agent. You will need it.

Expedia's teams talk about this as "safe deploys" — canary rollouts, shadow traffic, and one-click rollback. Your solo version is: deploy to 10% of traffic for an hour, watch the eval score and error rate, then promote. Every serious production system, at every scale, has some version of this loop. Skipping it is how you find out about a bad deploy from an angry customer instead of a dashboard.

Cost discipline is a scaling issue in disguise

Token cost per action is the number that determines whether your agent is a business or a hobby. Track it per customer, per workflow, per day — from day one.

The math that kills unprofitable agent deployments looks like this: your prototype cost $0.04 per invoice to process. You didn't notice because you were testing with 50 invoices a month. Six months in, you're at 12,000 invoices a month, someone added retrieval-augmented context that tripled the prompt size, and you're spending $1,800/month to save a client $400 of admin time.

Cost observability is the same as behavioral observability — same logging, one extra field. Then you can answer:

• Which customer is the most expensive to serve?
• Which workflow is dragging down margin?
• Did that "small" prompt tweak add 30% to inference cost?

A working rule: any prompt change that increases average token count by more than 15% needs the same review as a code change. Otherwise you'll discover the cost only when the bill lands.

What the "agent" reframing actually changes

Everything above applies to a single-shot LLM call. Agents — models that plan, call tools, and iterate — make each discipline harder, not easier.

• Evals now have to cover multi-step trajectories, not just final outputs. A right answer reached through five wrong tool calls is still a bug.
• Observability has to trace the whole chain. Which tool call failed? Which reasoning step went off the rails?
• Cost is non-deterministic. A stuck loop can 10x the token bill for a single interaction. Set hard step and budget limits.
• Rollback might need to include tool schemas, not just the prompt.

None of this is new. It's the same discipline the Expedia engineering org has been documenting for a decade, applied to a system with more moving parts. The teams that ship durable agents in 2026 are the ones treating agents like production systems, not like ChatGPT wrappers.

How BizFlowAI approaches this

When we build agents for clients, the first week isn't prompt engineering — it's the eval harness, the logging schema, and the rollback path. We build MCP-based integrations so tool calls have typed contracts instead of brittle REST wrappers, and every deploy runs against a real-data eval set before it touches production traffic. Boring on purpose. It's the reason our client agents are still running six and twelve months later instead of being quietly turned off.

If you have an automation that works in a demo but you're not sure it will survive real customer traffic — or one that's already in production and you can't confidently answer "what did it do yesterday" — that's exactly the conversation we have on a discovery call. We'll pressure-test the roadmap against the failure modes above and tell you honestly what's ready to scale and what needs a rework.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.