AI Agents Drift Because Your Data Pipeline Rots

By Lazar Milicevic · Published July 23, 2026 · 10 min read

You shipped a support agent in March. Accuracy was 94% on your eval set, the CEO demoed it to the board, and Slack was full of celebration emojis. It's now July, ticket deflection has quietly slid from 61% to 38%, and the last three escalations were the bot confidently quoting a refund window that changed in April, a plan tier that was renamed in May, and an integration that was deprecated last week. Nobody touched the prompt. Nobody swapped the model. The bot didn't get worse — the world moved, and your knowledge base stayed frozen in March.

This is the failure mode nobody warned you about. Prompt engineering gets the headlines, RAG gets the tutorials, and evals get the conference talks. Meanwhile, the actual thing that kills agents in production is a data pipeline that quietly rots because no one owns freshness.

The real failure mode is staleness, not hallucination

When a shipped agent starts being confidently wrong, the instinct is to blame the model. It's almost never the model. In post-mortems on agent regressions, the pattern is depressingly consistent: the LLM faithfully retrieved and summarized a chunk of source material that was correct in Q1 and wrong in Q3.

That is not a hallucination. A hallucination is the model inventing something not in context. This is the opposite — the model is doing exactly what you asked, grounded in retrieved context, and the context itself is a lie. Which is worse, because every guardrail you built (citations, "answer only from sources," retrieval confidence thresholds) will happily approve the response. The citation link even works. It just points to a document that stopped being true.

Once you internalize this, you stop treating a RAG system as an ML problem and start treating it as a data engineering problem. The model is a rendering layer over a knowledge store. If the store is stale, no amount of model tuning saves you.

Why "we'll re-index weekly" doesn't work

The naive fix is a cron job that re-ingests everything on a schedule. This fails for three reasons, and they compound.

First, most knowledge bases are heterogeneous. You have Notion pages, a Zendesk help center, a couple of Google Docs someone insists on maintaining, PDF policy documents from Legal, and a Postgres table of live product SKUs. A weekly full re-index treats all of these as equally volatile. Product pricing changes daily; the ISO 27001 policy changes once a year. If you re-embed everything on the same cadence, you either burn money on embedding calls or you're always a week behind on the things that actually move.

Second, "re-indexing" doesn't detect deletions or renames. A page that was archived on Monday will keep serving stale chunks until you diff the source of truth against the vector store. Most teams don't. They just append.

Third, and worst: re-embedding preserves stale chunks that no longer have a source. Someone deletes a Notion page about the legacy pricing tier. The next re-index doesn't create that chunk again, but the old chunk with the old embedding is still sitting in Pinecone/Qdrant/pgvector because nothing garbage-collected it. Your retriever will happily return it for the next relevant query.

A freshness-aware ingestion pipeline

Here is the mental model that works. Every chunk in your vector store has three timestamps and a source fingerprint:

{
  "chunk_id": "notion:page_abc123:section_4",
  "content": "Pro plan includes 50,000 API calls/month...",
  "embedding": [...],
  "source_uri": "notion://workspace/page_abc123",
  "source_hash": "sha256:9f2c...",
  "source_updated_at": "2026-06-14T08:22:00Z",
  "indexed_at": "2026-06-14T09:00:00Z",
  "last_verified_at": "2026-07-21T02:00:00Z",
  "ttl_class": "pricing"
}

Four things this buys you:

• source_hash lets you detect real changes cheaply. Poll the source, hash the current content, and only re-embed if the hash changed. Embedding APIs are not free at scale; hashing is.
• source_updated_at vs. indexed_at tells you when you're behind.
• last_verified_at tells you the last time you confirmed the source still exists. This is how you catch deletions.
• ttl_class encodes volatility. Pricing might be 24h; a security policy might be 90d.

The ingestion job then looks like this:

def sync_source(source):
    live_docs = fetch_live_documents(source)
    live_ids = {d.uri for d in live_docs}
    stored_ids = vector_store.list_uris(source=source)

    # 1. Detect deletions
    for uri in stored_ids - live_ids:
        vector_store.delete_by_uri(uri)
        log.info("purged", uri=uri)

    # 2. Detect changes and new docs
    for doc in live_docs:
        stored = vector_store.get_metadata(doc.uri)
        if stored and stored.source_hash == doc.hash:
            vector_store.touch(doc.uri, last_verified_at=now())
            continue
        chunks = chunk(doc)
        embeddings = embed(chunks)
        vector_store.upsert(doc.uri, chunks, embeddings,
                            source_hash=doc.hash,
                            source_updated_at=doc.updated_at)

Two loops, no magic. It handles adds, updates, deletes, and it doesn't waste embedding tokens on unchanged content. This is not sophisticated engineering. It is the boring pipeline work nobody wants to do because RAG demos don't require it.

Match refresh cadence to content volatility

Not all knowledge decays at the same rate. Here is a cadence table that maps to how most B2B knowledge bases actually behave. Adjust for your business, but the shape holds.

Two rules of thumb that save real money:

1. Prefer webhooks over polling where the source supports them. Notion, Zendesk, Intercom, GitHub, Stripe all fire events on content changes. A webhook-driven pipeline is both fresher and cheaper than any polling schedule.
2. Separate the "check if changed" step from the "re-embed" step. Hashing a Notion page is basically free. Embedding it costs tokens. If 90% of your polls result in no change (they will), your embedding bill drops by ~10x versus a naive daily re-embed.

Freshness checks the agent runs at query time

Ingestion freshness handles the write side. You also need a read-side check, because ingestion will always have some lag and some sources you don't fully control.

The cheapest and most useful pattern is a freshness gate in the retrieval step. Every retrieved chunk carries its metadata. Before you hand the chunks to the LLM, you evaluate them:

def freshness_gate(chunks, query_intent):
    max_age = TTL_BY_CLASS.get(query_intent, timedelta(days=30))
    fresh, stale = [], []
    for c in chunks:
        age = now() - c.source_updated_at
        (fresh if age < max_age else stale).append(c)

    if not fresh and stale:
        # Trigger targeted re-sync, then retry
        for c in stale:
            enqueue_priority_resync(c.source_uri)
        return {"status": "stale", "chunks": stale, "warn_user": True}

    return {"status": "ok", "chunks": fresh}

Three behaviors this unlocks:

• If a user asks about pricing and the only retrieved chunk is 4 days old, you queue an immediate re-sync of the pricing source and either wait 2 seconds for it, or answer with an explicit caveat: "Based on our pricing page as of [date] — I'm refreshing it now, ask again in a moment for the latest."
• If a query intent maps to a volatile class (pricing, availability, current promotions), you can force a live source fetch and bypass the vector store entirely. Sometimes the right answer for "what does it cost right now" is not RAG at all — it's a direct API call to the source of truth.
• Your logs now contain a metric that matters: stale retrieval rate. This is the single most useful health signal for a shipped agent. Track it.

The evals nobody runs: temporal regression tests

Standard evals check whether the agent answers correctly today. They tell you nothing about drift. A temporal eval suite checks whether the agent still answers correctly, and knows when it doesn't.

Structure a temporal eval like this:

- id: pricing_pro_plan_limits
  question: "What's included in the Pro plan?"
  ground_truth_source: "https://acme.com/pricing"
  ground_truth_refresh: daily
  assertions:
    - retrieved_chunks_max_age_hours: 24
    - answer_contains_current_price: true
    - answer_cites_source_uri: "acme.com/pricing"
  volatility_class: pricing

- id: refund_policy_window
  question: "How many days do I have to request a refund?"
  ground_truth_source: "legal/refund_policy.pdf"
  ground_truth_refresh: weekly
  assertions:
    - retrieved_chunks_max_age_days: 14
    - answer_matches_current_policy: true
  volatility_class: policy

Run this suite nightly. The pass rate should be a dashboard everyone can see. When it drops, you know exactly which source rotted and which agent behavior is at risk, before a customer finds out.

Two things to build into the suite:

• Auto-refreshed ground truth. The ground truth for the pricing eval should itself be scraped/pulled from the pricing page nightly, not hardcoded. Otherwise your evals rot alongside your agent.
• A "known-changed" set. Whenever a source changes (you get a webhook), add that change to a queue of eval cases that specifically test whether the agent picked it up within your SLA.

What "confidently wrong" looks like in the logs

If you're reading this and thinking "I don't know if my agent has this problem," here are the signals to look for in production logs:

• Rising thumbs-down rate on answers with high retrieval confidence. The retriever is finding chunks, the model is answering with certainty, and users are marking it wrong. Classic stale-source signature.
• Escalations clustered around specific topics. If 70% of your human escalations touch pricing, integrations, or a specific policy area, that's not a training problem. That's one specific source that rotted.
• Widening gap between source_updated_at and last_verified_at in the vector store. Query your metadata. If the median age of retrieved chunks in the last 7 days is 45+ days, and your business changes faster than that, you have drift.
• Support tickets that reference the bot by name. "The bot told me X but your website says Y" is the tell.

None of these show up in a pre-launch eval. All of them show up in month three.

How BizFlowAI approaches this

Most of what we ship for clients isn't the agent — it's the pipeline underneath it. When we build a RAG or document workflow, we set up webhook-driven ingestion where the source supports it, hash-based change detection to keep embedding costs down, TTL classes per source so pricing gets checked hourly and legal gets checked weekly, and read-side freshness gates so the agent can either refresh a stale source on demand or caveat its answer honestly. Every chunk carries the metadata described above, and every deployment includes a nightly temporal eval suite the client can actually read.

The result is agents that hold accuracy months after launch instead of degrading quietly. If you have a chatbot, internal knowledge agent, or document workflow that felt sharp at launch and doesn't anymore, book a discovery call — we'll look at your pipeline and tell you honestly whether the fix is a two-week ingestion rebuild or something bigger.

The one-page checklist

If you only implement six things, implement these:

1. Every chunk in your vector store has source_uri, source_hash, source_updated_at, last_verified_at, and a ttl_class.
2. Ingestion is webhook-driven where possible, hash-diffed everywhere, and it deletes chunks whose source no longer exists.
3. Refresh cadence is set per source based on how fast that source actually changes, not on a uniform weekly cron.
4. Retrieval includes a freshness gate that can trigger on-demand resyncs and warn users when only stale chunks are available.
5. You have a temporal eval suite with auto-refreshed ground truth that runs nightly, and stale retrieval rate is on a dashboard.
6. For the most volatile classes (live pricing, availability, inventory), the agent bypasses RAG and hits the source of truth directly.

The reason agents go bad in production isn't that LLMs are unreliable — they are, but that's a solved-enough problem. It's that teams treat the knowledge store as a one-time load instead of a living system with a maintenance cost. Budget the maintenance up front and the agent stays useful. Skip it and you'll be re-launching the same bot every six months, wondering why nothing sticks.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.