The cleanup trap: Stop asking RAG to fix bad data

By Lazar Milicevic · Published July 20, 2026 · 9 min read

Your RAG pilot works fine on the 20 hand-picked PDFs in the demo. Then you point it at the real SharePoint drive — 40,000 documents, half of them scanned, three overlapping versions of the same policy, PDFs with headers repeated on every page — and the answers turn into confident nonsense. The instinct is to swap models, widen the context, or tune the prompt. That's the trap.

Retrieval augmented generation is a lookup layer. It doesn't clean data. It doesn't reconcile contradictions. It doesn't decide which of the three "final_v2_APPROVED.pdf" versions is actually current. If you hand it messy ingestion, it will retrieve messy chunks and the model will happily blend them into a plausible-sounding wrong answer. The fix is upstream of the vector store, not downstream of the LLM.

This post is about what that upstream work actually looks like — the pipeline, the choices, the boring parts that decide whether your agent survives contact with production.

Why RAG can't rescue bad source data

RAG is three steps: chunk, embed, retrieve. None of those steps understand your business. A chunker doesn't know that page 4 of a contract overrides page 2. An embedding model doesn't know that "Client Onboarding v3" supersedes "Client Onboarding v2." A vector search returns the top-k most semantically similar passages — which, if your corpus has near-duplicates, means it returns the same wrong thing three times and the model treats that as consensus.

The failure mode people actually see in production:

• Contradiction blending. Two documents disagree. Retrieval surfaces both. The model averages them into a hallucinated middle position.
• Stale wins. Old drafts embed just as well as current ones. Without metadata filters, recency is random.
• Scanned-PDF rot. OCR errors turn "Net 30" into "Nel 3O" and now that clause is invisible to search.
• Boilerplate dominance. Legal footers repeat in every chunk, and cosine similarity loves repetition.

You can't prompt your way out of this. You can't fine-tune your way out of it either — the model will just learn to hallucinate more confidently. The only fix is to feed the retriever a clean, deduplicated, versioned, well-structured corpus.

The ingestion pipeline that actually works

Before you touch a vector database, you need a document pipeline with clear stages. Skip any stage and you'll pay for it later in support tickets from users saying "the AI told me the wrong policy."

Here's the minimum viable pipeline for a serious RAG system:

raw_source → parse → normalize → deduplicate → 
enrich_metadata → chunk → embed → index

Each stage is a checkpoint. Each stage produces something you can inspect and test independently. If your pipeline is a single script that goes from S3 to Pinecone in 200 lines, you have no visibility when things break.

A concrete example — ingesting a folder of mixed PDFs, DOCX, and HTML:

from pathlib import Path
import hashlib

def ingest(source_path: Path):
    for doc in source_path.rglob("*"):
        raw = load(doc)                    # parser per file type
        text = normalize(raw)              # unicode, whitespace, OCR fix
        doc_hash = hashlib.sha256(text.encode()).hexdigest()
        
        if already_indexed(doc_hash):
            continue                       # skip byte-identical dupes
        
        meta = extract_metadata(doc, text) # author, date, version, source
        if is_superseded(meta):
            mark_archived(doc_hash)        # keep but exclude from search
            continue
        
        chunks = chunk_semantic(text, meta)
        vectors = embed(chunks)
        index(vectors, chunks, meta)

Boring code. That's the point. Every ingestion decision is explicit, logged, and reversible.

Parsing and normalization: where 60% of the problem lives

Most RAG failures I get called in to debug trace back to garbage coming out of the parser. PDFs are the worst offender. A PDF isn't a document — it's a set of drawing instructions for a printer. Text extraction is guesswork.

Rules I follow:

Use the right parser per file type. Don't send everything through one library. For born-digital PDFs, pdfplumber or pymupdf are fine. For scanned PDFs, you need OCR — Tesseract works for clean scans, but for real-world mixed quality you want a service that handles layout: AWS Textract, Google Document AI, or Azure Document Intelligence. For DOCX, go straight to the XML rather than converting through PDF.

Detect scanned pages before you process. A common mistake is running text extraction on a scanned PDF and getting back an empty string, then indexing that empty string as a "document." Check text-per-page: if a page has fewer than ~50 characters and contains an image, it's scanned. Route it to OCR.

Normalize aggressively. Collapse whitespace. Fix unicode (curly quotes, non-breaking spaces, ligatures like "fi" that break search). Strip page numbers, running headers, and footers — they are noise that pollutes every chunk. This alone often improves retrieval precision noticeably.

Preserve structure. Headings, lists, tables — these carry meaning. If your parser flattens a table into a wall of comma-separated numbers, you've destroyed the information. Extract tables as tables (markdown or HTML), then either embed them separately or convert them to natural-language summaries before chunking.

Deduplication and versioning: the invisible killer

In any real corpus older than a year, 20–40% of documents are near-duplicates. Draft 1, draft 2, "final," "final final," the same policy re-uploaded to three shared drives. If you index all of them, retrieval quality collapses because your top-k results are the same document five times.

Two levels of dedup:

Exact dedup — hash the normalized text. Skip byte-identical content. Cheap, catches 30% of the mess.

Near-dedup — use MinHash or SimHash on shingled text. Cluster documents with Jaccard similarity above some threshold (0.85 is a reasonable starting point). Within each cluster, pick a canonical version using metadata: most recent modification date, highest version number, most authoritative source folder.

Versioning is the other half. Every chunk should carry:

{
  "doc_id": "policy_onboarding_2026-03",
  "supersedes": "policy_onboarding_2025-11",
  "effective_date": "2026-03-15",
  "status": "current",
  "source": "sharepoint://legal/policies/",
  "authority": "legal_team"
}

Then your retrieval filters on status = "current" by default. Old versions stay in the index for audit and historical questions, but they don't show up in normal answers. This one change fixes more "the AI gave me outdated info" complaints than any prompt tweak.

Chunking: stop using fixed 512-token windows

The default advice — "chunk your documents into 512-token pieces with 50-token overlap" — is a decent starting point for a demo and terrible for production. Real documents have structure. Cutting a contract in the middle of a clause, or splitting a table from its header row, destroys the meaning that made the passage retrievable in the first place.

Better approaches, in rough order of increasing effort and payoff:

For most business document corpora, a hybrid works: split by heading first, then apply size limits within sections, and treat tables and lists as atomic units that never get split. Prepend the section heading to each chunk so the embedding captures context ("Under section 4.2 — Termination — the notice period is…").

Hybrid search: dense vectors alone are not enough

Pure vector search is bad at three things: exact terms, rare terms, and short queries. If someone asks "what's our SLA for tier-3 tickets," a dense retriever might return semantically similar passages about "support response commitments for premium customers" — close, but not the exact policy. Meanwhile it misses the document that literally says "tier-3 SLA: 4 hours" because the surrounding prose is generic.

Hybrid search fixes this. Run BM25 (lexical) and dense (semantic) retrieval in parallel, then combine with Reciprocal Rank Fusion or a learned reranker. A simple version:

def hybrid_retrieve(query: str, k: int = 20):
    bm25_hits = bm25_index.search(query, k=k)
    dense_hits = vector_index.search(embed(query), k=k)
    
    fused = reciprocal_rank_fusion(bm25_hits, dense_hits)
    
    # rerank top candidates with a cross-encoder
    reranked = cross_encoder.rerank(query, fused[:k*2])
    return reranked[:k]

Add a cross-encoder reranker (like a BGE or Cohere rerank model) on the top ~40 candidates and precision jumps meaningfully. Rerankers are slower per document but you're only running them on a small candidate set, so latency stays reasonable.

Also, use metadata filters as first-class citizens. If the user asks about "the 2026 pricing policy," parse that intent and filter to effective_date >= 2026-01-01 before you even embed the query. Filters cut candidate space before the expensive similarity math happens.

Evaluation: without it, you're guessing

The other thing that separates production RAG from stalled pilots: a real eval harness. Not vibes-testing in a chat window. A held-out set of 100–300 real questions with known-good answers, run automatically on every change to the pipeline.

Minimum eval metrics:

• Retrieval recall@k — for each question, is the correct source document in the top-k retrieved chunks? If it isn't, no model can save you.
• Answer faithfulness — does the generated answer only make claims that are actually in the retrieved context? Use an LLM-as-judge with a strict rubric.
• Answer completeness — does it answer the actual question, or dodge it?
• Citation accuracy — do the cited passages actually support the claim?

Build these before you tune anything. Every change — new chunk size, new embedding model, new reranker — gets scored against the same eval set. Without this you're doing superstition-driven development.

Where to source questions: the first two weeks of real user queries. Not synthetic ones you made up. Real questions expose the gaps in your corpus and the ambiguities in your data model.

How BizFlowAI approaches this

We get called in when a RAG or agent pilot works in the demo and falls apart in production. The pattern is almost always the same: the model is fine, the vector database is fine, the ingestion pipeline is a 100-line script that treats every PDF like clean prose. We rebuild ingestion first — parser routing per file type, OCR for scanned pages, aggressive normalization, exact and near-duplicate detection, version resolution with metadata, and structural chunking that respects headings and tables.

Then we layer hybrid search (BM25 + dense + reranker), metadata filtering, and an eval harness tied to real user questions so every future change is measurable. The result is agents that give the same answer twice, cite sources that actually contain the claim, and don't quietly regress when someone uploads a new folder. If you're stuck on the "works in demo, fails in prod" step, book a discovery call and we'll walk through your current pipeline.

What to do this week

If you have a stalled RAG project, don't touch the model. Do this instead:

1. Sample 50 real user questions and their current answers. Score them honestly. This tells you what's actually broken.
2. Audit your parser output. Pick 20 documents at random, look at what text your pipeline extracted. Count the ones where structure was lost, OCR failed, or content is missing.
3. Run dedup on your corpus. Count how many near-duplicates you have. If it's over 15%, that's your first fix.
4. Add metadata to your chunks. At minimum: source, date, version, status. Filter on status in every query.
5. Add BM25 alongside your vector search and fuse the results. This is a one-day change that meaningfully improves precision.
6. Build the eval harness. 100 questions. Run it on every change.

None of this is glamorous. None of it involves a new model or a bigger context window. That's exactly why it works — because the problem was never the model. It was the data going in.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.