Raw OCR Text Broke My Invoice Parser — 26% Error Rate

By Lazar Milicevic · Published July 5, 2026 · 8 min read

I fed Claude raw OCR text from 180 invoices and it got 29% of every field wrong. Tax amounts mistaken for discounts, line items collapsed into a single garbage row, invoice numbers read as zip codes. If you're dumping OCR output straight into a prompt and hoping the model figures it out, this is exactly why your extraction fails.

The fix isn't a better prompt. It's a better data representation — and it takes about 15 lines of Python.

The Problem: OCR Gives You Characters, Not Structure

OCR engines like Tesseract, AWS Textract, and Google Document AI are excellent at reading characters from a page. What they're terrible at is preserving the spatial relationships between those characters. An invoice is fundamentally a table — amounts live in columns, labels sit to the left, tax is separate from discounts. OCR flattens all of that into a single stream of text where the only structural signal is whitespace and newlines.

Here's what raw Tesseract output looks like from a real supplier invoice:

INVOICE #2024-0847
Date: March 15, 2024
Bill To: Acme Construction LLC
                          Qty    Unit Price     Tax     Total
Widget A Premium           12       $45.00      10%    $594.00
Widget B Standard           8       $22.50       8%    $194.40
                          Subtotal                       $788.40
                          Tax (10%)                       $78.84
                          Discount                         -$50.00
                          TOTAL                          $817.24

That looks readable to a human. But here's what happens after Tesseract processes a scanned or photographed version of the same invoice — the way it arrives in production:

INVOICE #2024-0847 Date: March 15, 2024 Bill To: Acme Construction
LLC Qty Unit Price Tax Total Widget A Premium 12 $45.00 10% $594.00
Widget B Standard 8 $22.50 8% $194.40 Subtotal $788.40 Tax (10%)
$78.84 Discount -$50.00 TOTAL $817.24

The column alignment is gone. The whitespace that told you "10% is in the tax column" is collapsed. "Tax (10%)" and "$78.84" are now adjacent to "Discount" and "-$50.00" with no delimiters between them.

When you feed this to Claude and ask it to extract the tax amount, it sees "Tax (10%) $78.84 Discount -$50.00" as a continuous string. Depending on whitespace, it may return 10% as the tax amount, or -$50.00, or merge the values. It's not hallucinating — it's making a reasonable guess from ambiguous input.

What breaks specifically

• Tax vs. discount confusion — both are adjustments to subtotal, but they live in different conceptual columns
• Line item merging — without row delimiters, Claude can't tell where one product ends and the next begins
• Percentage vs. amount — "10%" next to "$78.84" with no column headers is genuinely ambiguous
• Header vs. data — "Qty Unit Price Tax Total" reads as data, not as column definitions

The Numbers: 71% vs 97.8% Field Accuracy

I tested this across 180 real invoices in production. Same OCR engine (Tesseract), same model (Claude), same prompt. The only variable was the data format I sent to Claude.

That's a 26.8-point accuracy swing. The remaining 2.2% error rate came from genuinely illegible scans — water damage on physical invoices, creases across barcode areas, and one supplier who prints invoices on patterned background stock.

The 20 remaining errors broke down like this:

• 11 invoice number errors (OCR misread "0" as "O" or "1" as "l")
• 5 supplier name errors (abbreviations expanded incorrectly)
• 3 line item description errors (product codes confused with descriptions)
• 1 total amount error (comma vs. period in European number format)

None of these are structural failures. They're OCR character recognition errors — a different problem that requires a different fix (confidence scoring and human review for low-confidence fields).

The Fix: Restructure OCR Output as Markdown Tables

The solution is to take the raw OCR output and restructure it into a format that preserves the table semantics before it ever reaches Claude. Markdown tables are ideal because Claude processes them natively and they're trivial to generate in Python.

Here's the same invoice data, restructured:

## Invoice Header
| Field | Value |
|---|---|
| Invoice Number | 2024-0847 |
| Date | March 15, 2024 |
| Bill To | Acme Construction LLC |

## Line Items
| Description | Qty | Unit Price | Tax Rate | Line Total |
|---|---|---|---|---|
| Widget A Premium | 12 | $45.00 | 10% | $594.00 |
| Widget B Standard | 8 | $22.50 | 8% | $194.40 |

## Totals
| Field | Amount |
|---|---|
| Subtotal | $788.40 |
| Tax (10%) | $78.84 |
| Discount | -$50.00 |
| TOTAL | $817.24 |

Same characters, same numbers, same labels. But now the tax amount is unambiguously in a "Tax" row under a "Totals" header. The line items are separated by pipe delimiters. The column headers tell Claude what each value represents.

The Python: 15 Lines That Fixed My Pipeline

Here's the actual converter I use in production. It takes structured OCR output (key-value pairs and line item lists) and produces markdown tables:

def invoice_to_markdown(ocr_result: dict) -> str:
    lines = []

    # Header section
    lines.append("## Invoice Header")
    lines.append("| Field | Value |")
    lines.append("|---|---|")
    for key in ["invoice_number", "date", "supplier", "bill_to"]:
        if key in ocr_result:
            lines.append(f"| {key.replace('_', ' ').title()} | {ocr_result[key]} |")

    # Line items
    if "line_items" in ocr_result:
        lines.append("\n## Line Items")
        lines.append("| Description | Qty | Unit Price | Tax Rate | Line Total |")
        lines.append("|---|---|---|---|---|")
        for item in ocr_result["line_items"]:
            lines.append(
                f"| {item['description']} | {item['qty']} | "
                f"{item['unit_price']} | {item['tax_rate']} | "
                f"{item['line_total']} |"
            )

    # Totals
    lines.append("\n## Totals")
    lines.append("| Field | Amount |")
    lines.append("|---|---|")
    for key in ["subtotal", "tax_amount", "discount", "total"]:
        if key in ocr_result:
            lines.append(f"| {key.replace('_', ' ').title()} | {ocr_result[key]} |")

    return "\n".join(lines)

The function expects structured OCR output — key-value pairs for the header and totals, a list of dictionaries for line items. If you're using Tesseract directly, you'll need a pre-pass to parse the raw text into this structure. If you're using AWS Textract or Google Document AI, you already get key-value pairs and table structures from the API — the conversion is even simpler.

The full prompt looks like this:

markdown_invoice = invoice_to_markdown(ocr_result)

prompt = f"""Extract the following fields from this invoice:

1. Supplier name
2. Invoice number
3. Tax amount (the dollar amount, not the percentage)
4. Total amount due
5. Line item descriptions (as a list)

Invoice data:

{markdown_invoice}
"""

That's it. No few-shot examples, no chain-of-thought scaffolding, no complex system prompt. The structure does the work.

Why This Works: Structure Is Disambiguation

Language models don't "see" whitespace the way humans do. When text is linearized, the spatial relationships that make an invoice readable — "this number is in the tax column" — disappear. The model has to reconstruct them from context, and it gets it wrong about 29% of the time.

Markdown tables solve this at the representation level. Pipe characters (|) are explicit column boundaries. Newlines between rows are explicit row boundaries. Header rows tell the model what each column means. The model doesn't have to guess whether "10%" is a tax rate or a discount percentage — the column header answers that question before the model even processes the value.

This aligns with findings from the Anthropic team's research on prompt engineering — structured input consistently outperforms unstructured input, especially when the task requires disambiguating similar-looking fields. XML tags work for hierarchical data. Markdown tables work for tabular data. The principle is the same: send the structure, not the dump.

Where this generalizes

This isn't just about invoices. The same principle applies to any document where tabular structure carries meaning:

• Bank statements — transaction rows with date, description, amount columns
• Purchase orders — line items with SKU, quantity, price
• Pricing sheets — product tiers with feature matrices
• Shipping manifests — item lists with weights and dimensions
• Expense reports — category, vendor, amount rows

In all of these, raw OCR flattening destroys the column semantics that the model needs to correctly identify which number means what.

Handling Edge Cases in Production

The markdown table approach handles 97.8% of fields correctly out of the box. Here's how I handle the remaining 2.2%:

European number formats. Some invoices use commas as decimal separators (1.234,56 instead of 1,234.56). Add a normalization pass before the markdown conversion:

import re

def normalize_amount(raw: str) -> str:
    # Handle European format: 1.234,56 -> 1234.56
    if re.match(r"^\d{1,3}(\.\d{3})*,\d{2}
quot;, raw):
        return raw.replace(".", "").replace(",", ".")
    # Handle US format: 1,234.56 -> 1234.56
    if re.match(r"^\d{1,3}(,\d{3})*\.\d{2}
quot;, raw):
        return raw.replace(",", "")
    return raw

Multi-page invoices. Large invoices that span multiple pages may have line items split across page breaks. Concatenate all line item tables before sending to Claude, and include a page number in a comment so you can trace errors back to the source page.

Mixed currency. If an invoice lists amounts in multiple currencies (common for international suppliers), add a currency column to the line items table rather than embedding it in the description string.

Confidence-based human review. For fields where accuracy is critical (invoice numbers, total amounts), run a second pass with a validation prompt: "Given this invoice data, verify that the tax amount plus subtotal minus discount equals the total. Return PASS or FAIL." Route any FAIL results to manual review.

How this looks in practice at bizflowai.io

For clients at bizflowai.io, this markdown-table preprocessing is baked into the invoice extraction pipeline — OCR output from Textract and Tesseract gets restructured into structured markdown before it hits the model, and the validation pass flags any total that doesn't reconcile against line items. The same pattern handles receipts, purchase orders, and bank statements. The pipeline runs on a home server with WSL Ubuntu and processes a few hundred documents per week with a 97%+ field accuracy rate and a human-review queue for the remainder.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.