Gemma 3 Beat Gemini Nano 89% To 71% On My Invoices

By Lazar Milicevic · Published July 12, 2026 · 8 min read

Google's official I/O 2026 recap ranked WebMCP number one, Built-in AI number two, and Skills in Chrome number three. I ran all three against 1,200 real invoice PDFs from a live client pipeline. The ranking flipped, and one of Google's top-billed updates scored a flat zero.

Here's what the benchmark actually said, and what it means if you're the person paying to extract fields from documents every day.

The test rig: 1,200 invoices, one pipeline, four extractors

The setup exists to isolate the model, nothing else. Same client, same n8n workflow, same downstream validation, same database schema. The only variable was the extraction node — the piece that reads a PDF and returns structured JSON: supplier name, tax ID, line items, totals, invoice date.

• Dataset: 1,200 invoices, mixed native PDFs and scanned images, pulled from a live SMB accounting pipeline
• Baseline: GPT-4o-mini via API — what the client currently pays for at ~$0.0004 per invoice
• Contenders: WebMCP, Gemini Nano (Built-in AI in Chrome), Gemma 3 4B Q4 running locally
• Metrics: field-level accuracy, hallucination rate (measured separately), latency, marginal cost per document

Everything downstream — validation rules, category mapping, the write to Postgres — stayed byte-identical across runs. If a number moved, the model moved it.

Ground truth was a hand-audited set built over two weeks by the client's bookkeeper. That's not glamorous, but it's the only way "89%" means anything. Public OCR benchmarks tell you nothing about your document mix.

                    Accuracy   Hallucination   Latency   $/invoice
GPT-4o-mini (base)    94.0%         3.1%        1.8s     $0.00040
Gemma 3 4B Q4         89.0%         1.9%        2.1s     $0.00000
Gemini Nano           71.0%        14.0%        0.4s     $0.00000
WebMCP                 0.0%          n/a         n/a      n/a

Why WebMCP scored zero (and it isn't broken)

WebMCP lets a browser expose tools to an agent through the Model Context Protocol. That's genuinely useful — for a consumer agent driving a webpage, clicking buttons, filling forms in your logged-in session. It's the plumbing for browser-side agentic UX.

It is not plumbing for a headless backend that ingests a PDF from S3, Google Drive, or an IMAP inbox. There is no browser in that loop. There is no page for the model to see. Trying to shoehorn WebMCP into a document pipeline is like installing a car stereo on a bicycle — the part works, the context doesn't.

Google's recap didn't say that part out loud. So to be clear: if your automation runs on a server and the input is a file, WebMCP is not on your shortlist. It will matter in about 18 months when there's an actual ecosystem of MCP-exposing websites for agents to traverse. Today it's a bet, not a tool.

Gemini Nano was fast, then invented a tax ID

Built-in AI ships Gemini Nano inside Chrome. To use it from n8n I ran headless Chrome with the Prompt API exposed, called it per invoice, and parsed the JSON out.

Speed was the standout: roughly 400 ms per invoice, which is faster than any cloud model I've measured on the same task. On a laptop. For free.

The problem showed up in the field breakdown. Overall accuracy hit 71%, which sounds workable until you check which fields it fumbled.

• Supplier name: 92% (fine)
• Totals and line items: 78% (usable with validation)
• Dates: 84% (rounding-error range)
• Tax IDs: 14% hallucination rate

That last number is the killer. Nano didn't leave the tax ID blank when it wasn't sure — it invented one. Plausible-looking 9- to 11-digit strings that passed a regex check but didn't correspond to any real business. In an accounts-payable pipeline that feeds a general ledger, an invented tax ID isn't a bug. It's a compliance incident. IRS Form 1099 matching, sales-tax filings, vendor master data — all of it depends on that field being right or explicitly null.

You can bolt on a second model to validate the first, or a lookup against a business registry, but at that point you've spent your latency budget and half your engineering week. Nano is a great fit for on-device summarization, autocomplete, quick classification. It is not a fit for extracting fields your accountant will sign off on.

Gemma 3 4B at Q4: the boring middle update that won

Update three, which Google buried in a side session, was Gemma 3 4B quantized to Q4_K_M. I ran it locally on the same box I use for every other client project: a mid-range GPU with 24 GB VRAM, Ubuntu on WSL2, Ollama serving the model, wired into n8n through a plain HTTP call.

Total setup time was under an hour, most of which was pulling the model weights.

# Pull and serve Gemma 3 4B quantized
ollama pull gemma3:4b-it-q4_K_M
ollama serve

# From n8n, hit it like any other endpoint
curl http://localhost:11434/api/generate -d '{
  "model": "gemma3:4b-it-q4_K_M",
  "prompt": "Extract supplier_name, tax_id, invoice_date, line_items, total from this invoice. Return strict JSON. If a field is not clearly present, return null. Do not guess.\n\n<invoice_text>",
  "format": "json",
  "stream": false
}'

The results across 1,200 invoices:

• Field-level accuracy: 89%
• Tax ID hallucination rate: under 2% (specifically 1.9%)
• Latency: 2.1 seconds per invoice
• Marginal cost: $0

The two numbers that matter are 89% accuracy paired with sub-2% hallucination. Q4 quantization on this model preserved numeric grounding better than Nano's more aggressive distillation. When Gemma 3 didn't know a field, the strict-JSON prompt plus the model's calibration meant it returned null far more often than it guessed. That's exactly the failure mode you want in a finance pipeline: a null triggers a human review queue, a hallucination silently corrupts your books.

The prompt discipline matters. Two lines that moved accuracy noticeably:

• "If a field is not clearly present, return null. Do not guess."
• "Return the total as a number without currency symbols or thousands separators."

At 2.1 seconds per invoice, throughput is roughly 1,700 invoices per hour on a single GPU, which is more than enough for every SMB client I run.

The economics: when local Gemma beats a cloud API

The interesting math is the crossover point. GPT-4o-mini costs about $0.0004 per invoice for this task (input + output tokens at current pricing). Gemma 3 running on hardware I already own costs $0 marginal, because the electricity is already sunk in a machine that's running anyway.

Hardware to run this comfortably is a single used RTX 3090 or equivalent — call it $800 all-in if you're patient on the used market, up to $1,500 for a new mid-range card and a decent PSU.

Monthly invoices    GPT-4o-mini/mo    Gemma 3 local/mo    Payback (at $800 rig)
       500              $0.20              $0                   never
     2,000              $0.80              $0                   never
    10,000              $4.00              $0                   ~200 months
    50,000             $20.00              $0                   40 months

Wait — that table looks like Gemma never wins. Except the $0.0004 number is only GPT-4o-mini's token cost. The real bill for a cloud-based extraction includes retries on rate limits, egress on the PDF payloads, a validation model on hallucinations, and vendor lock at renewal. The full-loaded cost per invoice I've measured on client accounts is closer to $0.03–$0.05 once you count retries, validation calls, and observability.

At $0.04 per invoice, break-even on an $800 rig sits around 800 invoices per month. Any client above that volume — and every accounts-payable client I have is well above that — the local Gemma 3 setup pays for the hardware inside a quarter, then runs free forever.

How to actually benchmark this on your own documents

Don't take my 89%. Take the method. Public OCR benchmarks tell you nothing about your document mix — your suppliers, your languages, your scan quality, your edge cases.

• Pull a real sample. 200 documents minimum, sampled across your actual date range and supplier distribution, not a curated "clean" set.
• Hand-label ground truth for the fields that matter. Yes, it takes a day. Everything downstream depends on it.
• Measure field-level accuracy and hallucination separately. Average accuracy hides the failures that cost you money. A 90% average can hide a 15% hallucination rate on the one field your auditor cares about.
• Swap only the extractor. Keep validation, storage, and downstream logic identical, or you're benchmarking your plumbing.
• Run it three times. Non-determinism is real even at temperature 0; you want to see the variance.

For anyone renewing a document-AI contract this quarter, run this against your incumbent before you sign. I've watched three clients cut annual OCR spend by 60–80% by benchmarking honestly.

Why bizflowai.io helps with this

Document-extraction pipelines are one of the workflows I set up most for clients through bizflowai.io — usually replacing a mix of manual data entry, an expensive OCR SaaS, and a spreadsheet that no one trusts. The pattern is boring on purpose: intake from email or drive, extraction via whichever model actually benchmarks best on the client's own documents, strict-JSON validation with a human-review queue for nulls, then a clean write into their accounting system or ERP. When the volume justifies it, the extraction step moves to a local model. When it doesn't, we stay on the API. The decision is a spreadsheet, not a preference.

The takeaway

Google's I/O ranking optimized for developer excitement. If you rank the same three updates by what survives contact with a real client pipeline, WebMCP drops off the list entirely, Nano needs a validation layer that eats its speed advantage, and the quiet middle update — Gemma 3 4B at Q4 — is the one you can deploy this month.

WebMCP will matter in 18 months. Gemma 3 matters this quarter. If you're above a few hundred documents a month and renewing a contract soon, benchmark before you sign.

Want more like this?

I publish practical AI automation, GenAI engineering, and faceless content workflows on YouTube every week.

Subscribe to bizflowai.io on YouTube — never miss a new tutorial.

Planning an AI automation project or need a second opinion on your architecture?

Connect with me on LinkedIn — Lazar Milicevic, GenAI Engineer & bizflowai.io Founder.

Visit bizflowai.io for our services, case studies, and AI consulting.