PayPal’s 43% to 95% AI Accuracy Claim: What Changed

Your AI assistant can read the right database and still give the wrong answer. If you run a small business, the problem may be familiar: the assistant finds a customer record or a policy document, then applies the wrong definition of “active customer,” misses an exception, or follows a procedure your team no longer uses.
PayPal reports that its internal data-science assistant improved from 43% to 95% question-answer accuracy without changing the underlying large language model. The useful question is not how to copy that result. It is what PayPal changed—and which parts a smaller team can put to work.
What PayPal reported—and what the numbers cannot prove
PayPal attributes the reported improvement to business knowledge placed around its assistant, not to a model change. According to VentureBeat’s report, the assistant already had access to vetted data. What it lacked was enough context about business definitions, established query patterns, and analytical procedures to answer reliably.
That distinction matters. Access to data tells an assistant what it can inspect. Business context tells it which data answers the question and how to interpret it.
The 43% and 95% figures are PayPal’s reported internal accuracy measures. The available report does not provide enough detail about the evaluation questions, scoring method, or sample size to treat 95% as an independently verified benchmark. It also does not identify the underlying model. The safe conclusion is narrower and more useful: PayPal reports a substantial improvement after adding a separate business-knowledge layer while leaving its model unchanged.
VentureBeat also reports that Ren Zhang said the assistant handled more than 1,000 questions and was estimated to save 1,600 hours over 10 weeks. Zhang cautioned that question volume alone does not establish productivity. That caution applies to small teams, too. An assistant can answer many questions and still create work if people must check or repair its answers.
For a founder or ops lead, the first diagnostic is therefore not “Which model should we buy?” It is: When the assistant answers incorrectly, was the needed information unavailable, or was the rule for using it missing? Those are different failures, and they need different fixes.
The missing layer is more than a document search
PayPal’s reported change was broader than adding retrieval to a chatbot. VentureBeat describes a knowledge layer containing business definitions, a “golden query” specifying how to retrieve data for a purpose, and analytical playbooks specifying how to investigate business questions.
A document search can find the page that says what “qualified lead” means. That may not be enough to answer, “How many qualified leads need follow-up?” The assistant may also need to know which system owns lead status, whether archived records count, what period the question refers to, and which follow-up states require action.
Think of the layer as three kinds of maintained knowledge:
| Knowledge | Question it answers | Small-business example |
|---|---|---|
| Business definition | “What does this term mean here?” | Which contacts count as customers rather than prospects? |
| Approved retrieval pattern | “Where and how do we get the data?” | Which CRM fields and filters produce the working follow-up list? |
| Investigation procedure | “What checks must happen before we answer?” | Check for duplicate contacts and stale statuses before reporting a count. |
These are examples, not descriptions of PayPal’s internal systems. They show why connecting an AI agent to SaaS tools is only the start. The tools expose records; they do not necessarily encode the decisions your team makes when using those records.
Separating this knowledge from the model has an operational advantage. Definitions and procedures can be reviewed as business assets instead of disappearing into prompts scattered across an application. PayPal’s stated intent was to preserve that knowledge when models change. A small business can use the same principle without building PayPal-scale infrastructure: keep approved rules in a place an owner can inspect and update.
Build a useful context layer from one recurring question
Start with a question your team answers repeatedly, then document the decision path a reliable employee would follow. Do not begin by indexing every file in the company. A narrow workflow makes it possible to see whether the assistant retrieved the right material, applied the right rule, and knew when to stop.
Suppose the recurring request is: “Which leads need a follow-up today?” Before an assistant touches the CRM, write down:
- The owner: Who decides what “needs follow-up” means?
- The source: Which system contains the authoritative status and activity history?
- The definition: Which records qualify, and which are excluded?
- The procedure: What should be checked before a lead is added to the list?
- The boundary: Does the assistant prepare a list, draft messages, or actually send them?
- The exception: What happens when a record is incomplete or two sources disagree?
An illustrative, human-readable specification might look like this:
question: "Which leads need follow-up?"
owner: "Sales operations"
source: "Approved CRM records"
definition:
include: "Leads marked for follow-up under the current team policy"
exclude: "Records opted out of contact or already assigned a completed action"
procedure:
- "Use the approved CRM view"
- "Check recent activity before proposing an action"
- "Flag missing or conflicting status for human review"
allowed_output: "A reviewable list with reasons and source records"
not_allowed: "Send messages without approval"
This is not a universal follow-up policy or executable configuration. It is an example of information you must settle before asking a model to improvise one. A real version should name the actual source, define the qualifying statuses precisely, and have an owner who approves changes.
The same approach works for invoice questions, support triage, and internal document answers. Pick one question, record the rule, identify the source, and make the permitted action explicit. If the team cannot agree on the answer without AI, the model will not resolve the ambiguity safely.
Retrieval must return the right rule, not just relevant text
Retrieval-augmented generation, or RAG, helps an assistant ground an answer in source material, but retrieval alone does not tell it which source has authority. Google Cloud’s documentation describes RAG as a way to ground responses by retrieving relevant source data. The engineering work is deciding what may be retrieved, what takes precedence, and what to do when the evidence is insufficient.
For a small-business document pipeline, useful records need more than text. Give each approved item a title, source, owner, status, and information about when it was last reviewed. Distinguish a current policy from an old draft. Keep access controls attached to records so an assistant cannot surface a restricted document merely because its wording matches a question.
A practical retrieval path looks like this:
- Classify the request. Is the user asking for a policy explanation, a record lookup, a calculation, or an action?
- Limit eligible sources. Apply the user’s permissions and the workflow’s approved-source list before searching.
- Retrieve the smallest useful context. Return the relevant definition and procedure alongside the underlying record or document.
- Check authority. If retrieved sources conflict, prefer the approved source only when that precedence has been defined.
- Produce a traceable answer. Cite the source records or documents used, and mark unresolved cases for review.
Anthropic’s discussion of context engineering frames the task as curating what enters an agent’s context window; retrieval is one way to surface relevant information. That is a better mental model than “put all our PDFs in a vector database.” More material is not automatically better context.
There is also an important distinction between reference text and instructions. A retrieved customer email, document, or web page may contain words that look like commands. It should not be allowed to override the approved procedure or the user’s permissions. Treat retrieved content as evidence to assess, not as an administrator of the assistant.
Test the decision path, not just whether an answer sounds good
An evaluation set should include questions with known, reviewable answers and cases where the correct behavior is to ask for clarification or decline to act. A fluent response is not evidence that the assistant used the right definition, retrieved the right record, or respected an approval boundary.
For the follow-up example, collect real questions from your workflow after removing sensitive details where needed. Have the process owner record the expected outcome and the reason. Include straightforward cases, exceptions, incomplete records, conflicting sources, and requests outside the assistant’s authority.
Score more than the final wording:
| Check | What to inspect |
|---|---|
| Source selection | Did it use an approved, accessible source? |
| Definition | Did it apply the team’s current qualification rule? |
| Procedure | Did it perform the required checks? |
| Uncertainty | Did it flag missing or contradictory evidence? |
| Action boundary | Did it stop before an unapproved send or edit? |
Keep the test set stable when comparing a model change or a context-layer change. Otherwise, you cannot tell what caused the difference. Record the assistant’s answer, the context it received, the source identifiers it cited, and the reviewer’s decision. When it fails, label the failure specifically: missing document, wrong retrieval, ambiguous definition, skipped procedure, or model misinterpretation.
This is where PayPal’s report becomes useful without becoming a promise. Its reported result suggests that knowledge outside the model can matter greatly. It does not establish that adding RAG will reproduce PayPal’s accuracy gain for another business. Your own test questions—and the work required to review or correct answers—are the measures that matter.
Put the assistant into a workflow with a safe stopping point
A production assistant needs a defined point where it hands work to a person. For many small teams, the first useful version prepares an answer or action for review rather than acting on its own. That still removes repetitive searching and drafting while keeping a human in control of consequential decisions.
Consider an invoice-status request. A safe early workflow might retrieve the customer’s invoice records, identify the relevant invoice, draft a response using the approved status definition, and attach links to the source records. If the records disagree or the request asks for a change to payment details, the assistant stops and routes the case to the responsible person. It should not infer financial facts from a customer’s message or make an account change because a document told it to.
Before expanding automation, watch for operational failures that an answer-only demo will miss:
- Stale knowledge: A policy changes, but the indexed copy does not.
- Broken provenance: The answer sounds right, but no one can locate its source.
- Permission drift: A connector exposes records the requesting user should not see.
- Silent action: The assistant updates a system when it was authorized only to suggest an update.
- Unowned exceptions: Ambiguous cases pile up because no one is responsible for resolving them.
Give the context layer an owner, make updates reviewable, and rerun relevant test questions when a definition, connector, or procedure changes. If a new model is introduced later, test it against the same business cases. Keeping knowledge separate makes that comparison possible; it does not remove the need to test.
How BizFlowAI approaches this
BizFlowAI builds practical AI automations for solopreneurs and small teams. For a document-heavy or retrieval-based workflow, a useful starting point is the business question, its approved sources, and the point where a person needs to review the result—not an assumption that a different model will fix a missing process.
One possible design approach is to give an assistant relevant business context through document retrieval while keeping definitions and procedures inspectable. If your team has a recurring question an assistant keeps getting almost right, a conversation about that process could help clarify what is missing: source access, business context, or a clearer decision boundary.
Work with BizFlowAI
If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.
Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.
More guides like this on the BizFlowAI blog.
Frequently asked questions
How did PayPal improve its AI assistant accuracy without changing the model?
PayPal reports that its internal data-science assistant improved from 43% to 95% question-answer accuracy after it added a business-knowledge layer. That layer included business definitions, approved ways to retrieve data, and procedures for investigating questions. The figures are PayPal’s internal measures; the available report does not provide enough evaluation detail to treat 95% as an independently verified benchmark.
Why does my AI assistant give wrong answers even when it can access the right data?
Access to a record does not tell an assistant how your business defines terms or which exceptions apply. It may find the right customer record but use the wrong definition of an active customer or follow an outdated procedure. Document the authoritative source, the applicable rule, and the checks required before it answers.
What should a small business put in an AI assistant’s knowledge layer?
Start with one recurring question and document its business definition, approved data source, retrieval method, and decision procedure. Name an owner who can approve changes, and specify what the assistant may do when information is missing or conflicting. For a lead follow-up workflow, that could mean producing a reviewable list while requiring human approval before sending messages.
Is RAG enough to make an AI assistant answer business questions accurately?
Retrieval-augmented generation, or RAG, can supply relevant documents and records, but it does not decide which source is authoritative. An assistant also needs rules for source precedence, user permissions, and what to do when evidence is insufficient. Retrieved text should be treated as evidence, not as instructions that can override an approved procedure.
How should I test whether my AI assistant is using business rules correctly?
Use a set of real recurring questions with answers and decision paths reviewed by the people who own those workflows. Check whether the assistant selects the approved source, applies the current definition, handles exceptions, and flags unresolved cases. Track the work needed to verify or repair answers, not just how many questions the assistant handles.