An Azure OpenAI Assistant Leaked Files. One Filter Fixed It.

By Lazar Milicevic · Published October 15, 2026 · 11 min read

Your RAG assistant passes every eval you wrote, answers customer email accurately, and cites the right documents. It may still be handing a low-privilege user content from SharePoint files they could never open. This post walks through a real case, the two-account test that catches the problem, and the query-time fix that closed it without a new identity platform.

What happened: an assistant that passed every evaluation

Egiziago Cioffi is the IT and Enterprise Architect and CEO of SynSphere Italia, a Microsoft partner based in Milan. He built an agent himself: he wrote the indexing job, configured the Azure OpenAI retrieval pipeline, and connected it to SharePoint. The assistant passed every evaluation his team ran. (VentureBeat, published September 1, 2026)

Then he ran a different kind of test. He took a low-privilege account and asked it questions a high-privilege account had already asked. The assistant returned SharePoint content that the requesting user could not have opened in SharePoint.

Two things about this story matter for how you read it:

• It was not an Azure platform bug. The sources don't describe it as one. It was a gap in a custom retrieval pipeline. Don't file this under "Azure OpenAI is vulnerable."
• The evals weren't sloppy. They were measuring the wrong thing. The next section covers why.

If you run any assistant that retrieves from internal documents (support inbox, sales enablement, internal wiki search), the question is the same: whose permissions does the retrieval layer use when it fetches source material?

Why evals didn't catch it

The evaluations tested factual accuracy, relevance, and task completion. They did not test whose permissions the retrieval pipeline uses when it fetches source material.

Think about what a typical RAG eval set looks like: a list of questions, expected answers or expected source documents, and a scorer. The questions are usually written by someone who knows the content, and they're run under a single service identity. If that identity can read everything in the index, then every eval passes, and "relevance" scores reward the pipeline for finding the best document, regardless of whether the asking user should see it.

An eval that rewards retrieving the most relevant chunk will, by construction, reward a permission leak when the most relevant chunk sits in a restricted file.

There's a second, structural reason. Cioffi's deployment used a custom retrieval pipeline that bypassed Azure AI Search's native ACL trimming layer. That is how the gap survived every evaluation. When you build the pipeline yourself, nothing is trimming results for you unless you wrote the trimming. The default behavior of "index everything the crawler can read, retrieve by similarity" is permission-blind.

So the failure has a simple shape:

1. The indexer runs with broad read access, so it ingests restricted and unrestricted files alike.
2. Chunks land in the index with no record of who may read them (or with a record nobody checks).
3. At query time, retrieval ranks by relevance only.
4. The model sees whatever ranked highest and paraphrases it to whoever asked.

SharePoint's access controls never get a vote, because the user isn't talking to SharePoint. They're talking to your assistant, which is talking to the index.

The two-account test you can run in thirty minutes

VentureBeat describes a test any team can run with two accounts in about thirty minutes. If the assistant returns more than the account's direct access allows, the retrieval permission boundary is not enforced at query time.

Here's how to run it against your own assistant.

Setup

• Account A: high privilege. Can open the restricted sites and libraries (HR, finance, legal, exec folders, whatever your tenant has).
• Account B: low privilege. A normal user who cannot open those locations in SharePoint. Confirm that by actually trying to open the files while signed in as B.

Procedure

1. As Account A, ask the assistant 10-20 questions whose answers live in restricted files. Save the responses and the cited sources.
2. As Account B, ask the same questions through the same assistant.
3. For every answer B receives, check the cited source or the content against SharePoint. Can B open that file directly? If not, you have a leak.
4. Also try indirect phrasing. Don't only ask for the document; ask for facts that only appear in it.

What to log

# Minimal harness sketch: replay the same questions under two identities
# and flag any answer whose sources B cannot open directly.
import json

def run_test(ask_assistant, questions, user_a_token, user_b_token, can_open):
    findings = []
    for q in questions:
        a = ask_assistant(q, token=user_a_token)   # returns {"answer": str, "sources": [url, ...]}
        b = ask_assistant(q, token=user_b_token)
        for src in b["sources"]:
            if not can_open(user_b_token, src):    # your own check against SharePoint / Graph
                findings.append({
                    "question": q,
                    "leaked_source": src,
                    "answer_for_b": b["answer"],
                })
    return findings

if __name__ == "__main__":
    # ask_assistant and can_open are yours to implement against your stack
    pass

The can_open check is the part people skip. Don't eyeball it; verify with the user's own identity against SharePoint, because "the citation looks harmless" is not evidence.

One caveat: this catches leaks through citations and content you can see. If your assistant doesn't expose sources, compare the substance of B's answer against the restricted document text.

The fix: filter at query time, before the model sees anything

The fix was a query-path filter that checks the requesting user's SharePoint permissions before the model sees a chunk. It runs at query time, not index time, and it needed no new identity platform.

In a separate VentureBeat piece, Cioffi said the filter was built from the asking user's group claims, so an unentitled chunk never becomes a candidate and never reaches the model. (VentureBeat)

The sources don't detail his exact API, field, or technology, so I won't pretend to reproduce his implementation. What I can show is the pattern Microsoft documents for this exact situation.

The documented security filter pattern

Microsoft describes security filters as an API-agnostic, generally available approach based on simple string matching. It recommends them for custom identity systems or push-model indexes. (Document-Level Access Control, Azure AI Search)

The mechanics, per Microsoft's security filters pattern: add a filterable Collection(Edm.String) field to each indexed document holding the user or group IDs allowed to read it, then apply a regular OData filter at query time. A typical filter looks like group_ids/any(g:search.in(g, ...)).

Index schema fragment:

{
  "name": "group_ids",
  "type": "Collection(Edm.String)",
  "filterable": true,
  "retrievable": false
}

Query-time filter, built per request from the caller's resolved group IDs:

def build_security_filter(group_ids: list[str]) -> str:
    # Join with a delimiter that cannot appear in a group ID.
    joined = ",".join(group_ids)
    return f"group_ids/any(g:search.in(g, '{joined}', ','))"

def retrieve(search_client, query: str, caller_group_ids: list[str]):
    return search_client.search(
        search_text=query,
        filter=build_security_filter(caller_group_ids),
        top=8,
    )

(Validate and escape group IDs before string-building a filter. Treat them as untrusted input.)

Two things the docs are blunt about

First: in this pattern, the principal is only a string used in a filter expression. There is no authentication or authorization through the principal, and the filter must be applied to every query. Microsoft is telling you the filter is only as trustworthy as the code that builds it. If your app lets the client supply the group list, or one code path forgets the filter, the protection is gone. Build the filter server-side from the verified identity, and put it in one function that every retrieval call goes through.

Second: every chunk must carry the ACL fields. In chunked indexing scenarios, a chunk missing them can't be returned to the right caller. If your chunking step copies document text but drops metadata, you've created either orphaned chunks (never returned) or, if you handle nulls badly, chunks that match everything. Test for both.

Native ACL enforcement: what's preview and what isn't

Azure AI Search also offers native, token-based enforcement. It validates the caller's Microsoft Entra token via the x-ms-query-source-authorization header and returns only documents whose synchronized permission metadata grants the caller access. Microsoft's docs label this capability as preview.

The SharePoint indexer's ACL ingestion is likewise in preview, per the Azure AI Search SharePoint ACL docs. Here's how the options compare on what the sources support:

A caution on SharePoint group support: Microsoft's documentation has been inconsistent across versions. One docs version says SharePoint groups are not supported in preview (2025-11-01-preview API), while the current Learn page lists SharePoint site groups as supported in preview starting with the 2026-05-01-preview API. Check the current page for the API version you're actually calling before you rely on group-based access.

Also note that VentureBeat's article says Azure AI Search is "currently shipping" query-time ACL trimming. Microsoft's own docs label the native approaches preview, so for a production decision, treat the docs as the authority and plan around preview limitations.

The staleness trap

With SharePoint ACL ingestion, permission changes on a parent scope (a site, library, list, or folder) are not detected automatically. They require a /resync call with options: ["permissions"].

Translation: if someone gets removed from a restricted library on Tuesday and nobody triggers a permissions resync, your index can keep granting that access. Whichever enforcement path you choose, put the resync on a schedule and alert on failures. A correct filter over stale ACL data is still a leak, just a slower one.

The tradeoff: a narrower assistant, honestly stated

The filter has a cost, and Cioffi was upfront about it. Some content the assistant used to rely on is now excluded, because the requesting user's permissions do not reach it.

That's the whole point, but it changes the product. Questions that used to get answered from a restricted file now get a thinner answer, a "I can't find that," or an escalation to a human. You've traded coverage for correctness.

On the numbers: with the filter live, the assistant still auto-resolves roughly 60% of inbound customer email, according to Cioffi. That is his own figure, given to VentureBeat, and he gave no pre-filter figure for comparison. So please don't read it as "the filter cost nothing." There's no baseline, and no independent verification. What it does suggest is that a permission-correct assistant can remain useful at meaningful volume.

Practical ways to handle the narrower scope:

1. Move shareable knowledge to shareable locations. If support needs a pricing policy or a return process, and it's buried in a restricted folder, the fix is a published, access-appropriate copy, not a wider assistant.
2. Decide the "can't answer" behavior on purpose. A graceful handoff to a human beats a confident guess.
3. Separate assistants by audience. An assistant for customer-email drafting doesn't need to retrieve from HR. Scope the index itself, not only the filter. Fewer sensitive documents in the index means fewer chances for a filter bug to matter.
4. Log filtered-out candidates (count only, not content). A rising count of "relevant but unentitled" hits tells you where your knowledge architecture is fighting your permissions.

Why this matters beyond one deployment

VentureBeat reports that this gap exists in custom Azure OpenAI pipelines, where the assistant answers with the indexer's permissions instead of the asker's, and quotes Netragard's Adriel Desautels on the problem. I found no independent prevalence statistic, so don't read it as a measured rate of how many production RAG deployments are affected.

The broader stakes are easier to source. Straiker's STAR Labs report (July 14, 2026), covering 1,700+ successful attacks, found that 91% of successful attacks on productivity agents ended in silent data exfiltration (Straiker). That figure applies to productivity agents specifically and to attacks, not to accidental over-retrieval like Cioffi's. But it frames the risk: when an agent can read widely, data leaving quietly is the common failure mode, not the exotic one.

The lesson isn't "buy a security platform." It's that the retrieval layer is an authorization boundary, and you have to treat it like one.

A checklist you can apply this week

1. Run the two-account test against your live assistant. Thirty minutes, two accounts.
2. Find where identity enters retrieval. Does the query carry the asking user's identity, or only the service account's?
3. Confirm every chunk carries ACL metadata. Spot-check the index for chunks missing the field.
4. Centralize the filter. One retrieval function, no bypass paths, filter built server-side from verified claims.
5. Schedule permission resyncs if you use SharePoint ACL ingestion, and monitor them.
6. Add permission cases to your eval set: low-privilege user, restricted question, expected result is "no answer." Make that a failing test when it leaks.
7. Shrink the index to what the assistant's audience actually needs.

Step 6 is the one that would have caught Cioffi's gap earlier. An eval suite that never switches identity can't see an identity bug.

Where to go from here

Most retrieval problems like this one don't need a platform migration. They need someone to trace where identity gets dropped in the query path, add the filter, scope the assistant, and write the permission-aware tests that should have existed from day one. I'm Lazar Milićević, and I build AI automations for small teams under the BizFlowAI name.

If you have an assistant that's passing its evals but you're not sure whose permissions it's retrieving with, start with the two-account test above.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.