Cohere Model Vault Encrypted: What Private AI Inference

By Lazar Milicevic · Published October 25, 2026 · 10 min read

You're about to send client contracts, medical intake forms, or financial statements to a hosted LLM, and the question you can't answer cleanly is: who can see this while the model reads it? Encryption at rest and in transit doesn't cover the moment of processing. Cohere's new confidential computing option for Model Vault is a serious attempt to close that gap, and it's worth understanding what it does, what it doesn't, and whether it fits a small team's budget.

What Cohere actually shipped

Cohere added confidential computing to Model Vault, its single-tenant inference product, so that prompts and responses are processed inside hardware-protected memory that neither Cohere nor the cloud provider can read. VentureBeat reported the launch on September 16, 2026, and Cohere announced it on X the same day.

Some context on the product itself. Model Vault is Cohere's existing dedicated-instance offering: you get your own isolated deployment of a model rather than sharing a multi-tenant endpoint. Per Cohere's documentation, every vault is now one of two types:

• Standard Vault: the existing single-tenant deployment.
• Encrypted Vault: a Standard Vault plus confidential computing, with verifiable remote attestation. Cohere recommends it for regulated or highly sensitive workloads.

The problem it targets is the "data in use" gap. Data is typically encrypted on disk and over the network, but when a model processes a prompt, that prompt sits in plaintext in memory. Anyone with sufficient privileged access to the host (an operator, a compromised hypervisor, a rogue admin) is a theoretical exposure point. Confidential computing moves that processing inside a hardware-isolated enclave.

A note on status, because sources disagree. VentureBeat described the capability as live at launch. Cohere's own X post says early access is opening to a limited group of beta customers, and Cohere's FAQ says Model Vault Encrypted is currently in beta and that interested customers should contact Cohere to join. If you're planning around this, treat it as beta and confirm access directly with Cohere before you commit a project timeline to it.

How the technology works (without the marketing)

Encrypted Vaults run inference inside trusted execution environments (TEEs) that span both the CPU and the GPU, and the decryption keys are generated inside that environment and never shared with Cohere. That key detail is what turns "we promise not to look" into "we technically can't."

According to Cohere's Encrypted Vault overview, the stack has two layers:

Both matter. A confidential VM alone would still leave model inputs exposed in GPU memory, where the actual computation happens. Spanning CPU and GPU is what makes this relevant to LLM inference rather than just a general-purpose secure VM.

Cohere's FAQ states that Cohere cannot see prompts and responses: they are decrypted only inside the TEE, using keys generated inside the environment.

Hardware and cloud specifics vary by source. Cohere's docs reference Intel TDX or AMD SEV-SNP with NVIDIA GPUs, and the attestation docs describe the Intel TDX flow. Some third-party coverage mentions specific NVIDIA GPUs and a particular cloud platform for the public beta. I couldn't confirm that from Cohere's own documentation, so check with Cohere which platform your vault would run on, especially if you have a cloud-region or vendor constraint.

Attestation: the part that makes it verifiable

Attestation is what separates confidential computing from a trust-me promise: every inference returns an attestation report that lets you verify the exact hardware, software and security configuration that handled your request. That's per Cohere's director of serving inference, Manoj Govindassamy, as quoted by VentureBeat.

Here's how the mechanics are described in Cohere's attestation docs:

• Attestation uses Intel Trust Authority, an external attestation service run by Intel, in a "Passport" model. A third party, not Cohere, vouches for the environment.
• Attestation tokens are short-lived and refreshed periodically, which limits the window in which a stale or compromised claim could be reused.
• The Encrypted Vault code is open source and reproducibly built. Customers or independent parties can rebuild the artifacts and check that the attestation measurements match inspectable source.

That last point is the one I'd weigh most heavily if I were advising a client's security reviewer. Without reproducible builds, an attestation tells you "this software ran" without telling you what that software does. With them, a technically capable reviewer can close the loop.

One caveat on cadence: verification happens before requests are sent and is refreshed periodically, and responses carry attestation information as described in Cohere's docs. The exact granularity (per request versus per session) is something I'd confirm directly. If your auditor needs a specific guarantee here, get it in writing from Cohere.

What it looks like in your code

The integration cost is deliberately low: you keep the same Cohere API, and add a client-side OHTTP proxy that verifies attestation and encrypts requests before data leaves your machine. Per Cohere's docs, the proxy is available as a standalone proxy or as a Python package, and application code stays unchanged.

OHTTP (Oblivious HTTP) is the mechanism that encrypts the payload to a key that only exists inside the TEE. In practice, that means the architecture looks like this:

Your app  →  local OHTTP proxy  →  (network, relay)  →  TEE inside Encrypted Vault
              ├─ verifies attestation
              └─ encrypts request before it leaves

I'm not going to paste a proxy configuration here, because Cohere's quickstart and proxy docs are the source of truth and the package is in beta. Check the current interface before you build. What you can plan for, architecturally, looks like this:

# Conceptual shape only; confirm the real client and proxy setup
# in Cohere's current Encrypted Vault docs before building on this.
import cohere

# Point the standard client at the local proxy endpoint
# instead of a public API URL. Application logic is unchanged.
client = cohere.ClientV2(
    api_key="YOUR_KEY",
    base_url="localhost  # local OHTTP proxy
)

response = client.chat(
    model="your-model-id",
    messages=[{"role": "user", "content": "Summarize this intake form: ..."}],
)

The design point worth internalizing: your application code doesn't change, but your deployment does. You now have a local component whose job is to refuse to send data to an environment that fails attestation. That's a new moving part to monitor, restart, and secure.

Another operational detail from the quickstart: the vault type (Standard or Encrypted) is chosen at creation and cannot be changed afterward. If you prototype on a Standard Vault and later need encryption, you're provisioning a new vault, not flipping a switch. Plan your migration accordingly.

What confidential inference does not protect

Confidential computing protects the contents of prompts and responses while they're processed; it does not hide metadata, guarantee data residency, or make you compliant. Cohere is upfront about this in its own docs, and you should be equally upfront with your clients.

Per the limits reported from Cohere's documentation:

• Not a data-residency mechanism. It says nothing about which country the data is processed in.
• Not a substitute for compliance or contractual controls. You still need your BAA, DPA, retention policies, and access controls.
• Not homomorphic encryption. Data is decrypted inside the enclave; the protection is isolation, not computing on ciphertext.
• Metadata stays visible. Message size, timing, traffic volume and routing remain observable. The API credential and model identifier sit outside the OHTTP-protected payload.

And a framing point I want to be blunt about: Cohere's docs say Encrypted Vaults are designed to help regulated organizations meet obligations such as GDPR, HIPAA and SOC 2. That is not the same as "makes you HIPAA compliant." Compliance is a property of your whole process: who has access, what you log, how long you retain, what your agreements say. A TEE is one control among many. Don't let it end up in a client deck as a compliance claim.

There's also a threat-model boundary the architecture doesn't touch: your own side of the proxy. If a user pastes sensitive data into a Slack channel that your agent reads, or your logs capture full prompts, the enclave protects none of that. Most real-world leaks in document pipelines happen before or after the model, not inside it.

Cost: what we know and what we don't

On price, the only claim is a launch-day statement: Cohere's Govindassamy said confidential computing costs Model Vault customers nothing extra and that pricing matches the standard Model Vault offering. That's from VentureBeat's report. Public beta information doesn't clearly specify pricing, and other coverage suggests the encrypted option may be billed separately from Standard Vaults. Sources conflict, so treat "no premium" as a claim to confirm in writing with Cohere, not a number to budget on.

The bigger cost question for a small business is the baseline. Model Vault is priced per instance, by model and performance tier. Third-party pricing trackers (not cohere.com), such as usagepricing.com, list rates that run from a few dollars per hour into the thousands of dollars per month for dedicated instances. Those aggregator figures differ across sources and over time, and the listed instances skew toward Embed, Rerank and similar models rather than every generative option. Check Cohere's current pricing page or ask for a quote.

Why this matters for a 1-10 person company: a dedicated instance is a fixed monthly cost regardless of usage. A simple illustrative example, with round numbers and not a quote:

For many small teams, the honest answer is that a dedicated vault is overkill unless a client contract, regulator, or genuine sensitivity forces the issue. If you process a few hundred documents a month, the math rarely works. If you're a small firm serving healthcare or legal clients who require demonstrable isolation, the fixed cost can be the price of winning the work.

Deciding whether you need this

Use confidential inference when a human or contract requires you to show that the model operator cannot read the data; otherwise, tighter pipeline hygiene usually buys you more risk reduction per dollar. Here's the decision framework I use:

1. Is the data regulated or contractually restricted? Health records, financial statements, privileged legal documents, or client data under a no-third-party-access clause push you toward Encrypted Vault.
2. Does someone need proof, not promises? If a client's security questionnaire asks "can your AI vendor access our data?", attestation gives you a technical answer instead of a policy answer.
3. Is your volume high enough to justify a dedicated instance? A fixed monthly cost needs steady use.
4. Have you fixed the cheaper leaks first? Before buying isolation, check for these:
   • Redact or tokenize identifiers before the model call where the task allows.
   • Strip full prompts from application logs.
   • Limit which employees and agents can read source documents.
   • Set retention windows on every intermediate artifact (extracted text, embeddings, cached outputs).
5. Is beta acceptable? The feature is in beta per Cohere's FAQ. For production workloads with an SLA, confirm support terms before depending on it.

One more factor: the company behind it. Cohere and Aleph Alpha announced a definitive merger agreement, per Yahoo Finance coverage, with the combined company operating as Cohere. The deal is subject to regulatory approval, so vendor-continuity questions are reasonable to raise in procurement, though I wouldn't over-read the merger as a risk or a benefit. Check current news for the deal's status.

Also worth saying plainly: I haven't researched whether Anthropic offers an equivalent confidential-computing inference option, so I won't imply that Claude has one. If a client is committed to Claude, the privacy conversation is about that vendor's own terms, retention settings, and architecture, a separate evaluation from this one.

Where this fits in practice

I build AI automations for small teams, and in document workflows with sensitive material, the model call is rarely where the exposure lives. The work that matters is upstream and downstream: what gets redacted before the call, what gets logged after it, who can read the extracted output, and how long anything persists. Keeping the model vendor swappable, and not letting your privacy controls hinge on any single provider's promise, is a sensible principle for any pipeline.

Confidential inference is a useful option inside that kind of design for anyone who needs a verifiable answer to "who can see this?" If you're weighing a privacy-safe Claude or Cohere workflow for a regulated or client-sensitive use case, the first step is mapping your data flows and deciding whether a dedicated encrypted vault is justified or whether tighter pipeline controls cover it. Get in touch to talk through your case, and be clear-eyed about what any of this does and doesn't do for your compliance position.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Request a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.