AWS OpenSearch, Agents and the Cost of Memory

By Lazar Milicevic · Published September 25, 2026 · 10 min read

Your agent works. It answers customer questions, drafts proposals, closes tickets. Then the AWS bill lands and your CFO wants a meeting. The culprit isn't the LLM — it's the retrieval layer running under it, doing five vector lookups per turn on infrastructure that scales by the hour whether the agent is busy or idle.

This is the tradeoff every team building agents on AWS OpenSearch is walking into right now, and it's the reason "serverless" versus "provisioned" is no longer just a database decision. It's an agent-architecture decision.

What AWS is actually pitching around OpenSearchCon

AWS's message ahead of OpenSearchCon is straightforward: OpenSearch is not just a search engine bolted onto Kibana anymore. It's being positioned as the retrieval substrate for enterprise AI agents — vector search, lexical (BM25) search, hybrid ranking, and increasingly, the long-term memory an agent reaches back into between sessions.

The three pieces to watch:

1. Hybrid retrieval — BM25 + dense vectors combined with a reranking step, all inside one query DSL. No stitching two systems together.
2. OpenSearch Serverless — capacity managed in OCUs (OpenSearch Compute Units), billed by usage rather than by cluster size. Good for spiky workloads, punitive if you leave a small always-on index running.
3. Agent memory patterns — using the same index for RAG (retrieve documents), episodic memory (revisit past decisions), and operational lookups (inspect a customer record) inside a single agent turn.

The pitch is real. The economics are the part that quietly determines whether your agent business works.

Why an agent hits retrieval five times, not once

A "simple" agent task is rarely one retrieval. Take a support agent handling a refund request:

• Retrieval 1: find the customer's order history (structured lookup)
• Retrieval 2: pull the refund policy relevant to the SKU (semantic)
• Retrieval 3: check past conversations with this customer (episodic memory)
• Retrieval 4: look up whether the shipping carrier logged a delivery exception (operational data)
• Retrieval 5: re-retrieve after the LLM asks a clarifying sub-question the planner didn't anticipate

Five queries. One task. If each query hits a vector index with a large embedding, plus a BM25 index, plus a rerank pass, you're doing 10-15 index reads per user turn. Multiply by conversation length. Multiply by DAU.

The lesson here is not "reduce retrievals." Agents get smarter when they're allowed to look things up. The lesson is: the cost per retrieval has to be low enough that the agent can be curious.

Hybrid search: the default that beats "just use vectors"

Pure vector search sounds elegant until a user searches for an exact SKU, an order ID, or a proper noun the embedding model has never seen. BM25 nails those. Vectors nail paraphrase and intent. Hybrid does both and reranks.

Here's a minimal OpenSearch hybrid query. This is the shape most production agents converge on:

{
  "size": 10,
  "query": {
    "hybrid": {
      "queries": [
        {
          "match": {
            "content": {
              "query": "refund policy for damaged electronics"
            }
          }
        },
        {
          "neural": {
            "content_embedding": {
              "query_text": "refund policy for damaged electronics",
              "model_id": "your-embedding-model-id",
              "k": 20
            }
          }
        }
      ]
    }
  }
}

Two things to note. First, k: 20 on the vector side but size: 10 on the response — you're retrieving more than you return so the reranker has something to work with. Second, both queries run in parallel inside OpenSearch, which matters for latency but also means you're paying for two searches per call. Budget accordingly.

For a rerank step, most teams either use the OpenSearch built-in normalization processor (score fusion, cheap) or an external cross-encoder call (better quality, more expensive per query). Start with normalization. Only reach for a cross-encoder when you can measure that the top-5 quality is genuinely hurting task success.

Serverless vs provisioned: the honest comparison

AWS positions OpenSearch Serverless as the default. In practice, it's the right choice for maybe 40% of agent workloads. Here's how to think about it:

The trap: teams pick Serverless because the docs suggest it, then discover that a small always-on collection has a minimum OCU floor. That floor, running 24/7, can cost more per month than a modest provisioned t3.medium.search cluster hosting the same data.

Rule of thumb I use when architecting: if your index does less than roughly 1 query/second averaged over a day, and the index fits comfortably on a small provisioned node, provisioned is almost always cheaper. If your load is bursty by 10x or more, serverless wins. Anything in the middle, model both on a spreadsheet before you commit.

Check current OCU pricing on the AWS pricing page before you build the spreadsheet — the numbers move.

Agent memory: three types, one index, careful design

The interesting shift in the OpenSearch-for-agents story is treating the same infrastructure as three different memory systems:

Working memory — the current conversation. Short-lived, high-write, low-value after 24 hours. Often better in Redis or DynamoDB with TTL. Do not put this in OpenSearch just because you can.

Episodic memory — past interactions, decisions, artifacts. "What did we tell this customer last month?" This is where OpenSearch shines: hybrid search, retained indefinitely, retrievable by both content and metadata filters.

Semantic memory — the reference corpus. Documents, policies, product data, wikis. Classic RAG territory.

The mistake teams make is dumping all three into one index with the same schema. You end up with a giant hot index full of ephemeral conversation turns, blowing up your storage bill and slowing every query.

Better pattern:

indices:
  agent_working_memory:
    backend: redis           # not OpenSearch
    ttl_seconds: 86400

  agent_episodic:
    backend: opensearch
    engine: hybrid           # BM25 + vector
    partitioning: monthly    # rollover for lifecycle mgmt
    retention_days: 730

  knowledge_base:
    backend: opensearch
    engine: hybrid
    partitioning: by_source  # policies/, products/, wikis/
    reindex_on_change: true

Monthly rollover on episodic memory matters more than most people realize. When a customer asks about a decision from two years ago, you don't need it in the hot tier. OpenSearch's ISM (Index State Management) policies can move older monthly indices to warm or cold storage automatically. This is where serious cost savings hide.

A concrete cost-tuning checklist

If you're already running an agent on OpenSearch and the bill is climbing faster than usage, work through this list before you re-architect:

1. Embedding dimensions. Are you using a 1536-dim embedding when 768 or 384 would score within 2% on your eval set? Storage and query cost scale directly with dimensions.
2. k values. Retrieving k=100 "to be safe" and reranking to 5 is often just paying 20x for a marginal quality bump. Measure it.
3. Index count. Every collection in Serverless has a minimum footprint. Consolidating 12 small collections into 3 well-partitioned ones often cuts baseline cost significantly.
4. Query caching. Agent workloads have surprisingly repetitive retrieval patterns (same policy lookups, same product docs). A short-TTL cache in front of OpenSearch — even 60 seconds — can cut query volume 30-50% for support agents.
5. Embedding batch jobs. If you re-embed the whole corpus nightly, you're paying for it. Only re-embed changed documents. Track content hashes.
6. ISM lifecycle. Move indices older than N days to warm nodes. Cold storage for compliance-only data.
7. BM25-first fallback. For queries that look like exact-match (IDs, SKUs, names — detectable with a cheap regex), skip the vector call entirely. You'll answer faster and pay less.

None of these is dramatic on its own. Together they routinely take an over-provisioned agent infra bill down by half without touching the LLM.

The retrieval eval loop nobody wants to build

Every team wants to skip this part. Every team that skips it ends up tuning retrieval by vibes and paying too much for infrastructure that isn't measurably helping.

The eval loop:

# Simplified — real version has more edge cases
def eval_retrieval(test_set, retriever_config):
    results = []
    for query, expected_doc_ids in test_set:
        retrieved = retriever.search(query, **retriever_config)
        retrieved_ids = [r["id"] for r in retrieved[:5]]

        recall_at_5 = len(set(retrieved_ids) & set(expected_doc_ids)) / len(expected_doc_ids)
        results.append({
            "query": query,
            "recall_at_5": recall_at_5,
            "latency_ms": retrieved.latency_ms,
            "cost_estimate": estimate_query_cost(retriever_config),
        })
    return summarize(results)

# Then sweep configs
for k in [10, 20, 50]:
    for weights in [(0.5, 0.5), (0.3, 0.7), (0.7, 0.3)]:
        config = {"k": k, "bm25_weight": weights[0], "vector_weight": weights[1]}
        print(eval_retrieval(test_set, config))

Build a test set of 100-300 realistic queries with known-good documents. Sweep your hybrid weights, k values, and reranker choices. Plot recall@5 against cost-per-query. Pick the point on that curve you're comfortable paying for. Re-run the sweep quarterly as your corpus grows.

This is the single highest-leverage thing an ops-minded founder can do for an agent product. It turns "our AI is expensive" from a mystery into a knob.

Where agents legitimately need multi-step retrieval

Some teams push back on multi-step retrieval, thinking it's a sign of bad planning. Sometimes it is. But often it's the correct behavior for tasks where the answer depends on things the agent can't know upfront.

Legitimate patterns:

• Query decomposition. User asks "compare our Q3 numbers to what we told the board last time." Agent has to (1) fetch current Q3 data, (2) find the last board meeting notes, (3) extract what was communicated. Three retrievals, all necessary.
• Iterative refinement. First retrieval returns a policy that references another policy by name. Agent fetches the referenced one. This is how humans read docs.
• Cross-domain synthesis. Support agent handling a billing question that turns into a technical one. Different indices, different retrieval calls.

The cost management here isn't "do fewer retrievals." It's making sure each retrieval is cheap enough that the agent's curiosity isn't a budget event. That, in turn, is why the OCU model matters — inefficient indexing gets punished twice, first on storage, then on every one of those five queries per turn.

How BizFlowAI approaches this

Retrieval cost, serverless-versus-provisioned tradeoffs, and layered agent memory are the exact levers we tune for clients every week. Most teams come to us already spending too much on OpenSearch or Pinecone because they defaulted to a big embedding model, one giant index, and pure vector search. We rebuild the retrieval layer as hybrid BM25 + vector with monthly episodic partitioning, appropriate lifecycle policies, and a real eval harness so quality changes are measured, not guessed.

The result is usually the same shape: retrieval cost drops meaningfully, latency improves because BM25 handles the easy cases, and agent task success goes up because the retriever finally returns the right documents for exact-match queries it was previously guessing on. If you're building on OpenSearch and the numbers aren't adding up, book a discovery call — we'll look at your setup and tell you honestly whether it needs re-architecture or just tuning.

What to take into OpenSearchCon week

Three questions worth asking any vendor or AWS solutions architect pitching you retrieval infrastructure for agents:

1. "What does one agent turn cost in your setup, end to end?" If they can't answer in cents, they haven't measured it.
2. "Show me the recall@5 curve as I vary k and hybrid weights." If they don't have an eval harness, they're guessing.
3. "When does Serverless become more expensive than provisioned for my workload?" There's always a crossover point. Anyone who won't name it isn't being straight with you.

The agent memory story on OpenSearch is real and the primitives are solid. The failure mode is treating it as a managed service you turn on and forget. Retrieval is the layer where agent products win or lose on unit economics, and it deserves the same rigor you'd apply to your database schema or your billing pipeline. Build the eval loop. Split your memory types. Model both pricing modes on a spreadsheet. The infrastructure will hold up. The bill will make sense.

Work with BizFlowAI

If you'd rather have this built for you, that's what we do: production AI automation for solo founders and small teams — agents, integrations, and document pipelines that actually ship.

Book a free discovery call — 30 minutes, we map the highest-ROI automation in your workflow. No pitch deck, just engineering.

More guides like this on the BizFlowAI blog.