RAG at scale: architecture patterns for enterprise knowledge retrieval on AWS

Vector similarity alone misses exact matches (product SKUs, error codes, legal citations) because dense embeddings optimize for semantic closeness, not lexical precision. The production fix is hybrid search: combine dense vector retrieval with BM25 keyword search, then rerank. The pattern that shows up consistently across 2026 production guides is retrieve top-50 candidates via hybrid search, rerank with a cross-encoder, keep the top-5 to top-10 for the LLM. That two-stage design reports 15-30% answer-quality improvement on standard RAG benchmarks.

‍

The reason cross-encoder reranking works where bi-encoder retrieval alone does not: a bi-encoder embeds the query and each document independently, which is fast but throws away interaction information. A cross-encoder reranker processes the query and a candidate document together in a single forward pass, capturing term-level interactions the bi-encoder never sees. That precision costs latency, which is why rerankers run on a shortlist of 50 candidates instead of the full corpus. Cohere Rerank and open BGE-Reranker models are the two most commonly cited choices in current production stacks.

‍

‍

Figure 3: Hybrid retrieval and reranking pipeline for enterprise RAG on AWS. The reranking stage is the accuracy-critical step: it is the only point where a slower, more precise model looks at query and document together.

‍

Two disciplines matter beyond the diagram. First, cap what reaches the model: past roughly 8 chunks, models start ignoring passages buried in the middle of a long context, the same "lost in the middle" effect that made naive top-k retrieval unreliable in the first place. Second, enforce permission inheritance at retrieval time: enterprise RAG platforms filter hybrid search results against the source system's access controls (Confluence space permissions, Salesforce record ownership) before a chunk ever reaches the reranker, not after.

Foundation models like Claude and the latest Gemini and GPT releases have gotten good enough that the model is rarely the weak link anymore. The weak link is what gets handed to it. A grounded RAG system can only answer as well as the passages the retrieval layer selects, and that selection happens before the language model sees a single token. Every architecture decision downstream of ingestion (chunk boundaries, index type, ranking strategy) either preserves that selection quality or erodes it.

Amazon Bedrock Knowledge Bases expose five chunking strategies for customer-managed knowledge bases: default fixed-size chunking, explicit fixed-size, semantic, hierarchical, and no chunking at all. Each strategy trades off differently. Semantic chunking splits text at meaning boundaries instead of arbitrary character counts, and AWS's own guidance recommends exploring chunk sizes between 128 and 1,024 characters adapted to document structure. Hierarchical chunking retrieves at a fine-grained child level but returns the parent chunk for surrounding context, which helps on long structured documents like compliance manuals or technical specifications.

‍

The honest caveat: semantic is not automatically better. A reproducible 2026 benchmark running 25 questions against Bedrock Knowledge Bases found semantic chunking underperformed no-chunking on some corpora, and concluded there is no single best strategy: only measurement per corpus matters. Hierarchical chunking also has a hard AWS limit worth knowing before it costs a debugging afternoon: Bedrock Knowledge Bases cap custom metadata at 1 KB and 35 keys per vector, and hierarchical chunking with high token counts can blow past that ceiling because parent-child relationships are stored as non-filterable metadata.

‍

python

# Customer-managed Bedrock Knowledge Base: semantic chunking config

chunking_configuration = {

    "chunkingStrategy": "SEMANTIC",

    "semanticChunkingConfiguration": {

        "maxTokens": 300,

        "bufferSize": 1,

        "breakpointPercentileThreshold": 95

    }

}

‍

‍

Figure 1: The ingestion path from source document to vector index. Chunking strategy is the single decision point that determines what the retrieval layer can ever find.

‍

For teams that don't need every tuning knob, AWS's newer Bedrock Managed Knowledge Base (generally available since June 17, 2026) collapses the entire ingestion pipeline (storage, embeddings, chunking, reranking) into a single managed primitive with built-in or fixed-size chunking only. That trade removes weeks of chunking experimentation but also removes the retrieval-tuning surface this article covers. Teams pick customer-managed Knowledge Bases specifically because retrieval-quality engineering requires the knobs the managed product hides.

‍

The teams shipping reliable enterprise RAG on AWS in 2026 are not the ones with the most exotic pipeline. They are the ones who measured chunking, vector store, and reranking against their own corpus and real user queries instead of copying a reference architecture wholesale. Start with hybrid search and a reranker before reaching for GraphRAG or agentic retrieval: the cheapest upgrade fixes the majority of retrieval failures. Build the offline evaluation set first; every chunking and vector-store decision after that becomes a measurement, not a guess.

The headline number worth anchoring a project plan to: naive top-k RAG fails at retrieval roughly 40% of the time, and hybrid search plus reranking is reported to lift answer quality 15-30% over that baseline on standard benchmarks. Neither number should be taken as a guarantee for a specific corpus; both come from aggregated 2026 production write-ups rather than a single controlled study, and the correct move is to build an offline evaluation set from real user queries against your own documents, then measure retrieval hit rate, context relevance, and final answer groundedness before and after each architectural change. Latency and cost need the same treatment: a two-stage hybrid-plus-rerank pipeline adds a reranker call on the critical path, which is a deliberate trade of milliseconds for accuracy that should be measured, not assumed acceptable.

Chunking strategy selection still lacks a reliable a priori rule. The benchmark evidence that semantic chunking can underperform no-chunking on certain corpora means teams cannot skip the measurement step, and AWS's own metadata limits mean hierarchical chunking needs a capacity check before it reaches production scale. Vector-store migration between OpenSearch and Aurora pgvector past the 10-50 million row ceiling is not yet a one-command operation; it requires re-indexing and a retrieval-quality re-validation pass. Reranker latency also remains an unresolved cost-versus-accuracy dial: cross-encoders are an order of magnitude slower per comparison than bi-encoder retrieval, and no current AWS-native service auto-tunes the retrieve-then-rerank candidate count for a given latency budget.

‍

Key takeaways
‍

1. Retrieval, not generation, is the bottleneck. Naive top-k RAG fails roughly 40% of the time; the fix lives in the retrieval layer, not a bigger model.
   ‍
2. No chunking strategy wins by default. Semantic and hierarchical chunking both carry documented failure modes; measure retrieval hit rate and context relevance per corpus before committing.
   ‍
3. OpenSearch and Aurora pgvector solve different problems. Pick pgvector to minimize moving parts on an existing Aurora workload; pick OpenSearch for search-heavy, high-QPS, or beyond-50-million-row corpora.
   ‍
4. Hybrid search plus reranking is the standard production pattern. Retrieve top-50 via vector + BM25, rerank with a cross-encoder, pass top-5 to top-10 chunks to the LLM.
   ‍
5. Cap what reaches the model. Beyond roughly 8 chunks, "lost in the middle" effects erase the accuracy gained from better retrieval.
   ‍
6. Enforce permissions at retrieval time. Source-system access controls must filter candidates before reranking, not after generation.

AWS ships at least eight services capable of storing and querying embeddings, and each one inherited its vector capability from a prior storage engine rather than being built as a pure vector database. That heritage decides fit. Amazon OpenSearch Service is distributed from the outset and built for high-QPS semantic search over millions of unstructured chunks, which is why independent 2026 decision guides call it the default for most production RAG. Aurora PostgreSQL with the pgvector extension supports HNSW and IVFFlat indexes at 10-100 ms typical query latency, and its real advantage is not raw speed: vector rows sit next to relational rows in the same ACID-compliant database, so a single SQL query can join a similarity search against customer records or order history.

‍

Figure 2: The scale ceiling that decides the vector-store migration. Below roughly 10 million rows, pgvector minimizes moving parts; past roughly 50 million, the lack of horizontal sharding for vector indexes forces a move to OpenSearch.

‍

The exit ramp is documented, not theoretical: pgvector "holds up well below roughly 10-50 million rows; past that, the lack of horizontal sharding for vector indexes becomes the ceiling and a move to OpenSearch is the usual exit," per a 2026 AWS vector-store decision guide. The practical heuristic one architect summarized well: if the team already operates PostgreSQL, pgvector minimizes moving parts; if the system already depends on search infrastructure, OpenSearch extends it naturally. Neither is a universal winner, which is exactly why Bedrock Knowledge Bases now support six vector-store backends directly rather than picking one for you.