Semantic RAG

Most production RAG is still the same shape. Chunk a corpus, embed each chunk, store the vectors, retrieve the top-k at query time, paste them into a prompt, call the model once. Semantic RAG is the version with dense vector similarity over learned embeddings, and tuned well it covers the majority of production lookup workloads at a cost the agentic alternatives cannot match.

This guide walks the pipeline, calls out the upgrades that actually move the numbers, and ends with two compact implementation sketches.

The pipeline

flowchart TD
    subgraph Index["Indexing (offline)"]
        direction TB
        DOC[Documents] --> CH[Chunker]
        CH --> EMB1[Embedding model]
        EMB1 --> VS[(Vector store)]
    end
    subgraph Query["Query (online)"]
        direction TB
        Q[Question] --> EMB2[Embedding model]
        EMB2 --> SEARCH[Top-k search]
        SEARCH --> RR[Reranker]
        RR --> PROMPT[Prompt]
        PROMPT --> LLM[LLM]
        LLM --> A[Answer]
    end
    VS --> SEARCH

Two timelines, sharing one component: the embedding model. The model that embeds documents at index time must embed the query at retrieval time, or the vector spaces do not line up.

Indexing: chunk, embed, store

Chunking. Recursive character splitting at 400 to 512 tokens with 10 to 20 percent overlap is the default. Two upgrades worth knowing: parent-child retrieval (index small chunks, return larger ones) and Anthropic's Contextual Retrieval, which prepends an LLM-generated 50 to 100 token context to each chunk and reduces top-20 retrieval failures by up to 67 percent when combined with a reranker.

Embeddings. OpenAI text-embedding-3-large (3072 dim, Matryoshka-truncatable) is the safe default. Voyage voyage-3-large beats it by roughly 10 percent on cross-domain evals. Cohere embed-v4 is the multimodal pick (text plus image in one payload). For open weights, BGE-M3 and jina-embeddings-v3 are strong.

Vector store. pgvector if Postgres is already in the stack. Qdrant for filtered HNSW search and multi-tenant payloads. Weaviate for tight hybrid sparse-dense. Milvus or Pinecone for billion-scale. HNSW is the default index algorithm. Quantization (int8 or binary plus a rescoring pass) is a near-free 4x to 32x memory saving.

Retrieval: hybrid, rerank, optionally rewrite

Hybrid search. Dense embeddings miss exact-token matches (SKUs, error codes, named entities). BM25 catches them. Run both and fuse with Reciprocal Rank Fusion (Cormack et al., 2009):

RRF(d) = sum over rankers r of  1 / (k + rank_r(d))      # k typically 60

RRF uses only ranks, sidestepping the impossible problem of normalizing BM25 scores against cosine similarities. Native in Elasticsearch, OpenSearch, Weaviate, Qdrant.

Rerankers. Run a cross-encoder over the top 50 to 100 candidates from the fast bi-encoder. Cohere Rerank 3.5 and Voyage rerank-2.5 lead the closed options; BGE rerankers are the open default. Reranking is usually the single highest-ROI upgrade after hybrid.

Query transforms. One LLM call before retrieval that stays non-agentic. HyDE drafts a hypothetical answer and embeds that document instead of the query (Gao et al., 2022). Multi-query expansion paraphrases the query, retrieves N times, fuses with RRF.

Generation: ground and cite

The prompt template that production converges on has three delimited sections.

[System]
Answer ONLY from the context. Cite as [doc_id].
Say "I do not know" if the context lacks the answer.

[Context]
<doc id="1" source="acme-10q-q2-2023.pdf#p7">...</doc>
<doc id="2" source="acme-press-release.html">...</doc>

[User]
What was ACME's Q2 cloud revenue?

Order chunks so the highest-scoring is first or last, not buried in the middle. The lost-in-the-middle effect is real even on long-context models. Prompt caching makes a stable system plus context prefix nearly free on cache hits and is the single largest cost lever for any system with a long system prompt and a stable knowledge base.

A compact implementation, two ways

Native OpenAI plus Qdrant. Indexing once, answer on the hot path.

from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

oai = OpenAI()
qd  = QdrantClient(url="http://localhost:6333")
COLL, MODEL, DIM = "docs", "text-embedding-3-large", 3072

def embed(texts):
    r = oai.embeddings.create(model=MODEL, input=texts, dimensions=DIM)
    return [d.embedding for d in r.data]

# index once
qd.recreate_collection(COLL, vectors_config=VectorParams(size=DIM, distance=Distance.COSINE))
chunks = load_and_chunk()                                            # {"id","text","source"}
qd.upsert(COLL, points=[PointStruct(id=c["id"], vector=v, payload=c)
                        for c, v in zip(chunks, embed([c["text"] for c in chunks]))])

# answer
def answer(q, k=5):
    qv   = embed([q])[0]
    hits = qd.search(COLL, query_vector=qv, limit=k, with_payload=True)
    ctx  = "\n\n".join(f'<doc id="{h.id}" source="{h.payload["source"]}">{h.payload["text"]}</doc>'
                       for h in hits)
    return oai.chat.completions.create(
        model="gpt-4.1",
        messages=[
            {"role": "system", "content": "Answer ONLY from the context. Cite as [doc_id]."},
            {"role": "user",   "content": f"<context>\n{ctx}\n</context>\n\n{q}"},
        ],
    ).choices[0].message.content

The Anthropic equivalent swaps the client and uses Voyage for embeddings, since Anthropic does not ship a first-party embedding model.

LangChain LCEL is the same idea with fewer lines.

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import PGVector
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

store     = PGVector.from_documents(docs, OpenAIEmbeddings(model="text-embedding-3-large"),
                                    connection_string="...", collection_name="kb")
retriever = store.as_retriever(search_kwargs={"k": 5})
prompt    = ChatPromptTemplate.from_template(
    "Answer ONLY from the context. Cite as [doc_id].\n\n<context>\n{context}\n</context>\n\n{question}"
)
def fmt(ds): return "\n\n".join(f'<doc id="{i}">{d.page_content}</doc>' for i, d in enumerate(ds))

chain = ({"context": retriever | fmt, "question": RunnablePassthrough()}
         | prompt | ChatOpenAI(model="gpt-4.1") | StrOutputParser())

Adding a reranker is one wrapper:

from langchain_cohere import CohereRerank
from langchain.retrievers import ContextualCompressionRetriever
retriever = ContextualCompressionRetriever(
    base_retriever=store.as_retriever(search_kwargs={"k": 20}),
    base_compressor=CohereRerank(model="rerank-3.5", top_n=5),
)

Adding hybrid search is EnsembleRetriever([bm25, vector], weights=[0.4, 0.6]), which uses RRF under the hood.

Where the pipeline breaks

The empirical failure-mode list from Barnett et al., 2024 is worth memorizing: vocabulary mismatch, chunk boundary issues, missed top-k, reranker drops, not-extracted (lost-in-the-middle), wrong format, hallucination over context, stale index. None of these failures are about the pipeline shape; they are tuning failures within a fundamentally sound design.

The pipeline is the wrong shape for exactly two situations.

• Multi-hop questions. The answer requires evidence from documents reached through inference, not a single retrieval.
• Dataset-level questions. "What are the main themes?" cannot be answered by top-k over chunks.

Both belong in the agentic-RAG article.

Where to spend the next dollar

A roughly reliable upgrade order:

1. Hybrid (dense plus BM25 with RRF).
2. Cross-encoder reranker over a larger candidate pool.
3. Parent-child or recursive chunking.
4. Contextual or late chunking for long-form or ambiguous corpora.
5. A real eval set; only then tune embeddings, k, and prompts against it.

Each step ships in a day to a week and tends to move retrieval recall a few points. When further tuning stops paying back, the bottleneck has moved from retrieval to reasoning. That is the gate that pushes a system from semantic into agentic.

Neighbors

• Agentic RAG: retrieval under model control with Self-RAG, CRAG, Adaptive RAG, and tool-using loops.
• Semantic vs Agentic RAG: the side-by-side decision framework.
• The Augmented LLM: Retrieval, Tools, and Memory: the broader family RAG belongs to.

References and Good Reads

• Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The foundational RAG paper. NeurIPS 2020.
• Gao et al. Retrieval-Augmented Generation for Large Language Models: A Survey. The canonical naive / advanced / modular taxonomy.
• Anthropic. Introducing Contextual Retrieval. September 2024. The most useful single recent improvement.
• Günther et al. Late Chunking. Drop-in alternative when you have a long-context embedding model.
• Cormack, Clarke, Büttcher. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR 2009. The one-line formula behind every hybrid system.
• Gao et al. Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE). The pre-retrieval LLM rewrite.
• Santhanam et al. ColBERTv2. Late interaction; back in production use.
• Cohere. Rerank 3.5. Voyage. rerank-2 family. The two leading closed rerankers.
• Liu et al. Lost in the Middle. Why chunk order matters.
• Es et al. Ragas. The de facto LLM-as-judge eval framework.
• Krishna et al. FRAMES. Retrieval-plus-reasoning benchmark; useful even if you only run semantic RAG.
• Barnett et al. Seven Failure Points When Engineering a RAG System. The failure-mode checklist.
• LangChain RAG documentation. The current LCEL idioms.
• Anthropic Prompt caching. The single largest cost lever once a system is in production.