RAG Systems: How to Build One and When You Actually Need It

An LLM only knows what it was trained on. Your internal documents, knowledge base, Jira tickets, contracts — it doesn’t know any of that. Fine-tuning is expensive and inflexible: every data update requires retraining. RAG (Retrieval-Augmented Generation) solves this differently — it pulls relevant fragments from external storage straight into the model’s context before generation.

The mechanics: a user asks a question, the system finds matching chunks from the knowledge base, passes them into the LLM prompt alongside the question, and the model generates an answer grounded in that context. No retraining needed. No need to keep everything in context either. Data updates in real time.

This article covers how to design a RAG system from scratch to production: architecture, component selection, retrieval strategies, advanced patterns, and a real cost breakdown.

What RAG Is and Why You Need It

RAG is an architecture pattern introduced by Facebook AI Research in 2020. The idea: a model’s parametric memory (what it “remembers” from training) gets supplemented with non-parametric memory — an external, searchable document store.

Three scenarios where RAG is essential:

Private data. The model doesn’t know the contents of your corporate wiki, internal documents, or support knowledge base. Fine-tuning on private data is expensive, slow, and the model can “forget” what it knew before (catastrophic forgetting). RAG plugs in any data corpus without retraining.

Freshness. The model is trained on data with a cutoff date. RAG pulls current data straight from storage. Update the docs, and RAG immediately uses the new version.

Citability. RAG returns not just an answer but sources. You can show users the exact document fragments the answer was generated from. For legal, medical, and financial applications, that’s a hard requirement.

Why Long Context Doesn’t Replace RAG

Modern models support context windows up to 1-2 million tokens. So the question comes up: why bother with RAG if you can just load all your documents into context?

Four reasons long context doesn’t replace RAG:

Cost. A million tokens of context on GPT-5.5 is $5 per request for input alone (GPT-5.5: $5/1M input). At 1,000 requests a day, that’s $5,000/day. RAG with embedding + retrieval costs two to three orders of magnitude less.

Latency. Longer context means longer generation time. 1M input tokens means several seconds of prefill alone. RAG returns 5-20 chunks — hundreds of tokens, not millions.

Needle-in-a-haystack. Models lose precision searching for a specific fact in long context — especially in the middle of a document. RAG does precision search and feeds the model only what’s relevant.

Data scale. A million tokens is roughly 750,000 words, or about 3,000 pages. For a corporate knowledge base of 50,000 documents, no context window fits that.

Rule of thumb: long context works for tasks where the entire corpus fits in the window and the cost is acceptable (analyzing a single document, working inside a codebase). For search over large, frequently updated data — use RAG.

RAG System Architecture

RAG consists of two pipelines: indexing (data preparation) and query (request processing).

Indexing Pipeline

Prepares documents for search. Runs once, plus incremental updates as data changes.

Documents → [Load] → [Split/Chunk] → [Embed] → [Store in Vector DB]

Load. Extracting text from sources: PDF, HTML, Markdown, Notion, Confluence, Google Docs. Tools: unstructured, docling, LangChain document loaders.

Split/Chunk. Breaking documents into fragments (chunks). Chunk size is a critical parameter — more on this in the chunking section.

Embed. Converting text chunks into numeric vectors (embeddings) via an embedding model. A fixed-dimension vector (768-3072) captures the semantic meaning of the text.

Store. Saving vectors and metadata into a vector database with indexes for fast similarity search.

Query Pipeline

Processes a user’s question. Runs on every request.

Query → [Embed] → [Retrieve top-k] → [Rerank] → [Generate with LLM]

Embed query. The same embedding model used during indexing converts the question into a vector.

Retrieve. Search for top-k nearest vectors in the database. Typical k values: 5-20. Distance metrics: cosine similarity, dot product, L2.

Rerank. Re-ranking retrieved chunks with a more precise model. Optional, but critical for quality.

Generate. Retrieved chunks are inserted into the LLM prompt alongside the question. The model generates an answer grounded in the provided context.

# Minimal query pipeline
async def rag_query(question: str, top_k: int = 5) -> str:
    # 1. Embed the question
    query_vector = await embed(question)

    # 2. Retrieve from vector store
    chunks = await vector_store.search(query_vector, limit=top_k)

    # 3. Build context
    context = "\n\n".join([c.text for c in chunks])

    # 4. Generate answer
    response = await llm.chat(
        system="Answer the question using only the provided context.",
        user=f"Context:\n{context}\n\nQuestion: {question}"
    )
    return response

Chunking: Document-Splitting Strategies

Chunking is the first, and one of the most influential, steps in the RAG pipeline. Chunks that are too small lose context. Chunks that are too large dilute relevance and waste tokens.

Fixed-Size Chunking

Splitting by a fixed number of characters or tokens with overlap.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_text(document)

Chunk size: 256-1024 tokens. The sweet spot for most tasks is 512 tokens.

Overlap: 10-20% of chunk_size. Overlap prevents context loss at chunk boundaries. If an important fact lands right on the seam between two chunks, overlap guarantees at least one of them contains it in full.

When to use: by default. Simple, predictable, works for 80% of cases.

Semantic Chunking

Splitting along semantic boundaries. The algorithm computes an embedding for each sentence, then finds points where semantic similarity between adjacent sentences drops sharply — those become chunk boundaries.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    OpenAIEmbeddings(model="text-embedding-3-small"),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=90
)
chunks = chunker.split_text(document)

When to use: documents with uneven structure, where fixed-size chunking cuts through logical blocks. Research papers, legal documents, long instructions.

Downside: more expensive (requires an embedding per sentence), less predictable chunk sizes.

Parent-Child (Hierarchical) Chunking

A two-level strategy. The document is split into large “parent” chunks (1,000-2,000 tokens) and small “child” chunks (200-400 tokens). Search runs against child chunks (more precise relevance), but the parent chunk gets inserted into the LLM’s context (more surrounding context).

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document
import hashlib

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)

# create_documents returns a list of Document objects with .page_content and .metadata
parent_docs = parent_splitter.create_documents([document])
for parent_doc in parent_docs:
    parent_id = hashlib.md5(parent_doc.page_content.encode()).hexdigest()[:8]
    child_docs = child_splitter.create_documents([parent_doc.page_content])
    for child_doc in child_docs:
        child_doc.metadata["parent_id"] = parent_id
        # Index child_doc in the vector store
        # On retrieval — return the parent_doc by parent_id

When to use: when you need a balance between search precision and context completeness. Technical documentation, FAQs, knowledge bases.

Late Chunking

An approach proposed by Jina AI. Instead of “chunk first, then embed” — run the whole document through a long-context embedding model (8192+ tokens) first, get contextualized token embeddings, then split into chunks and average. Each chunk retains context from the entire document. Jina Embeddings v4 supports 32K context and late interaction (ColBERT-style).

When to use: documents where context is critical for understanding individual fragments. Requires embedding models with long-context support (Jina Embeddings v4, voyage-4, Qwen3-Embedding).

Comparing Strategies

Recommendation: start with fixed-size (512 tokens, 64 overlap). Move to parent-child if retrieval finds the right chunks but the LLM lacks enough context to answer.

Embedding Models: Comparison and Selection

The embedding model is the core of a RAG system. It determines search quality: if the model encodes document semantics poorly, no amount of reranking will save you.

Key Parameters

Dimensions. Vector length. Larger means more precise, but more expensive to store and search. Modern models support Matryoshka embeddings: you can use truncated vectors (256, 512) with minimal quality loss.

Context window. Maximum input text length. For 512-token chunks, any model handles it fine. For late chunking, you need models with 8192+ tokens.

MTEB Score. A benchmark of embedding quality on retrieval, classification, and clustering tasks. Not the only criterion, but a useful reference point.

Model Comparison (June 2026)

How to Choose

Budget option: text-embedding-3-small ($0.02/1M tok) or jina-embeddings-v4 ($0.02/1M tok). Good enough for an MVP. text-embedding-3-small supports Matryoshka — you can use 256 dims to cut storage costs. Jina v4 adds multimodality (text + images + PDF) and 32K context.

Best balance: voyage-4 ($0.06/1M tok). Long context (32K), flexible dims 256-2048, Matryoshka + quantization. Owned by MongoDB — integrates well with Atlas Vector Search.

Maximum quality (managed): voyage-4-large ($0.12/1M tok) or Cohere Embed v4 ($0.12/1M tok). voyage-4-large leads on the RTEB retrieval benchmark. Cohere Embed v4 wins where you need multimodal RAG: text, images, PDF in one embedding space.

Self-hosted, open source: Qwen3-Embedding-8B. 32K context, 100+ languages, instruction-aware, flexible dims — one of the best open-source options on MTEB 2026 (70.6). Needs a GPU with 16 GB VRAM. For a lighter option — BGE-M3 (dense + sparse + ColBERT, 4 GB VRAM).

Multimodal RAG: Cohere Embed v4 ($0.12/1M text tokens, $0.47/1M image tokens) or Gemini Embedding 001 ($0.04/1M) for workloads already living in the Google ecosystem.

Vector Stores: Where to Store Embeddings

A vector store is a database optimized for similarity search over vectors. Two paths: extend an existing database (pgvector) or use a specialized solution (Pinecone, Qdrant).

Comparing Vector Stores

When to Choose What

pgvector (Supabase). PostgreSQL is already in your stack, you don’t want to add another database, and you’re working with up to 5-10M vectors. One backup story, one set of transactions, SQL filtering on metadata. pgvector 0.8+ with an HNSW index handles 1-5M vectors on ordinary Postgres hardware. Slower than specialized solutions at large scale and with selective filters.

Pinecone. Zero-ops: no index tuning, scaling, or backup configuration needed. Minimal DevOps. The serverless tier is convenient for getting started; pod-based is faster but pricier. Downside: vendor lock-in, no self-hosted option.

Qdrant. Best performance for filtered search (Rust implementation). Flexible filters, sparse vectors, payload indexing. Available as managed cloud or Docker for self-hosting. The best choice when you need maximum speed with filtering.

ChromaDB. Prototyping and small projects. Runs in-process, zero configuration. The fastest way to get started. Doesn’t hold up in production with millions of vectors.

LanceDB. Embedded/lakehouse-style vector DB. Good for multimodal data (text + images + video), offline scenarios, and integration with Apache Arrow and DuckDB. A ChromaDB alternative for prototypes needing a more flexible schema.

Recommendation: if you’re already on Supabase or PostgreSQL, start with pgvector. Don’t add infrastructure until pgvector hits its limits. For high-load scenarios with 10M+ vectors or complex filtering — Qdrant or Pinecone.

Retrieval: Search Strategies

RAG quality is determined by retrieval quality. If the system doesn’t find the right chunks, the LLM gets irrelevant context and generates a wrong answer. Garbage in, garbage out.

Vector Search (Baseline)

Cosine similarity between the query vector and chunk vectors. Fast, simple, but vulnerable to vocabulary mismatch: a query like “how to set up authentication” might miss a chunk that only mentions “OAuth 2.0 configuration.”

-- pgvector: cosine similarity search
SELECT id, content, 1 - (embedding <=> query_embedding) AS similarity
FROM documents
ORDER BY embedding <=> query_embedding
LIMIT 10;

Hybrid Search (BM25 + Vector)

Combines lexical search (BM25, exact keyword matching) with semantic search (vector similarity). BM25 catches exact terms, vector search catches semantically related ones.

-- pgvector + pg_trgm for hybrid search in PostgreSQL
WITH vector_results AS (
    SELECT id, content, 1 - (embedding <=> query_embedding) AS vector_score
    FROM documents
    ORDER BY embedding <=> query_embedding
    LIMIT 20
),
keyword_results AS (
    SELECT id, content, ts_rank(to_tsvector('english', content),
        plainto_tsquery('english', 'authentication OAuth')) AS keyword_score
    FROM documents
    WHERE to_tsvector('english', content) @@ plainto_tsquery('english', 'authentication OAuth')
    LIMIT 20
)
SELECT COALESCE(v.id, k.id) AS id,
    COALESCE(v.content, k.content) AS content,
    COALESCE(v.vector_score, 0) * 0.7 + COALESCE(k.keyword_score, 0) * 0.3 AS combined_score
FROM vector_results v
FULL OUTER JOIN keyword_results k ON v.id = k.id
ORDER BY combined_score DESC
LIMIT 10;

When to use: whenever you can — hybrid search consistently beats pure vector search by 5-15% on retrieval metrics.

Reciprocal Rank Fusion (RRF)

An algorithm for merging results from multiple retrieval strategies. Doesn’t require score normalization — it works with ranks.

def reciprocal_rank_fusion(results_lists: list[list], k: int = 60) -> list:
    """
    results_lists: a list of N ranked document lists
    k: smoothing constant (standard: 60)
    """
    scores = {}
    for results in results_lists:
        for rank, doc in enumerate(results):
            if doc.id not in scores:
                scores[doc.id] = 0
            scores[doc.id] += 1 / (k + rank + 1)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

RRF is the de facto standard for fusion in hybrid search. Simple, effective, no weight training required.

Multi-Query Retrieval

A single user question gets rephrased into 3-5 variations, each searched separately, and results merged via RRF. Solves vocabulary mismatch at the query level.

async def multi_query_retrieve(question: str, top_k: int = 10) -> list:
    # LLM generates variations of the question
    variations = await llm.chat(
        system="Generate 3 different phrasings of this question for searching a knowledge base.",
        user=question
    )

    all_results = []
    for q in [question] + variations:
        results = await vector_store.search(await embed(q), limit=top_k)
        all_results.append(results)

    return reciprocal_rank_fusion(all_results)

Cost: 3-5 extra embedding calls plus one LLM call to generate variations. At $0.06/1M tok (voyage-4) and $0.02/1M tok (embedding) — fractions of a cent per query.

HyDE (Hypothetical Document Embeddings)

Instead of directly embedding the query, the LLM generates a hypothetical answer to the question, and that answer gets embedded for search. The idea: a hypothetical answer is semantically closer to a real document than a short question is.

async def hyde_retrieve(question: str, top_k: int = 10) -> list:
    # LLM generates a hypothetical answer
    hypothetical = await llm.chat(
        system="Write a detailed answer to the question as if you were an expert. Don't mention that this is a hypothetical answer.",
        user=question
    )

    # Embed the hypothetical answer, not the question
    hyde_vector = await embed(hypothetical)
    return await vector_store.search(hyde_vector, limit=top_k)

When to use: for complex, abstract questions where the direct query vector sits too far from the documents. Not recommended for factoid questions — the LLM can hallucinate a hypothetical answer in the wrong direction.

Comparing Retrieval Strategies

*HyDE: high precision for conceptual queries, medium for factoid.

Reranking: A Second Pass for Precision

Retrieval returns top-k candidates. A bi-encoder (embedding model) is a speed/precision tradeoff: it encodes the query and document independently. A cross-encoder (reranker) processes the “query + document” pair jointly — more precise, but slower.

Typical architecture: a bi-encoder finds the top 50 candidates in milliseconds, and a cross-encoder re-ranks them down to the top 5 in 100-300 ms.

Why You Need Reranking

Without reranking, retrieval results contain noise. Vector search can return chunks that are semantically close to the query on surface features but don’t actually contain the answer. A reranker evaluates relevance more deeply — at the level of specific facts and claims.

Practical effect: reranking lifts precision@5 by 10-25%, depending on the task.

Available Rerankers

Cohere Rerank 4 (rerank-v4.0-pro and rerank-v4.0-fast) — managed API, $2.00 per 1,000 searches (1 search = 1 query + up to 100 documents). Multilingual, 32K context, best quality among managed solutions.

Voyage Rerank 2.5 (rerank-2.5 and rerank-2.5-lite) — integrated with Voyage AI, pairs with Voyage embeddings. 32K context, instruction-following, multilingual.

BGE Reranker v2-m3 — open-source cross-encoder from BAAI. Runs locally on GPU. Free, but limited to 8K tokens — not the best choice for long-document RAG.

Qwen3-Reranker (0.6B, 4B, 8B) — open source from Alibaba. 32K context, 100+ languages, instruction-aware. Strong results on MTEB-R (69.76) and CMTEB-R (75.94). A good BGE replacement for multilingual scenarios.

Cohere Rerank 4 (self-hosted) — available via AWS Bedrock and Azure AI for full data control.

Implementation

import cohere

co = cohere.Client()

async def retrieve_and_rerank(question: str, top_k: int = 5) -> list:
    # Step 1: Broad retrieval (top-50)
    query_vector = await embed(question)
    candidates = await vector_store.search(query_vector, limit=50)

    # Step 2: Rerank down to top-k
    reranked = co.rerank(
        model="rerank-v4.0-pro",
        query=question,
        documents=[c.text for c in candidates],
        top_n=top_k
    )

    return [candidates[r.index] for r in reranked.results]

In practice: retrieve 50-100 candidates → rerank → top 5-10 into the LLM. Reranking costs $2 per 1,000 requests and lifts precision@5 by 10-25%. Skipping it only makes sense under a hard latency budget (<100 ms total).

Advanced RAG: Next-Level Patterns

Basic RAG (retrieve → generate) covers 70% of use cases. For the other 30%, advanced patterns let the system decide how and what to search. In production, two are most valuable: Adaptive RAG (saves cost on simple queries) and Agentic RAG (adds power for complex ones).

Agentic RAG

Instead of a fixed “embed → search → generate” pipeline, an LLM agent decides what actions to take. The agent can:

• Rephrase the query if the first retrieval comes back empty
• Pull data from multiple sources (Wiki + CRM + Jira)
• Run multi-step reasoning: find fact A, use it to formulate query B, find fact B
• Decide retrieval isn’t needed at all — the answer already lives in the model’s parametric memory

from openai import OpenAI

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_knowledge_base",
            "description": "Search the company knowledge base",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "source": {"type": "string", "enum": ["docs", "jira", "confluence"]}
                }
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "search_web",
            "description": "Search the web for current information",
            "parameters": {
                "type": "object",
                "properties": {"query": {"type": "string"}}
            }
        }
    }
]

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": question}],
    tools=tools,
    tool_choice="auto"  # The agent decides which tool to call
)

When to use: complex questions requiring reasoning; multi-source search; tasks where retrieval sometimes isn’t needed at all.

Self-RAG

After generating a response, the model evaluates its own answer:

1. Does this question need retrieval? (retrieve / no-retrieve)
2. Are the retrieved documents relevant? (relevant / irrelevant)
3. Is the answer supported by the sources? (supported / contradicted / no info)
4. Is the answer useful? (useful score 1-5)

If the answer isn’t supported, the system retries retrieval with a different query, or admits it can’t answer.

async def self_rag(question: str) -> str:
    # Step 1: Determine whether retrieval is needed
    need_retrieval = await llm.chat(
        system="Determine whether a knowledge base search is needed to answer this question. Reply 'yes' or 'no'.",
        user=question
    )

    if need_retrieval.strip() == "no":
        return await llm.chat(user=question)

    # Step 2: Retrieve and generate
    chunks = await retrieve_and_rerank(question)
    context = "\n\n".join([c.text for c in chunks])
    answer = await llm.chat(
        system=f"Context:\n{context}",
        user=question
    )

    # Step 3: Check faithfulness
    check = await llm.chat(
        system="Assess whether the answer is supported by the provided context. Reply 'supported', 'contradicted', or 'no_info'.",
        user=f"Context:\n{context}\n\nAnswer:\n{answer}"
    )

    if check.strip() != "supported":
        # Retry retrieval with a rephrased query
        return await self_rag_retry(question, previous_answer=answer)

    return answer

Adaptive RAG

In production, a RAG system often spends the same resources on a simple question (“what’s the office address?”) as on a complex multi-hop query (“find all projects tied to client X that are affected by vulnerability CVE-2025-1234”). Adaptive RAG fixes this with routing.

The system classifies each query and picks a strategy:

async def adaptive_rag(question: str) -> str:
    # Classify the query
    route = await llm.chat(
        system="""Determine the search strategy for this question.
Reply with one word:
- simple: a fact from a single document
- hybrid: needs keyword + semantic search
- multi_hop: multiple documents, a reasoning chain
- no_retrieval: a general question with no private data involved""",
        user=question
    )

    if route == "no_retrieval":
        return await llm.chat(user=question)
    elif route == "simple":
        chunks = await vector_store.search(await embed(question), limit=5)
    elif route == "hybrid":
        chunks = await hybrid_search(question, limit=10)
    elif route == "multi_hop":
        chunks = await multi_query_retrieve(question, top_k=15)

    chunks = await rerank(question, chunks, top_k=5)
    return await generate(question, chunks)

Practical effect: 60-70% of queries in enterprise RAG systems are simple factoid questions. Adaptive RAG routes them down the cheap path (vector search without multi-query or heavy reranking), cutting latency and cost without sacrificing quality on complex queries.

When to use: production RAG with heterogeneous query mixes — factoid questions, analytical queries, and multi-hop reasoning all in the same system. Pairs well with Agentic RAG: an agent is really an extreme case of adaptive routing.

Corrective RAG (CRAG)

A modification of Self-RAG with external verification. If the internal knowledge base doesn’t yield a confident answer, the system reaches out to external sources (web search) to correct course.

Three scenarios:

• Correct: retrieval found relevant documents → generate from them
• Incorrect: documents are irrelevant → web search → generate from web results
• Ambiguous: partial relevance → combine internal + web results

GraphRAG

RAG built on a knowledge graph instead of (or alongside) vector search. Documents get converted into a graph of entities and relationships. A query becomes a graph traversal.

Microsoft GraphRAG supports several search modes:

• Local Search — questions about specific entities.
• Global Search — questions “about the whole corpus” (summarization, corpus-level patterns).
• DRIFT Search — a Local + Global hybrid: starts from community summaries, then refines through follow-up traversal. Balances quality and cost via flexible graph traversal.

Alternatives:

• LightRAG — a lighter open-source GraphRAG approach, simpler for prototypes.
• Neo4j GraphRAG — a good fit if you already run Neo4j / Cypher.
• LlamaIndex Property Graph — convenient if you’re already inside the LlamaIndex stack.

When you need GraphRAG:

• Questions about relationships between entities (“which of our clients is connected to company X?”)
• Multi-hop reasoning (“find every project that depends on a library built by team Y”)
• Summarizing large corpora through hierarchical community summaries

Complexity and cost: building a knowledge graph is an expensive operation. Entity extraction plus community summaries require many LLM calls at indexing time. You need NER, relation extraction, entity resolution, plus prompt tuning to get a usable graph.

Recommendation: GraphRAG complements vector search, it doesn’t replace it. Start with vector RAG, and add a graph layer only for multi-hop and corpus-level tasks that vector search genuinely can’t handle.

Hands-On Tutorial: RAG with Supabase pgvector + Python

A complete working example of a RAG system built from scratch. Stack: Supabase (PostgreSQL + pgvector), OpenAI embeddings, Python.

Step 1: Setting Up Supabase

Enable the pgvector extension in Supabase Dashboard → Database → Extensions → search for vector.

Create the table and search function:

-- Create the documents table
CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    metadata JSONB DEFAULT '{}',
    embedding VECTOR(1536)
);

-- Index for fast search (HNSW — the primary choice up to 1M vectors)
CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- Function for similarity search
CREATE OR REPLACE FUNCTION match_documents(
    query_embedding VECTOR(1536),
    match_threshold FLOAT DEFAULT 0.7,
    match_count INT DEFAULT 5
)
RETURNS TABLE (
    id BIGINT,
    content TEXT,
    metadata JSONB,
    similarity FLOAT
)
LANGUAGE plpgsql
AS $$
BEGIN
    RETURN QUERY
    SELECT
        documents.id,
        documents.content,
        documents.metadata,
        1 - (documents.embedding <=> query_embedding) AS similarity
    FROM documents
    WHERE 1 - (documents.embedding <=> query_embedding) > match_threshold
    ORDER BY documents.embedding <=> query_embedding
    LIMIT match_count;
END;
$$;

Step 2: Python — Indexing Documents

import os
from openai import OpenAI
from supabase import create_client

# Initialize clients
openai_client = OpenAI()
supabase = create_client(
    os.environ["SUPABASE_URL"],
    os.environ["SUPABASE_SERVICE_KEY"]
)

def get_embedding(text: str) -> list[float]:
    """Get an embedding via the OpenAI API."""
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

def chunk_text(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
    """Split text into overlapping chunks."""
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

def index_document(text: str, metadata: dict = None) -> int:
    """Split a document into chunks and index them."""
    chunks = chunk_text(text)
    rows = []
    for i, chunk in enumerate(chunks):
        embedding = get_embedding(chunk)
        rows.append({
            "content": chunk,
            "metadata": {**(metadata or {}), "chunk_index": i},
            "embedding": embedding
        })

    result = supabase.table("documents").insert(rows).execute()
    return len(result.data)

Step 3: Python — Search and Generation

def search(query: str, top_k: int = 5, threshold: float = 0.7) -> list[dict]:
    """Semantic search over the document store."""
    query_embedding = get_embedding(query)

    result = supabase.rpc("match_documents", {
        "query_embedding": query_embedding,
        "match_threshold": threshold,
        "match_count": top_k
    }).execute()

    return result.data

def ask(question: str, top_k: int = 5) -> str:
    """RAG: retrieve context + generate an answer."""
    # Retrieve
    chunks = search(question, top_k=top_k)

    if not chunks:
        return "No relevant documents found to answer this question."

    # Build context
    context = "\n\n---\n\n".join([
        f"[Source {i+1}] (similarity: {c['similarity']:.2f})\n{c['content']}"
        for i, c in enumerate(chunks)
    ])

    # Generate
    response = openai_client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are an assistant answering questions based on the provided context. "
                    "Use only the information in the context. "
                    "If the answer isn't in the context, say so. "
                    "Cite sources as [Source N]."
                )
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {question}"
            }
        ],
        temperature=0.1
    )

    return response.choices[0].message.content

Step 4: Usage

# Indexing
with open("knowledge_base.md") as f:
    text = f.read()

count = index_document(text, metadata={"source": "knowledge_base", "version": "1.0"})
print(f"Indexed {count} chunks")

# Asking a question
answer = ask("How do I set up authentication in the project?")
print(answer)

The entire code is about 70 lines (excluding comments). A working RAG system with semantic search and source citation. You can add metadata filtering through match_documents — pass a filter condition into the SQL function as a parameter.

Evaluation: Measuring RAG Quality

RAG without metrics is a black box. Answers look plausible, but without measurement you have no idea whether a new chunking strategy is improving or degrading results.

Retrieval Metrics

Precision@k — the share of relevant documents among the top-k results. Precision@5 = 0.6 means 3 of the 5 returned chunks are relevant.

Recall@k — the share of relevant documents found out of all relevant documents in the corpus. Recall@10 = 0.8 means retrieval found 8 out of 10 relevant chunks in the database.

MRR (Mean Reciprocal Rank) — the average inverse rank of the first relevant result. MRR = 1 means the first result is always relevant. MRR = 0.5 means the relevant result lands, on average, at position two.

NDCG@k (Normalized Discounted Cumulative Gain) — accounts not just for the presence of relevant documents, but their position. A relevant document at position 1 is worth more than one at position 5.

Generation Metrics

Faithfulness — how well the answer matches the context. The answer contains no information absent from the sources (no hallucinations). Range: 0-1.

Answer Relevancy — how well the answer matches the question. An answer can be faithful (grounded in context) but irrelevant (doesn’t answer the question).

Context Precision — how much of the retrieved context contains information actually needed for the answer. Low precision means a lot of noise in the context.

Context Recall — whether the retrieved context contains all the information needed for a complete answer. Low recall means part of the answer is missing from the context.

The RAGAS Framework

RAGAS is an open-source framework for automatically evaluating RAG systems. An LLM acts as judge. Most metrics are reference-free (don’t require ground-truth answers): faithfulness, answer_relevancy, context_precision. The exception is context_recall: this metric compares context against ground_truth, so reference answers are required.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)
from datasets import Dataset

# Prepare evaluation data
eval_data = {
    "question": ["How do I set up OAuth?", "What are the API rate limits?"],
    "answer": [rag_answer_1, rag_answer_2],
    "contexts": [[chunk1, chunk2], [chunk3, chunk4]],
    "ground_truth": ["OAuth is configured via...", "API rate limits: 1000 req/min..."]
}

dataset = Dataset.from_dict(eval_data)

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)

print(results)
# {'faithfulness': 0.92, 'answer_relevancy': 0.88,
#  'context_precision': 0.85, 'context_recall': 0.78}

What to Measure in Production

A minimal metric set for production RAG:

1. Faithfulness — the primary metric. Hallucinations in RAG are unacceptable.
2. Answer Relevancy — the answer needs to actually address the question.
3. Retrieval Precision@5 — search quality.
4. Latency p95 — time from request to answer.
5. Cost per query — the cost of a single request (embedding + retrieval + generation).

Recommendation: build the eval pipeline before optimizing. Without metrics, changes to chunking, retrieval, or prompts are blind experiments.

When You Don’t Need RAG

RAG isn’t a silver bullet. There are scenarios where it’s overkill or loses to alternatives.

Long Context Windows

If the entire data corpus fits in the context window and the cost is acceptable, you don’t need RAG.

Example: analyzing a single 50-page document (~25K tokens). Loading it whole is cheaper than building a RAG pipeline. Cost: ~$0.019 per request with GPT-5.4-Mini ($0.75/1M input).

Prompt Caching

Providers (Anthropic, OpenAI, Google) cache repeated context prefixes. If you’re asking different questions against the same document, cached context costs 10x less than full context (90% discount).

Example: 100K tokens of documentation, 100 questions a day. Without caching (GPT-5.5, $5/1M): 100K × 100 × $5/1M = $50/day. With prompt caching ($0.50/1M cached): about $5/day plus fractions of a cent for the unique part of each request. More expensive than RAG at scale, but simpler to implement.

Small Corpus

A corpus of 10-50 documents under 200K tokens. Loading everything into context via prompt caching is simpler than maintaining RAG infrastructure.

Fine-Tuning

If the data is static and the task isn’t “answer a question about a document” but “behave a certain way” (style, format, domain expertise), fine-tuning handles that better than RAG.

Decision Framework

Corpus > 500K tokens?
  ├─ Yes → RAG
  └─ No
      ├─ Does the data update frequently? → RAG
      ├─ Do you need source citations? → RAG
      ├─ Is your inference budget constrained? → RAG
      └─ None of the above → Long context + caching

The Cost of RAG in Production: A Real Calculation

Let’s calculate the cost of a RAG system for a real scenario: a corporate knowledge base, 10,000 documents, 1,000 queries a day.

Parameters

• 10,000 documents, average size 2,000 words (~2,700 tokens)
• Chunking: 512 tokens, 64 overlap → ~6 chunks per document → 60,000 chunks
• Embedding model: text-embedding-3-small (1536 dims, $0.02/1M tok)
• Vector store: Supabase Pro ($25/month)
• Reranker: Cohere Rerank 4 ($2/1,000 searches)
• LLM: GPT-5.4-Mini ($0.75/1M input, $4.50/1M output)
• 1,000 queries/day

One-Time Costs (Indexing)

Monthly Costs

Comparison with Long Context

The same corpus (27M tokens) via long context + GPT-5.4-Mini ($0.75/1M input):

RAG is three to four orders of magnitude cheaper than full long context, and roughly 280x cheaper than cached context at 1,000 queries a day. The gap widens as load grows.

Cost Optimization

Embedding: switching to voyage-4 ($0.06/1M) gives better quality, but the difference in query embedding cost is negligible at 1,000 req/day. The real savings show up during bulk re-indexing.

Reranking: $60/month — the biggest line item after the LLM itself. If budget is tight, swap in self-hosted BGE Reranker (free, but needs a GPU).

LLM: GPT-5.4-Mini ($0.75/1M input) is a sensible default for most tasks. For complex multi-hop queries — GPT-5.5 ($5/1M input) or Gemini 3.5 Flash — the choice depends on query complexity and budget.

Batch embedding: the OpenAI Batch API gives a 50% discount on embeddings. Use it for bulk indexing runs.

Bottom Line

RAG isn’t a stopgap hack until models learn to remember everything. It’s an architecture pattern that solves fundamental problems: data privacy, freshness, citability, and inference cost.

A minimal working configuration:

1. Chunking: fixed-size, 512 tokens, 64 overlap
2. Embedding: text-embedding-3-small (for MVP) or voyage-4 (for production)
3. Vector store: pgvector via Supabase
4. Retrieval: hybrid search (BM25 + vector)
5. Reranking: Cohere Rerank 4 (rerank-v4.0-pro / fast)
6. Generation: GPT-5.4-Mini with a prompt that requires citations
7. Evaluation: RAGAS (faithfulness + answer relevancy at minimum)
8. Routing (production): Adaptive RAG — a query classifier to cut latency and cost on simple queries

Start simple: fixed-size chunking, one vector store, basic vector search. Measure quality with RAGAS. Add hybrid search, reranking, and advanced strategies iteratively — measuring impact at each step. A RAG system you can’t evaluate is a system you don’t control.