Production RAG: Chunking, Hybrid Search, Reranking and Evaluation

Treat retrieval as a product dependency

A retrieval-augmented generation demo often succeeds because the author asks questions whose answers are obvious in a small document collection. Production users ask incomplete questions, use different terminology, lack access to some documents and expect yesterday's policy change to be reflected immediately.

This guide is for engineers building production RAG and decision makers evaluating its reliability. The objective is a pipeline that can explain which evidence it retrieved, whether the user was allowed to see it, why an answer was supported and what happens when the corpus does not contain a useful answer.

Key takeaways

• Preserve document identity, versions, permissions and deletion state at ingestion.
• Choose chunks around meaning and retrieval needs, not one universal token count.
• Evaluate lexical retrieval, vector retrieval and fusion separately.
• Use reranking to improve a candidate set, not to repair missing evidence.
• Measure retrieval quality and answer grounding as different stages.

Build an ingestion path you can replay

Start with the source of truth. Each document needs a stable identity, source location, version or content hash, ownership and access metadata. Record when the source changed and when the index incorporated that change. Without both timestamps, a freshness complaint cannot be separated into source delay, ingestion delay or stale cache behavior.

Parsing should preserve useful structure. Headings, numbered steps, table relationships and code boundaries affect interpretation. A PDF extraction that interleaves two columns can produce plausible but false statements when a model reads the result. Sample parsed documents visually and test representative formats before treating a successful parser exit as proof of usable content.

+----------+   +----------------+   +------------------+
| Sources  |-->| Parse + version|-->| Chunk + metadata |
+----------+   +----------------+   +--------+---------+
                                            |
                              +-------------v-------------+
                              | Lexical and vector indexes|
                              +-------------+-------------+
                                            |
+----------+   +----------------+   +--------v---------+
| Question |-->| Scope + filters|-->| Retrieve + fuse  |
+----------+   +----------------+   +--------+---------+
                                            |
                              +-------------v-------------+
                              | Rerank + evidence budget  |
                              +-------------+-------------+
                                            |
                              +-------------v-------------+
                              | Answer + citation checks  |
                              +---------------------------+

Make ingestion idempotent. Reprocessing the same source version should not create duplicate chunks. When a document changes, replace or retire the old version deliberately. When it is deleted or access is revoked, remove or filter its chunks and invalidate any answer caches that might expose the old content.

A useful chunk identifier combines document identity, source version and a stable location within the parsed representation. Content hashes detect changed text, but a hash alone does not express ownership or source location. Store enough provenance to display a citation and investigate a disputed answer without reverse engineering the embedding.

Separate publication from preparation

Build a new index version in an isolated namespace when changing parsing or embedding strategy. Validate coverage, permissions and representative queries before switching traffic. This avoids serving a partially rebuilt corpus in which some documents use one representation and others use another accidentally.

Chunk for the question and the source

Chunk size is a trade-off between focused retrieval and sufficient context. A short paragraph can match a query precisely while omitting the heading that defines its scope. A very large chunk may include the answer but spend most of the context budget on unrelated material. Choose a starting policy, then inspect failures by document type.

Prefer structural boundaries when available. Keep a procedure's prerequisite with its steps, repeat relevant table headers when splitting rows, and preserve a function's signature with its explanation. If overlap is used, deduplicate repeated evidence before generation so several nearly identical chunks do not crowd out another relevant source.

The following runnable Python example splits paragraphs into bounded groups using words as an intentionally simple proxy. It is an ingestion sketch, not a tokenizer-aware production chunker. It deliberately raises on oversized paragraphs so the caller must choose an explicit secondary splitting policy.

def paragraph_chunks(text, max_words=120):
    current, size = [], 0
    for paragraph in text.split("\n\n"):
        paragraph = paragraph.strip()
        if not paragraph:
            continue
        count = len(paragraph.split())
        if count > max_words:
            raise ValueError("paragraph needs secondary splitting")
        if current and size + count > max_words:
            yield "\n\n".join(current)
            current, size = [], 0
        current.append(paragraph)
        size += count
    if current:
        yield "\n\n".join(current)

sample = "Returns require an order ID.\n\nKeep the receipt."
print(list(paragraph_chunks(sample, max_words=8)))

Attach metadata consistently: document ID, version, section path, language, tenant, permission group and source URL where applicable. Keep display metadata separate from authorization metadata. A model-generated document summary should never determine whether a user may retrieve the document.

A parent-child strategy can retrieve smaller chunks and then load a larger surrounding section. This works when local context matters, but the expansion step must use the same permission checks and budget limits as initial retrieval. Track which small chunk triggered expansion so the result remains explainable.

Combine retrieval signals deliberately

Lexical search is useful for exact terms, identifiers, product codes and unusual names. Vector search is useful when the question and answer use different wording. Neither is automatically better across an entire corpus. Evaluate them against queries that represent both exact lookup and conceptual explanation.

Hybrid retrieval combines candidate lists. Avoid adding raw lexical and vector scores unless you understand their scales and have calibrated them. Rank-based fusion offers a simple starting point because it combines positions rather than assuming comparable scores. The constants and candidate limits still require evaluation on your own data.

This runnable example implements reciprocal rank fusion for document identifiers. The value of k is illustrative. Duplicate identifiers within a single result list are ignored so that one retriever cannot accidentally vote for the same document multiple times.

from collections import defaultdict

def fuse(rankings, k=60):
    scores = defaultdict(float)
    for ranking in rankings:
        seen = set()
        for rank, document_id in enumerate(ranking, start=1):
            if document_id in seen:
                continue
            seen.add(document_id)
            scores[document_id] += 1 / (k + rank)
    return sorted(scores, key=lambda key: (-scores[key], key))

lexical = ["doc-a", "doc-c", "doc-b"]
semantic = ["doc-b", "doc-a", "doc-d"]
print(fuse([lexical, semantic]))

Retrieval approach	Strength	Main limitation
Lexical	Exact terminology and identifiers	Misses some paraphrases
Vector	Semantic similarity	Can miss exact constraints or rare codes
Hybrid fusion	Combines different candidate signals	Adds tuning and operational work
Reranking	Compares query and candidates more closely	Cannot recover a missing candidate

Apply access filters as part of retrieval, before candidates reach a reranker or model. Filtering only the final answer still exposes restricted text to intermediate systems. Validate filter behavior with adversarial tests, including users with no permissions and documents whose access changed after indexing.

When selecting a database, examine metadata filtering, index update behavior, deletion semantics and operational visibility alongside similarity search. Those concerns are central to vector database integration, because the index is part of the application's authorization and freshness path.

Rerank and assemble useful evidence

Reranking should receive a bounded candidate set and return an ordered list with stable identifiers. Measure whether it improves relevant evidence near the top of the list and whether the improvement justifies its latency and cost. A reranker that sees the wrong language or truncated documents may produce confident but unhelpful ordering.

Assemble the context using a token budget that leaves room for instructions, the user request and the answer. Prefer coverage over repetition. Several chunks supporting the same fact may be less useful than one supporting chunk plus another that explains an exception. Keep source labels near the text so citations can be mapped back reliably.

The generation prompt should distinguish instructions from source excerpts and explicitly permit an insufficient-evidence response. That is necessary but not sufficient. Retrieved documents remain untrusted content and may contain instructions aimed at the model. Keep tool permissions and external actions outside the authority of those documents.

Validate citations after generation. Check that each cited identifier belongs to the supplied evidence set and that the relevant passage supports the associated claim. A citation that merely points to a real document is not necessarily a supported citation. For high-impact answers, require review or a narrower answer format with extractive evidence.

Handle conflicting and absent evidence

If two policy versions disagree, the system should use explicit source authority and effective dates rather than selecting whichever chunk ranks first. Preserve the conflict when the authoritative source cannot be identified. An honest answer explaining the ambiguity is more useful than a synthetic compromise that appears in no source.

Do not use an arbitrary similarity threshold as a universal truth test. Scores vary by embedding model, corpus and query type. Calibrate any threshold against labeled cases, including questions that have no answer in the corpus. Review false refusals as well as unsupported answers.

Evaluate and operate the complete pipeline

Build a question set with expected evidence and answer criteria. Include exact identifiers, paraphrases, multi-document questions, outdated information, permission restrictions and unanswerable questions. Keep a held-out set that is not used for tuning chunk sizes or prompts, otherwise the evaluation becomes a record of optimization effort rather than a credible release check.

Measure retrieval recall at the candidate stage, ranking quality after reranking and evidence coverage after context assembly. Then evaluate answer correctness, citation support, completeness and appropriate refusal. Separating these stages tells you whether to fix ingestion, search, context selection or generation.

The LLM evaluation guide develops the release and trace workflow around these measurements. Track corpus and configuration versions with every evaluation run. A score change after a document refresh is not necessarily caused by the model. Preserve the query, expected evidence version, parser version, chunking policy, embedding configuration, retriever settings and generator configuration so comparisons are reproducible.

Run a deletion and freshness drill

Choose a staging document, ask a question it answers and record the evidence identifiers. Update the document and verify that the new version becomes searchable within your intended freshness window. Then revoke access and confirm that the same user can no longer retrieve it through lexical search, vector search, reranking inputs or cached answers.

Repeat the drill with a failed ingestion worker and a partially completed rebuild. The system should expose the stale state and avoid presenting mixed index versions as a single consistent corpus. Document whether callers receive an older approved version, a reduced answer or a temporary unavailable response. That operational decision belongs in the product contract.

For production monitoring, group failures by query type and corpus segment. A global average can conceal a broken parser for one file format or missing documents for one tenant. Sample traces for manual review, with redacted content and strict retention. Connect alerts to actions such as pausing an index rollout, reprocessing a source or disabling an unsafe cache.

Common mistakes

• Choosing one chunk size before examining source structure or retrieval failures.
• Combining uncalibrated similarity scores as though they share a scale.
• Expecting a reranker to recover documents that retrieval never returned.
• Checking permissions only after restricted evidence reached the model.
• Counting valid-looking citations without checking whether they support the claim.

Checklist

• Preserve source identity, versions, permissions and deletion markers.
• Inspect parsed output and chunk boundaries for representative documents.
• Compare lexical, vector and hybrid retrieval on labeled questions.
• Bound reranking and context budgets while preserving evidence diversity.
• Evaluate retrieval, grounding, refusal and citation support separately.
• Test updates, revocation, deletion and interrupted index rebuilds.
• Record pipeline versions and define a rollback path.

FAQ

Is a vector database enough for production RAG?

No. It is one retrieval component within a larger ingestion, authorization, generation and evaluation system. Exact-term search, metadata filtering and reliable update behavior may matter as much as semantic similarity.

What is the best chunk size?

There is no universal value. Start from document structure and the questions users ask, then compare retrieval coverage and answer quality across representative examples.

Should every RAG pipeline use reranking?

Use it when evaluation shows a useful improvement within your latency and cost constraints. A small corpus with precise retrieval may not need it, while a broad candidate set may benefit substantially.

How should RAG answer an unsupported question?

It should explain that the available evidence is insufficient and identify what information is missing when possible. Avoid filling the gap with an uncited statement that sounds like it came from the corpus.

Final thoughts

Production RAG is an evidence delivery system. Its reliability depends on preserving meaning, permissions and provenance from the source document through the final answer, with enough evaluation to identify which stage needs attention.

To discuss a pipeline for your corpus, explore my RAG development service and book a conversation.