RAG Chunking and Retrieval Guide: Strategies, Metadata, and Evaluation Templates

Overview

Retrieval-augmented generation (RAG) combines search with language generation. A typical pipeline extracts source documents, divides them into chunks, creates embeddings, stores those embeddings with metadata, retrieves relevant chunks for a user question, and passes the selected context to a language model.

Chunking is the first major quality decision. Chunks that are too large may contain several unrelated subjects and consume unnecessary context. Chunks that are too small may lose definitions, conditions, or references needed to interpret an answer. There is no universally correct chunk size: the right choice depends on document structure, question types, embedding behavior, context limits, and the amount of evidence your application needs to return.

Evaluation should therefore measure retrieval separately from generation. An answer can be wrong because the retriever missed the relevant passage, because irrelevant passages crowded it out, or because the model misread adequate evidence. Testing these stages independently makes debugging more practical. For broader LLM prompt and response testing, see the LLM Prompt Testing evaluation framework.

Core indexing workflow

1. Define the retrieval unit. Decide whether users need paragraphs, procedures, product sections, policy clauses, code blocks, or another meaningful unit.
2. Ingest and normalize documents. Preserve headings, lists, tables, page references, URLs, and document identifiers where possible. Remove repeated headers, navigation text, and extraction artifacts.
3. Choose a chunking strategy. Start with a method that respects the source structure, then create measurable alternatives for comparison.
4. Attach metadata. Store fields that support filtering, citations, access control, debugging, and later reprocessing.
5. Generate embeddings. Record the embedding model and configuration alongside the indexed collection so results remain reproducible.
6. Store and index vectors. Confirm the vector dimension, distance metric, payload schema, and any limits imposed by the selected vector store.
7. Retrieve candidates. Test top-k values, filters, hybrid search if applicable, and reranking independently.
8. Evaluate with known questions. Compare retrieved chunks with expected evidence before judging the final generated answer.

Checklist by scenario

Fixed-size chunking

Fixed-size chunking divides text by a target number of tokens or characters, usually with overlap. It is easy to implement and provides a useful baseline for a first RAG prototype.

Use overlap only when it helps preserve continuity. Treat chunk length and overlap as experiment variables rather than permanent defaults.

Recursive chunking

Recursive chunking attempts larger separators first, such as headings or paragraphs, and falls back to smaller separators when a section exceeds the target. It is a practical general-purpose option for mixed prose.

Sentence-based chunking

Sentence chunking groups complete sentences until a target size is reached. It reduces mid-sentence cuts and can work well for FAQs, support content, and explanatory prose.

Semantic chunking

Semantic chunking groups text according to shifts in meaning rather than only length or punctuation. It can preserve topical boundaries, but its behavior depends on the segmentation method and thresholds.

Parent-child chunking

Parent-child chunking stores smaller child chunks for precise retrieval while retaining a larger parent section for generation. The child helps match a query; the parent supplies surrounding context.

Metadata field template

A consistent metadata schema makes filtering and diagnosis easier. Adapt the fields to your application rather than adding fields that will never be queried.

• document_id: stable identifier for the source document.
• source_uri: canonical URL, file path, or repository reference.
• title: document or page title.
• section_path: headings leading to the chunk, such as Guide > Installation > Configuration.
• chunk_id: stable identifier for the chunk version.
• parent_id: related parent section when using parent-child retrieval.
• document_type: handbook, ticket, API reference, policy, or another controlled value.
• language: detected or declared language.
• version and updated_at: source revision information.
• access_scope: permission or tenant boundary used for filtering.
• content_hash: value for detecting unchanged content during re-indexing.

What to double-check

Retrieval evaluation worksheet

Create a test set of representative questions before tuning the pipeline. Each record should include the question, expected document or passage, permitted answer evidence, and any required metadata filter.

Keep retrieval and generation labels separate. If recall is poor, revise ingestion, chunking, metadata filters, query handling, or search settings. If recall is adequate but faithfulness is poor, inspect context assembly and the generation prompt. The LLM hallucination troubleshooting guide can help organize that second investigation.

Index and model checks

• Confirm every vector in a collection was created with the intended embedding model and version.
• Verify that the stored vector dimension matches the model output and collection configuration.
• Check the distance or similarity metric used by the index and keep it consistent with your embedding setup.
• Test metadata filters separately from semantic search so an overly restrictive filter is not mistaken for poor embeddings.
• Measure the final context size after deduplication, parent expansion, and reranking.
• Record chunking parameters, extraction code version, embedding configuration, and index creation date for each corpus build.

Common mistakes

• Choosing chunk size by convention. A common starting value is not evidence that it fits your documents. Compare alternatives against representative questions.
• Discarding document structure. Removing headings, table labels, or page references makes both retrieval and citations harder.
• Embedding noisy extraction output. Repeated navigation, broken columns, and duplicated headers can attract queries away from the useful passage.
• Returning too many chunks. More context can introduce contradictions and reduce the prominence of the relevant evidence.
• Evaluating only final answers. A fluent answer does not prove that retrieval was correct. Inspect retrieved chunks and evidence coverage.
• Ignoring access boundaries. Metadata filters for tenant, role, or document permissions should be applied as part of retrieval, not as an afterthought.
• Re-indexing without traceability. If old and new chunks cannot be distinguished, regressions become difficult to reproduce.
• Changing embeddings without a comparison run. A new model or dimension may require a new collection and a side-by-side evaluation before migration.

When to revisit

Revisit this checklist whenever the corpus, question mix, or retrieval stack changes. At minimum, review it before a major planning or release cycle and after any workflow change that affects extraction, chunking, metadata, embeddings, search, or answer generation.

Run a focused evaluation when you add a new document type, update source formatting, change chunk sizes, switch embedding models, alter vector-store settings, introduce reranking, or observe a drop in answer quality. Keep a small regression set that represents common questions, difficult questions, permission-filtered questions, and questions whose answers span multiple sections.

A practical maintenance loop is:

1. Choose a documented corpus snapshot and evaluation set.
2. Record the current indexing and retrieval configuration.
3. Change one major variable at a time where possible.
4. Compare recall, precision, context relevance, and answer faithfulness.
5. Inspect failures manually and classify their cause.
6. Promote the change only when it improves the target use cases without creating unacceptable regressions.
7. Save the winning configuration and schedule the next review.

For production readiness beyond retrieval quality, pair this process with the LLM app production deployment checklist. Treat chunking and retrieval as versioned parts of the application, not one-time setup decisions.