Designing a RAG pipeline: chunking and indexing

What you’ll be able to do

• Choose a chunking strategy that follows document structure rather than a fixed character count
• Preserve context in chunks through headings, contextual prefixes and parent-document links
• Design index metadata that supports filtering, access control, freshness and provenance
• Plan ingestion as a versioned pipeline so refreshes, re-embedding and deletions stay consistent

What you need to know

Retrieval quality is decided at ingestion

When a RAG system gives a confident wrong answer, the model is usually reasoning faithfully over the wrong context. Most of what decides whether the right context comes back is settled before any query arrives: how documents were split, what each chunk carries with it, and how the index was built and refreshed. Professional-level questions often describe a symptom ("answers got worse after the knowledge-base migration") and expect you to look at the pipeline first, not the prompt or the model.

Chunk along the document’s structure

Fixed-size chunking (every N tokens, some overlap) is easy and often the baseline, but it ignores meaning. It splits a table from its header, a procedure from its prerequisites, or a contract clause from the definition it relies on. Better strategies follow the document's own shape:

• Structural chunking. Split on headings, sections, list items, or one record per chunk for semi-structured data such as FAQs, tickets or product entries.
• Type-specific handling. Keep tables whole (or convert rows to self-describing text with column names), keep code functions intact, and split long transcripts by speaker turn or topic shift.
• Size by the query, not the document. Small chunks are precise for fact lookups but lose context. Large chunks keep context but dilute relevance and spend tokens. The right size depends on what users ask, so test a few on your own queries.
• Overlap as a patch, not a fix. Overlap reduces boundary losses with fixed-size splitting, but it duplicates content and doesn't fix splits that ignore meaning.

Make every chunk stand on its own

A chunk that says "the limit is raised to the amount above for enterprise customers" is useless out of context, both for the embedding and for the model reading it. Techniques that keep context attached:

• Heading paths. Prepend the document title and section hierarchy ("Travel Policy › International › Per-diem rates").
• Contextual prefixes. Generate a short, chunk-specific description of where the chunk sits in the document and add it before embedding and keyword indexing. Anthropic has described this approach as contextual retrieval. Prompt caching keeps the cost of generating the prefixes manageable, because the full document is the stable part of every request.
• Parent-document (small-to-big) retrieval. Index small chunks for precise matching, but return the enclosing section or neighbours to the model so it sees the surrounding context.

Metadata is half the index

Every chunk should carry structured metadata alongside its vector and text:

• Provenance: source document ID, URL or path, section, page. Needed for citations and for tracing a bad answer back to its source.
• Version and freshness: document version, effective date, ingestion timestamp. Lets retrieval prefer current policy over superseded policy, and lets you find stale chunks.
• Access control: tenant, team, classification or ACL group, so permission filters run inside the retrieval query (see 3.2).
• Facets: product, region, document type, language. Used for filtering and routing (see 3.6).

Without metadata, you can't filter, can't cite, can't enforce access and can't clean up. Adding it later means re-ingesting everything.

Choose the indexes for your queries

Dense vector indexes capture semantic similarity. Keyword (lexical) indexes such as BM25 capture exact terms: error codes, part numbers, names, legal citations. Many production systems keep both and combine the results. Domain-specific vocabulary is also a reason to evaluate embedding models on your own corpus rather than assuming a general-purpose one handles your terminology well.

Treat the index as a versioned build artifact

Ingestion is a pipeline with versions, not a one-time load:

• Consistency. Queries must be embedded with the same model (and preprocessing) used for the documents. Changing the embedding model or chunking scheme means re-indexing the corpus. Mixing versions in one index quietly degrades retrieval.
• Atomic refresh. Build a new index version alongside the live one, validate it against a retrieval test set, then switch over. Partial in-place updates leave a mix of old and new chunks.
• Deletion and supersession. When a document is removed or replaced, its chunks must go too. Stable document IDs make that possible. Leftover chunks from retired policies are a classic source of confident, outdated answers.
• Monitoring the pipeline. Track parse failures, empty chunks and chunk-count changes per source. A PDF parser upgrade that suddenly extracts tables as garbage shows up here before users notice.

Key concept

Chunk by structure, give every chunk the context and metadata to stand alone, and version the index so every refresh is consistent, validated and complete, including deletions.

When answers turn confidently wrong after a migration, refresh or re-index, the expected first move is to inspect what retrieval returned and how ingestion changed. Watch for mixed embedding versions, stale chunks, broken parsing or lost context. Prompt tweaks and model swaps are the distractors.

Practice scenario

Build exercise — Rebuild a small index the professional way

Advanced · 60 min

What you’ll learn

• Comparing fixed-size and structural chunking on your own queries
• Adding contextual prefixes and metadata that support filtering and provenance
• Running a versioned, validated index refresh

1. Take 20–50 structured documents (policies, runbooks or product docs) and index them twice: fixed-size chunks, and structural chunks with heading paths. Run 20 real questions against both and record whether the right section is in the top results.

   • Why: The right chunking strategy depends on your documents and queries. Measuring beats guessing.
   • You should see: Structural chunks winning on questions that depend on section context, and the gap you’d miss without measuring.

2. Add metadata (document ID, version, effective date, access group) and a short contextual prefix per chunk, then re-run the questions with a filter that excludes superseded versions.

   • Why: Context and metadata fix whole classes of errors (ambiguous chunks, outdated answers) that tuning can’t.
   • You should see: Fewer retrievals of stale or context-free chunks, and citations that resolve to a specific section and version.

3. Simulate a refresh: replace two documents and delete one, build a new index version, validate it against your question set, and switch over only if it passes.

   • Why: Most production retrieval regressions come from inconsistent refreshes, not from the model.
   • You should see: No chunks from the deleted or replaced documents in the new version, and a pass/fail check run before cutover.

Exam traps

Splitting every document into fixed-size chunks regardless of its structure

Fixed windows cut tables, procedures and clauses apart from the context that gives them meaning. Follow headings, records and content type.

Stripping headings and surrounding context so chunks can’t be understood on their own

Context-free chunks embed poorly and mislead the model. Add heading paths, contextual prefixes or parent-document retrieval.

Indexing without metadata for source, version, date, tenant and access control

Without metadata you can’t filter, cite, enforce permissions or clean up stale content.

Mixing embeddings from different models or chunking versions in one index

Similarity scores across versions aren’t comparable. Re-index the full corpus whenever the embedding model or scheme changes.

Updating documents in place with no way to remove stale or deleted chunks

Leftover chunks from superseded documents produce confident, outdated answers. Use stable IDs and versioned, atomic rebuilds.

Blaming the model for wrong answers before inspecting the retrieved chunks

When answers degrade after a data change, look at what retrieval returned first.

Sources

• Claude Developer Platform Docs
• Anthropic
• Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)