Building RAG Systems with the Claude API

Claude’s context windows are large enough that teams periodically ask whether they can skip retrieval and paste everything in. For small, stable corpora, sometimes yes — especially with prompt caching making repeated large contexts affordable. But RAG survives the long-context era for reasons that have little to do with window size:

• Access control. Retrieval is where you enforce per-user document permissions; a shared mega-prompt cannot.
• Freshness. An index updates in seconds; re-shipping a giant prompt across every request does not scale operationally.
• Cost discipline. Sending only relevant passages keeps token spend proportional to the question, not the corpus.
• Attribution. Retrieved chunks give you natural citation units for auditability.

The practical answer is a spectrum: cache stable reference material in the prompt, retrieve the volatile and permission-sensitive remainder.

A production Claude RAG pipeline has five stages, each independently testable. Ingestion parses documents, normalises formats, and attaches metadata — source, owner, timestamps, ACLs. Indexing embeds chunks into a vector store, usually alongside a keyword index for hybrid search. Retrieval takes the user query (often rewritten by a fast model like Claude Haiku 4.5), pulls candidates, and reranks them. Assembly builds the Messages API request: system prompt, retrieved passages with identifiers, then the question. Generation calls Claude and post-processes citations.

Keep the boundary clean: everything before assembly is deterministic infrastructure you can unit-test; everything after is model behaviour you evaluate statistically. For document-heavy sources exposed through internal systems, an MCP server is a natural retrieval interface — the same connector then serves both your RAG pipeline and future agent use cases. Our Claude API & MCP training builds this architecture end to end.

Chunking is where most RAG quality is won or lost, because Claude can only reason over what retrieval surfaces. Guidelines that survive contact with real corpora:

• Chunk along semantic boundaries — sections, headings, clauses — not fixed character counts. A contract clause split mid-sentence retrieves badly and reads worse.
• Keep chunks self-contained: prepend the document title and section path so a chunk makes sense out of context.
• Use hybrid retrieval. Vector search catches paraphrases; keyword search catches exact identifiers, SKUs, and error codes that embeddings blur.
• Rerank. Pull a wide candidate set (say, 30–50), then rerank down to the handful you actually send.
• Store rich metadata and filter on it — date ranges, product lines, ACLs — before similarity ever runs.

Resist tuning chunk size by intuition. Change one variable, rerun your evaluation set, and let retrieval hit-rate metrics decide.

How you present retrieved passages shapes how faithfully Claude uses them. Wrap each chunk in a delimited block with a stable identifier, and instruct the model to answer only from the provided material and to cite the identifiers it used.

system = """Answer using ONLY the documents provided.
Cite sources as [doc_id]. If the documents do not
contain the answer, say so explicitly."""

context = "\n".join(
    f"<doc id='{c.id}' title='{c.title}'>{c.text}</doc>"
    for c in reranked_chunks
)

Two grounding disciplines pay off immediately. First, give Claude explicit permission to say “not found” — models hallucinate most when silence feels forbidden. Second, verify citations mechanically after generation: check that every cited identifier exists in the supplied context, and flag or regenerate responses that cite phantom sources. Structured outputs work well here, returning the answer and its citation list as a typed JSON payload your application can validate.

RAG requests have a characteristic shape — a long, stable prefix (system prompt, output instructions, sometimes shared reference documents) followed by a variable tail (retrieved chunks and the user question). Prompt caching exploits exactly this shape: mark the stable prefix as cacheable and repeated calls reuse it at substantially reduced cost and latency. Order matters, so put stable content first and volatile content last. For pipelines that reprocess a corpus offline — summarising, tagging, generating synthetic Q&A for evaluation — the batch API cuts costs further on non-interactive volume.

Match models to stages rather than using one everywhere: Claude Haiku 4.5 for query rewriting and relevance filtering, Claude Sonnet 5 as the default answer generator, and Claude Opus 4.8 or Claude Fable 5 where answers require synthesis across many conflicting sources. Consult Anthropic’s documentation for current capabilities rather than hard-coding assumptions.