Learn › AI Agents: Core Capabilities › RAG for Agents

RAG for Agents

Intermediate 🕐 12 min Lesson 4 of 7
What you'll learn
  • Explain what problem RAG solves and how it works in three stages
  • Describe what agentic RAG adds beyond standard single-pass retrieval
  • Identify the four key components of a RAG system

The Fundamental Problem

Language models are trained on data with a cutoff date, and they can't know about events after that cutoff. Even within their training data, they're unreliable on specific facts — product specs, internal company policies, current prices, recent research. For agents that need to give accurate, grounded answers on specific topics, this is a serious limitation.

Retrieval-Augmented Generation (RAG) solves it by giving the agent a way to look things up before answering.

How RAG Works

RAG has three stages:

  • Index — your documents are split into chunks, converted to vector embeddings (numerical representations of meaning), and stored in a vector database
  • Retrieve — when the agent needs information, it converts the query to an embedding and finds the chunks most semantically similar to it
  • Augment — the retrieved chunks are added to the prompt, giving the model the context it needs before it generates a response

The result: the model answers from your specific documents rather than from its general training data. It can cite exactly where the information came from.

Why Agents Need RAG

Agents especially benefit from RAG because they often need to answer questions mid-task — checking a policy before taking an action, looking up a customer record before making a recommendation, verifying a fact before including it in a report. RAG makes this dynamic knowledge access possible without hardcoding the information into the agent's context.

Agentic RAG: The 2025 Development

Standard RAG retrieves once at the start and generates from that result. If something important is missing from the initial retrieval, the system can't correct course.

Agentic RAG addresses this with iterative retrieval. The agent can retrieve, discover that it needs more context, and retrieve again with a refined query. It can also decide which retrieval strategy to use — broad keyword search, semantic vector search, or a SQL query against a structured database — based on what kind of answer it needs.

Common frameworks that support agentic RAG include LangGraph and LlamaIndex, both of which expose retrieval as a tool the agent can call dynamically.

Key Components of a RAG System

  • Vector database — stores embedded chunks (Pinecone, Weaviate, Chroma, pgvector are common choices)
  • Embedding model — converts text to vectors; must be consistent between indexing and querying
  • Chunking strategy — how you split documents affects retrieval quality significantly; paragraphs, sentences, and sliding windows produce different results
  • Retrieval strategy — vector similarity, BM25 keyword search, or hybrid (combining both)

What RAG Cannot Fix

RAG improves factual grounding but doesn't eliminate hallucination. If the retrieval misses the relevant content, the model may still fabricate an answer. Good RAG systems include a "not found" fallback — instructions for the model to say it doesn't have enough information rather than make something up. Evaluating retrieval quality is a separate discipline from evaluating generation quality, and both matter.

Key takeaways
  • RAG solves the knowledge cutoff problem by retrieving relevant documents before generating — grounding the model in your specific data
  • Standard RAG retrieves once; agentic RAG retrieves iteratively, refining queries based on what the agent discovers it needs
  • The four RAG components are: vector database, embedding model, chunking strategy, and retrieval strategy — each affects quality
  • RAG reduces hallucination but doesn't eliminate it — always include a not-found fallback so the agent acknowledges gaps rather than fabricating answers
Learn more every week Get 5 hand-picked prompts + new AI tools every Monday. Free.