Agentic RAG

Decide. Retrieve. Grade. Rewrite. Answer.

Classic RAG always retrieves. Agentic RAG adds control flow — so the model retrieves only when needed, checks relevance, and rewrites weak queries before answering.

Start the walkthrough See the architecture

Classic vs Agentic

Hope vs validate

Most RAG apps do the same thing every time: retrieve, then generate. Agentic RAG asks better questions first.

Classic RAG

A straight line. No check on whether retrieval helped.

question → vector search → answer

Agentic RAG

Control flow with loops, relevance grading, and a rewrite budget.

Should I retrieve at all?
Are these docs relevant?
If not — rewrite and try again (not forever).

The map

Architecture walkthrough

Every query enters decide. From there the graph either answers directly or runs retrieve → grade → generate / rewrite. Click a node to inspect its role.

flowchart TD
    START([START]) --> Decision["decide\nanswer or retrieve"]
    Decision -->|tool_calls| Retriever["retrieve\nChroma tool"]
    Decision -->|no tool_calls| END1([END])
    Retriever --> Grade{"grade_documents\nrelevant?"}
    Grade -->|yes| Generate["generate\ngrounded answer"]
    Grade -->|no + rewrites left| Rewrite["rewrite\nrephrase question"]
    Grade -->|no + max rewrites| Generate
    Generate --> END2([END])
    Rewrite -->|loop| Decision
    classDef decision fill:#E6F4F1,stroke:#0F766E,color:#134E4A;
    classDef terminal fill:#F1F5F9,stroke:#64748B,color:#334155;
    classDef node fill:#FFFFFF,stroke:#334155,color:#0F172A;
    class Decision,Retriever,Generate,Rewrite node;
    class Grade decision;
    class START,END1,END2 terminal;

From	Condition	To
START	always	decide
decide	tools_condition = tools	retrieve
decide	no tool calls	END
retrieve	grade = yes	generate
retrieve	grade = no & rewrites left	rewrite
retrieve	grade = no & max rewrites	generate
generate	always	END
rewrite	always	decide

Notebook path

Learn step by step

Follow the same sequence as agentic-rag-langgraph.ipynb. Finish one lesson, then advance.

0 · What you will build

An agentic RAG pipeline where the LLM decides whether to retrieve, grades document relevance, generates an answer, or rewrites the query and loops — with a max-rewrite guard (MAX_REWRITES = 2).

Decide: answer small talk / general facts, or call the retriever
Retrieve: similarity search over indexed blogs
Grade: binary relevance check on context
Generate: grounded answer from context
Rewrite: rephrase the question to improve retrieval

Requirements: Python 3.10+, OPENAI_API_KEY in .env (or Colab userdata).

1 · Install & environment

Install the LangChain / LangGraph stack, then load your API key.

# Install once
%pip install -q langchain_core langchain-openai \
  langchain-community langgraph chromadb \
  langchain-text-splitters beautifulsoup4 \
  python-dotenv lxml tiktoken

# Load OPENAI_API_KEY from .env or Colab
from dotenv import load_dotenv
load_dotenv()
os.environ.setdefault("USER_AGENT", "agentic-rag-langgraph-notebook")

Chat model: gpt-4o-mini (cheap & fast for demos)
Embeddings: text-embedding-3-large (better retrieval)
Smoke-test both before building the corpus

2 · Knowledge corpus

Index three public posts on different topics so “should I retrieve?” is a real decision — not a toy with one document.

Topic	Source
Reward hacking	Lilian Weng
Hallucination	Lilian Weng
Transformers	Jay Alammar — Illustrated Transformer

Load with WebBaseLoader, prefer <article> / <main>
Chunk with tiktoken splitter (~400 tokens, 80 overlap)
Store in Chroma collection agentic-rag-corpus, retriever k=4
Sanity-check: “What are common types of reward hacking?”

text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=400,
    chunk_overlap=80,
)
vectorstore = Chroma.from_documents(
    documents=doc_splits,
    collection_name="agentic-rag-corpus",
    embedding=embeddings,
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

3 · Retriever tool & graph state

The tool description is not decoration — it teaches the model when to call retrieval.

retriever_tool = create_retriever_tool(
    retriever,
    name="retrieve_knowledge_corpus",
    description=(
        "Search the local knowledge corpus about: reward hacking, "
        "LLM hallucination, and Transformer architecture. "
        "Use ONLY for those topics. Do NOT use for greetings or trivia."
    ),
)

MAX_REWRITES = 2

class AgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], add_messages]
    rewrite_count: int

Strict tool descriptions keep chit-chat out of the vector store
rewrite_count + max rewrites prevent infinite loops
Without the guard, a bad corpus can spin forever

4 · Graph nodes

Each node prints a banner so you can watch the path in notebook output.

Node	One-line job
`decide`	Bind tools → RETRIEVE or DIRECT ANSWER
`retrieve`	`ToolNode` runs Chroma search
`grade_documents`	Structured yes/no with Pydantic
`generate`	Grounded answer from latest context
`rewrite`	Improved `HumanMessage` + bump counter

def grade_documents(state: AgentState) -> Literal["generate", "rewrite"]:
    result = grader.invoke({...})
    if score == "yes":
        return "generate"
    if rewrite_count >= MAX_REWRITES:
        return "generate"  # best-effort
    return "rewrite"

5 · Wire the LangGraph workflow

Connect nodes with edges that match the architecture diagram.

workflow = StateGraph(AgentState)
workflow.add_node("decide", decide)
workflow.add_node("retrieve", retriever_node)
workflow.add_node("generate", generate)
workflow.add_node("rewrite", rewrite)

workflow.add_edge(START, "decide")
workflow.add_conditional_edges(
    "decide", tools_condition, {"tools": "retrieve", END: END},
)
workflow.add_conditional_edges("retrieve", grade_documents)
workflow.add_edge("generate", END)
workflow.add_edge("rewrite", "decide")

app = workflow.compile()

START → decide
decide → retrieve or END
retrieve → grade → generate or rewrite
rewrite → decide (loop)

6 · Run with `ask()`

Wrap app.invoke and print a clean summary: tools called, rewrite count, final answer.

def ask(question: str):
    result = app.invoke({
        "messages": [HumanMessage(content=question)],
        "rewrite_count": 0,
    })
    # print tools / rewrite_count / FINAL ANSWER
    return result

ask("What types of reward hacking does Lilian Weng describe?")

Next: explore the four live paths below — corpus, greeting, trivia, multi-topic.

Go to demos →

Live paths

Four demo paths

Same agent, different questions — watch which route the graph takes.

Path A — Happy path

“What types of reward hacking does Lilian Weng describe?”

decide retrieve grade YES generate END

Expect banners showing RETRIEVE → grade: YES → grounded answer from the reward-hacking post.

Path B — Greeting

“Hi! How are you today?”

decide DIRECT ANSWER → END

No tool call. If your model still retrieves, tighten the tool description — that is a teaching moment, not a failure.

Path C — General knowledge

“What is the capital of India?”

decide no retriever → END

The agent should not waste vector search on trivia outside the corpus.

Path D — Multi-topic

Self-attention plus hallucination causes — one agent across a multi-topic corpus.

decide retrieve grade generate or rewrite → decide

May pull transformers and/or hallucination chunks, then grade and generate (or rewrite if weak).

Design tips

Five takeaways

Production-minded habits from the notebook and teaching script.

Diverse sources

Multi-topic corpora make “should I retrieve?” meaningful.

Readable printers

Summarize docs and context — don’t dump raw Document objects.

Rewrite budget

rewrite_count + max rewrites stop infinite loops.

Strict tool text

Tell the model when not to retrieve (hi / trivia).

Log decisions

For production: larger k, metadata filters, log every grade/rewrite.

Homework — Corrective RAG (CRAG)

If grading still fails after max rewrites, call a web search tool, grade again, then generate. That is the natural upgrade from this notebook to CRAG.

Ship it

Deploy this page

This file is a single static HTML page — no build step. Pair it with the notebook in your repo.

GitHub Pages

Push this file to your repo, then enable Pages on the branch (or /docs folder).

agentic-rag-langgraph-guide.html

Netlify

Drag-and-drop the file (or the repo root). Set publish directory to / and open the HTML URL.

Publish directory: .

Companion files

Point learners to the runnable notebook and architecture notes.

agentic-rag-langgraph.ipynb
agentic-rag-langgraph-architecture.md

Stack

LangGraph · LangChain · Chroma · OpenAI embeddings · gpt-4o-mini

Decide. Retrieve. Grade. Rewrite. Answer.

Hope vs validate

Classic RAG

Agentic RAG

Architecture walkthrough

Learn step by step

0 · What you will build

1 · Install & environment

2 · Knowledge corpus

3 · Retriever tool & graph state

4 · Graph nodes

5 · Wire the LangGraph workflow

6 · Run with ask()

Four demo paths

Path A — Happy path

Path B — Greeting

Path C — General knowledge

Path D — Multi-topic

Five takeaways

Homework — Corrective RAG (CRAG)

Deploy this page

GitHub Pages

Netlify

Companion files

Stack

6 · Run with `ask()`