Data Ingestion & Parsing · Week 05

LangChain Document Model

The Document is the universal container in LangChain's RAG pipeline — it carries raw text (page_content) and traceability data (metadata) from ingestion all the way to retrieval.

1 Load files
→
2 Split into chunks
→
3 Embed vectors
→
4 Store in DB
→
5 Retrieve & generate

01 · Foundation

What is a LangChain Document?

Every file you ingest — PDF, webpage, CSV row, or API response — becomes one or more Document objects. These are the atomic units passed to embedders, vector stores, and retrievers. LangChain v0.3+ splits packages: core types live in langchain_core, loaders in langchain_community, splitters in langchain_text_splitters.

Document

from langchain_core.documents import Document

A lightweight dataclass with three fields. You rarely subclass it — instead, loaders and splitters produce Documents automatically.

page_content · str metadata · dict id · str (optional)
Why it matters for RAG: Embeddings are computed from page_content, but when a user asks a question, the LLM needs metadata (source file, page number) to cite answers and filter results by date, department, or access level.
# Modern LangChain (v0.3+) — see notebook 1-dataingestion.ipynb
from langchain_core.documents import Document

# Manual creation (loaders do this for you)
doc = Document(
  page_content="RAG grounds LLM answers in your private data.",
  metadata={
    "source": "policies/handbook.pdf",
    "page": 12,
    "doc_type": "policy",
    "last_updated": "2025-11-03"
  },
  id="handbook-p12-chunk-0" # optional, for dedup in vector DB
)
page_content
Gets embedded & semantically searched
metadata
Enables filtering, citations & audit trails
1 → N
One file often becomes many chunk Documents
500–2k
Typical chunk size in tokens

02 · Anatomy

Document Components in Detail

Click any example or metadata field below to see how a real Document object is assembled. Good metadata design is often more important than chunk size for production RAG quality.

📝 page_content — the searchable text
The raw string that gets tokenized, embedded, and compared against user queries. Vector similarity search operates only on this field — not on metadata.

Real-world examples — click to preview:

Internal knowledge base Policy or wiki excerpt that employees query via a chatbot. ~180 tokens · single topic
Compliance / legal document Specific clause the system must cite with exact source reference. precise citation needed
FAQ pair (support ticket bot) Question-answer format works well for helpdesk RAG. Q&A structure
⚠️ Common mistakes Dumping HTML tags, headers/footers, or table-of-contents noise into page_content degrades retrieval. Clean text before embedding. Very long pages should be split — not stored as one giant chunk.
✓ Best practices One coherent idea per chunk · strip boilerplate · match chunk size to your embedding model's context window · include section headings in the text for context
🏷️ metadata — context & filtering
A plain Python dict attached to every chunk. Vector stores (Chroma, Pinecone, pgvector) index metadata for pre-filtering — e.g. "only search docs from 2025 in the legal department." Metadata is copied to child chunks when you split a Document.

Click fields to toggle in live preview:

source
Original file path or URL
→ Powers "according to handbook.pdf, page 12" citations
page
Page number within source
→ PyPDFLoader sets this automatically per page
doc_type
Category: policy, faq, contract…
→ Filter: only search doc_type == "policy"
department
Owning team or business unit
→ Multi-tenant access control
last_updated
Freshness timestamp
→ Exclude stale docs from retrieval
access_level
public · internal · confidential
→ Security filtering before LLM sees content
💡 Pro tip Add chunk_index and parent_id when splitting — helps reconstruct full sections and debug retrieval gaps.
🔑 id — optional unique identifier
An optional string for deduplication and upserts in vector databases. If you re-ingest the same document, a stable id prevents duplicate vectors. Convention: {source}-p{page}-chunk{index}

Live Document Builder INTERACTIVE

This is the exact object shape your vector store receives after loading and splitting.

page_content
Retrieval-Augmented Generation (RAG) retrieves relevant document chunks at query time and injects them into the LLM prompt, reducing hallucinations and enabling answers over proprietary data without fine-tuning.
metadata (+ optional id)

            
            
          

        

      

    


    
    

      
03 · Context

      
Where Documents Fit in the RAG Pipeline

      

        Documents are the handoff format between every stage. Understanding this flow clarifies
        why loaders, splitters, and metadata design all matter.
      


      

        

          

            
📂

            
Document Loaders

            
Read raw files (PDF, HTML, CSV) and output List[Document]

          

          
→

          

            
✂️

            
Text Splitters

            
Break large docs into chunk-sized Documents. Metadata is inherited + enriched.

          

          
→

          

            
🧮

            
Embeddings

            
embeddings.embed_documents([d.page_content])

          

          
→

          

            
🗄️

            
Vector Store

            
Stores vectors + metadata. Chroma, Pinecone, FAISS, pgvector.

          

          
→

          

            
💬

            
Retriever → LLM

            
Top-k chunks injected into prompt. Metadata shown as citations.

          

        

      

    


    
    

      
04 · Ingestion

      
Document Loaders

      

        Loaders handle the "messy" part — parsing binary formats, fetching URLs, walking directories.
        They all return the same Document type.
      


      

        
langchain_community.document_loaders

        

          

            Install (local or Colab): pip install langchain langchain-community langchain-core langchain-text-splitters pypdf pymupdf docx2txt jq tiktoken unstructured networkx msoffcrypto-tool.
            Each loader implements .load() → list of Documents
            or .lazy_load() for memory-efficient streaming on large corpora.
            Click a card for details. Cards below map to this week's notebooks.
          


          

            

              
TextLoader

              
Notebook 1 · single .txt files

              

from langchain_community.document_loaders import TextLoader

loader = TextLoader("data/text_files/python_intro.txt",

  encoding="utf-8")

docs = loader.load()  # 1 file = 1 Document
              

              

                Simplest loader. Pair with DirectoryLoader and glob="**/*.txt" to batch-load a folder.
                Always set encoding="utf-8" on Windows.
              

            


            

              
PyPDFLoader / PyMuPDFLoader

              
Notebook 2 · PDF reports & papers

              

from langchain_community.document_loaders import (

  PyPDFLoader, PyMuPDFLoader

)

docs = PyPDFLoader("data/pdf/attention.pdf").load()
              

              

                Automatically sets metadata["source"] and metadata["page"].
                PyMuPDFLoader is faster with richer metadata. Clean ligatures/whitespace before chunking
                (see SmartPDFProcessor in notebook 2).
                
    
                  
  • Sample file: data/pdf/attention.pdf
    • 
                  
  • Password-protected PDFs: pass password=
    • 
                

              

            


            

              
Docx2txtLoader

              
Notebook 3 · Word .docx proposals

              

from langchain_community.document_loaders import Docx2txtLoader

docs = Docx2txtLoader("data/word_files/proposal.docx").load()
              

              

                One plain-text Document per file. For element-level structure (Title, NarrativeText),
                use UnstructuredWordDocumentLoader with mode="elements".
              

            


            

              
CSVLoader

              
Notebook 4 · product catalogs, tabular rows

              

from langchain_community.document_loaders import CSVLoader

loader = CSVLoader("data/structured_files/products.csv",

  encoding="utf-8")

docs = loader.load()  # 1 row = 1 Document
              

              

                Each row becomes one Document with metadata["row"].
                For Excel, use pandas per-sheet docs or UnstructuredExcelLoader.
                Custom processing adds rich metadata (category, price) for filtering.
              

            


            

              
JSONLoader

              
Notebook 5 · nested JSON with jq

              

from langchain_community.document_loaders import JSONLoader

loader = JSONLoader(

  "data/json_files/company_data.json",

  jq_schema=".employees[]",

  text_content=False

)
              

              

                Requires pip install jq. The jq_schema extracts subtrees (e.g. each employee).
                For complex nesting, flatten to natural-language profiles with custom Python (see notebook 5).
              

            


            

              
SQLDatabase + custom SQL→Document

              
Notebook 6 · SQLite relational data

              

from langchain_community.utilities import SQLDatabase

db = SQLDatabase.from_uri("sqlite:///data/databases/company.db")

db.get_table_info()  # schema + sample rows
              

              

                SQLDatabase exposes schema for Text-to-SQL agents. For RAG ingestion, convert tables
                and JOIN results into readable Documents (employee–project relationships).
              

            

          


          
Also in the ecosystem (not covered this week):

          

            DirectoryLoader
            UnstructuredWordDocumentLoader
            UnstructuredExcelLoader
            WebBaseLoader
            NotionLoader
            S3FileLoader
            YoutubeLoader
            WikipediaLoader
            ArxivLoader
          

        

      

    


    
    

      
05 · Chunking

      
Text Splitters

      

        Embedding models have token limits. Splitters convert one large Document into many smaller ones
        while preserving metadata — each child chunk knows its source file and page.
      


      

        
langchain_text_splitters

        

          

            Install: pip install langchain-text-splitters tiktoken.
            Import from langchain_text_splitters (not deprecated langchain.text_splitter).
            RecursiveCharacterTextSplitter is the default choice for most RAG apps.
            Click a splitter to simulate chunking on your preview text above.
          


          

            

              
CharacterTextSplitter

              
Simple fixed-size splits

              

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(

  separator="\n",

  chunk_size=200,

  chunk_overlap=20

)
              

            

            

              
RecursiveCharacterTextSplitter

              
★ Recommended default

              

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(

  chunk_size=200,

  chunk_overlap=20,

  separators=["\n\n","\n"," ",""]

)
              

            

            

              
TokenTextSplitter

              
Token-accurate for OpenAI models

              

from langchain_text_splitters import TokenTextSplitter

splitter = TokenTextSplitter(

  chunk_size=50,

  chunk_overlap=10

)
              

            

            

              
SemanticChunker

              
Embedding-based boundaries

              

splitter = SemanticChunker(

  embeddings,

  breakpoint_threshold_type="percentile"

)
              

            

          


          

            
            # After loading documents from any loader

            from langchain_text_splitters import RecursiveCharacterTextSplitter


            splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)

            chunks = splitter.split_documents(docs)

            # Each chunk is a new Document — metadata copied from parent

            # Add chunk-specific metadata:

            for i, chunk in enumerate(chunks):

              chunk.metadata["chunk_index"] = i
          


          
        

      

    


    
    

      
06 · Lab

      
Notebook Map (Week 05)

      

        Each Jupyter notebook installs dependencies via a Colab-ready %pip install cell,
        then walks through one data format. No .env or API keys required — ingestion is fully local.
      

      

        

          
            

              #
              Notebook
              Focus
            

          
          
            
11-dataingestion.ipynbDocument model, TextLoader, DirectoryLoader, splitters

            
22-dataparsingpdf.ipynbPyPDFLoader, PyMuPDFLoader, SmartPDFProcessor

            
33-dataparsingdoc.ipynbDocx2txtLoader, UnstructuredWordDocumentLoader

            
44-csvexcelparsing.ipynbCSVLoader, pandas/Excel, UnstructuredExcelLoader

            
55-jsonparsing.ipynbJSONLoader + jq_schema, custom flattening

            
66-databaseparsing.ipynbSQLDatabase, sql_to_documents() with JOINs

          
        

      

      

        See also: RAG Architecture ·
        Vector Store vs Database ·
        RAG & Agents (Part 2) ·
        Run all locally: python scripts/run_notebooks.py