Skip to content
Forward Deployed
Forward Deployed
Lessons and contents
Document Ingestion & Metadata DesignLesson 1 of 8
8 lessons
Learn

Document Ingestion & Metadata Design: overview

Source
From the RAG & Agents Engineering course
By
Sourav Banerjee
Last updated
8 Nov 2026

Report an issue

Retrieval can only return text that ingestion extracted, labelled with metadata that ingestion attached. T5.2 rests on four ideas: what a loader does compared with a parser, why element-level parsing matters for PDFs, how to design a metadata schema before you need it, and why loaders should stream.

The running example is page 7 of the Larkspur Cloud customer handbook: two columns, a pricing table, a refund section, a header and a footer.

In short

  • Every loader returns Document(page_content, metadata). page_content is embedded and shown to the model; metadata is never embedded, and it is the only thing you can filter on exactly.
  • A loader gets bytes from a source; a parser turns those bytes into text and structure. Which parser you choose decides what the index can contain.
  • Element-level parsing gives one typed Document per title, paragraph, table or list item, so you can drop boilerplate, keep tables whole and tag every chunk with its section.
  • Design the metadata schema for the filters, citations and access rules you will need later. Adding a field after indexing means re-ingesting the corpus, and every loader names its fields differently (page counts from 0 in PyPDF, seq_num from 1 in JSONLoader), so map them onto the schema at load time.
  • Implement lazy_load so a large source streams through the pipeline instead of loading into memory at once.

Mental model

The ingestion pipeline A blob loader fetches the handbook PDF as bytes with a mimetype and source, and a parser turns the bytes into Documents with text and metadata. Page 7 becomes 1 Document with PyPDF or 10 with Unstructured in element mode. The Documents are validated for counts, lengths and empty text, then go to the splitter in T5.3. Loader: fetch and parse Check, then split Source Blob loader Blob Parser handbook.pdf file, bucket or URL application/pdf bytes + source pypdf, PyMuPDF or Unstructured Documents Validate Split T5.3 page 7 becomes 1 with PyPDF 10 with Unstructured count per source no empty text lengths in range chunks
  1. The handbook PDF sits in a folder, a bucket or behind a URL.
  2. A blob loader reads the raw bytes without interpreting them.
  3. The bytes travel with their mimetype and source, so any parser can take them.
  4. The parser turns bytes into text and structure. Layout, tables and headers are kept or lost here.
  5. Out come Documents: page 7 gives 1 with PyPDF and 10 with Unstructured in element mode.
  6. Count Documents per source and look for empty or tiny text before anything is embedded.
  7. The checked Documents go to the splitter in T5.3.

Loading is lossy. A PDF parser drops layout, an HTML loader drops visual hierarchy, a Word loader drops comments, and a CSV loader drops column types. Know what each loader drops; you cannot retrieve what was never extracted.

Interactive lab

One PDF page, three parsers: open the file in a browser. Switch between PyPDFLoader, PyMuPDFLoader and UnstructuredPDFLoader, hover each output Document to see which part of the page it came from, and compare how many schema fields each loader fills.

How it connects

  • Before: T5.1 puts Load first in the lifecycle and gives you the baseline to measure against.
  • Next: T5.3 splits these Documents into chunks; structure-aware splitting works best when the parser kept the structure. T5.5 uses the metadata for filtered search, and T5.6 copies section titles into chunk text.

Key Takeaways

  • page_content is what gets embedded; metadata is what you filter and cite by, and it is never embedded.
  • Choose the parser by what the documents contain. Element-level parsing keeps structure and costs speed.
  • Write the metadata schema down before the first ingest, including access control, and normalise values at load time.
  • Stream with lazy_load, batch the downstream work, and validate counts after every run.

Rate your confidence and keep notes on the module page.