Skip to content
Forward Deployed
Forward Deployed
RAG Foundations: Ingestion & Indexing: sections and modules
T5.2

Document Ingestion & Metadata Design

Load text, Markdown, CSV, JSON, PDF, Word, web pages, transcripts and papers into `Document` objects.

Lessons
8 lessons
By
Sourav Banerjee
Prerequisites
None

Lessons

Start module
8 lessons
  1. 1
    Document Ingestion & Metadata Design: overview

  2. 2
    The Document abstraction and metadata design

  3. 3
    Plain text and Markdown (single vs element mode)

  4. 4
    Semi-structured sources: CSV rows and JSON / JSONL with jq schemas

  5. 5
    PDF and Word: parser comparison and element-level parsing

  6. 6
    Web, YouTube and arXiv sources; bulk directory loading

  7. 7
    Custom loaders with lazy loading

  8. 8
    Document Ingestion & Metadata Design: interview questions

Practise and Check

Deliverable
Multi-source ingestor: load a mixed folder (PDF, DOCX, Markdown, CSV, one web page) into Documents that share one metadata schema (source, type, section, page, date).
Assessment
Answer the five Interview questions without notes, then run the lab.
Interview probe
A customer uploads 4,000 product manuals as PDFs: two-column layouts, spec tables, scanned appendices. You have a week to make them searchable. How do you choose a parsing approach?
Common mistakes
  • Storing a title only in metadata and expecting queries to match it
  • Moving the plan name into metadata_columns, so it is never embedded
  • Trusting a silent_errors=True run without counting distinct source values

Your Progress

How confident are you with this module?

1 means "I could not explain it yet"; 5 means "I could teach it". Rate again any time.

 

Saved in this browser when you leave the box.

Related Modules