The Document abstraction and metadata design

What you'll learn

• Explain the difference between a loader and a parser, and which one decides the extracted text.
• Design a metadata schema with provenance, filtering, access-control and quality fields.
• Diagnose a title that never matches a query and a filter that silently returns nothing.
• Choose which fields go in the embedded text, which in metadata, and which in both.

T5.1 put Load first in the lifecycle and measured a baseline that every later change must beat. This module opens that first step, because retrieval can only return text that ingestion extracted, labelled with metadata that ingestion attached.

Larkspur's assistant answers a refund question from page 7 of the customer handbook PDF. The answer is right, but the chunk behind it starts with the page header "Larkspur Cloud · Customer Handbook", carries no section name and no policy date, and its page number counts from 0. The assistant cannot say which version of the policy it quoted, and its citation points one page early.

Every loader returns the same shape: a Document, with page_content (the text that gets embedded and shown to the model) and metadata (a dictionary that is never embedded). People say "loader" for the whole step that produces Documents, but LangChain separates two jobs:

Table 1: The two jobs inside "loading".

Most built-in loaders bundle both: PyPDFLoader fetches the file and runs pypdf on it. GenericLoader keeps them apart, a blob loader plus a parser, which lets one parser read from a folder, a bucket or an API without new code:

Figure 1: A bundled loader and a composed one. Only the parser box changes the extracted text.

When someone says "the PDF loader is bad", they usually mean the parser extracts the wrong text. Swapping the parser fixes that. Changing how files are fetched does not, and we call that attempt a fetch-side fix: work on the layer that was never the problem.

Metadata answers questions about a chunk that similarity search cannot: where it came from, who may see it, how recent it is, what kind of content it is. A metadata schema is the fixed list of fields, with their meanings and types, that every Document must carry. It has to exist before ingestion, because every field you forget costs a full re-ingest to add.

Plan four kinds of field:

Table 2: Four kinds of metadata field, and what each makes possible.

The filtering and access fields run before similarity search. A question then only competes against chunks it is allowed to see:

Figure 2: A metadata pre-filter narrows the index before similarity search runs.

The T5.2 lab asks for one shared schema across PDF, DOCX, Markdown, CSV and a web page, with five fields: source, type, section, page, date. On the handbook page, the loaders fill different parts of it:

Table 3: How much of the lab's five-field schema each PDF loader fills on the handbook page.

A filled field can still be wrong for your purpose. The file's modification time is not the policy date. Take date from your CMS.

Three hard constraints shape the design, and each has a failure we name:

Table 4: Three constraints on metadata, the failure each one causes and its guard.

For the refund chunk in the opening example, the schema decisions come before the first load:

1. Choose the parser for the layout. Page 7 has two columns and a pricing table. An element-level parser (lesson 4 of T5.2) is worth its cost there, and it also separates the page header from the text.
2. Fill provenance on every Document. source is the file, page is stored 1-based so citations match the printed page.
3. Supply what the parser cannot. type comes from the route that loaded the file, date from the CMS record of the policy.
4. Keep the section in two places. As metadata for filtering, and as a short header in the chunk text so a question that names "Refunds" can match it.

Next: lesson 2 of T5.2 starts with the simplest sources, plain text and Markdown, where the single choice is whether one file becomes one Document or many.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• A loader fetches the source and a parser turns bytes into text. When the extracted text is wrong, swap the parser.
• Design the metadata schema before ingestion: provenance, filtering, access control and quality fields. Every forgotten field costs a full re-ingest.
• Metadata is not embedded, accepts only flat value types, and filters match exactly. Put fields that should affect similarity into the text, flatten values and normalise case.