Web, YouTube and arXiv sources; bulk directory loading

What you'll learn

• Explain how a directory loader routes each file type in a mixed folder to its own loader.
• Compare the metadata keys and counting rules that eight loaders write.
• Design one mapping from every loader's output onto the module's five-field schema.
• Diagnose a skipped file, an indexed error page and a page that loads empty.

Lessons 2 to 4 of T5.2 loaded one kind of file at a time. A real corpus arrives as a mixed folder, plus sources that are not files at all.

Larkspur's nightly ingestion job loads a folder of handbook material and reports success. A week later, a customer's question about the onboarding guide gets no answer: the DOCX failed to parse on the first night, the job skipped it, and the only record is a line in a log nobody reads. We call this a silent skip.

The T5.2 lab asks for exactly this kind of folder: the handbook as PDF, the onboarding guide as DOCX, release notes in Markdown, plans.csv and the status page from the web. The directory loader loads a folder with one loader class per file type. The notebook builds one directory loader per extension, each with its own file pattern, loader class and loader settings, because the default loader class is the generic Unstructured file loader for everything.

Two defaults matter on a real folder:

• Stop on error, or skip. By default the whole run stops at the first file that fails to parse. A nightly job usually turns on silent errors to keep going. Then compare the number of files with the number of distinct source values after every run, because a skipped file only shows up in the log.
• Progress and threads are off by default. Turn them on for folders of thousands of files. The default concurrency is 4.

Loading the folder is the easy half. Each loader writes its own metadata, with its own key names and counting rules. This is what each one sets, in langchain-community 0.4.2, run on files in this repo's data/sample/ (the last three read from the loader source, since they need the network):

Table 1: What eight loaders put in metadata.

None of these fill the lab's five fields (source, type, section, page, date) the same way. Map each loader's output onto the schema in one place, right after loading and before splitting, so every later stage sees one shape:

Figure 1: Four loaders, four conventions, one mapping step before splitting.

The mapping chooses one meaning per field:

Table 2: The mapping rule for each of the five schema fields.

The mapping drops the fields a loader did not set instead of storing None, so a filter on page only ever meets numbers. section stays missing on routes that use a page-level parser, because only an element-level parser can fill it.

Remote sources fail more quietly than files, because the loader returns whatever the server sent.

Web pages. The recursive URL loader crawls from a root URL to a depth of 2 by default and stays under that root. Without an extractor, page_content is the raw HTML: tags, scripts and navigation menus get embedded. The URL notebook passes a BeautifulSoup extractor to keep only the text. Two failures come from its defaults:

Table 3: Two web-loader defaults and the failure each one causes.

YouTube transcripts. By default the YouTube loader returns one Document per video, with the video id as source. In chunked transcript mode it returns one Document per time window (120 seconds by default; the notebook uses 10 and 15), and each source becomes the video URL with a start time, so a citation opens the video at that moment. Adding the video's title and author needs another network call, which is where the notebook's SSL errors come from.

arXiv papers. The arXiv loader downloads each matching PDF, reads it with PyMuPDF, and joins every page into one Document. The page numbers are lost. A citation can name the paper but not the page. Published holds the date of the latest version, not the first submission. The loader fetches up to 100 papers by default, and it can return only the abstracts, which is often enough for a "find me papers about X" index and far cheaper to embed.

For the lab's mixed folder, the nightly job runs in this order:

1. Route by extension: PDF to the element-level parser for layout pages (lesson 4 of T5.2), DOCX to the Word loader, Markdown in element mode (lesson 2 of T5.2), CSV by row (lesson 3 of T5.2).
2. Load the status page with an extractor and the status check on, and flag it if its text is much shorter than its HTML.
3. Map every Document onto the five-field schema before splitting, dropping unset fields.
4. Count files against distinct source values. Five files in, five sources out, or the run fails loudly instead of skipping silently.

Next: lesson 6 of T5.2 writes a loader for a source no library covers, Larkspur's ticket API, and makes it stream.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Route each file type in a mixed folder to its own loader. With silent errors on, compare the file count with the distinct source values to catch skipped files.
• Every loader writes different metadata keys and counting rules. Map them onto one schema right after loading, and drop unset fields instead of storing None.
• Remote sources fail quietly. Pass an extractor for web pages, check the response status, and flag pages whose text is much shorter than their HTML.