Plain text and Markdown (single vs element mode)

What you'll learn

• Explain what a plain text loader and a Markdown loader in single and element mode return for the same file.
• Compare the three by Document count, markup kept and metadata.
• Choose a mode by what the next step in the pipeline needs.

Lesson 1 of T5.2 defined the Document and the schema it should carry. Plain text and Markdown are the simplest sources: there is no layout to recover. The main decision is how many Documents one file becomes.

The Larkspur handbook ships as 13 Markdown files. Loaded with the plain text loader, the refunds page becomes one Document of 660 characters that begins with "# Refunds". Its five rules (monthly plans, the 30-day window, proration, the 5 to 10 business days, how to request) sit in one string, and nothing records that they belong under the Refunds heading. The splitter in T5.3 will cut that string without knowing where the sections were.

We loaded the refunds page three ways. The plain text loader reads the file as it is. The Unstructured Markdown loader parses the Markdown first, and its mode setting decides the output: single mode returns the whole file as one Document, and element mode returns one Document per element (a title, a paragraph, a list item, a table), each tagged with a category:

Table 1: The refunds page through three loaders (unstructured 0.27.37, langchain-community 0.4.2, run on data/larkspur/handbook/02_refunds.md).

In element mode, every paragraph's parent_id points at the Title element above it:

Figure 1: One file, two modes. Element mode keeps the link from each paragraph to its heading.

That link is what fills the section field of lesson 1 of T5.2's schema. Single mode loses it before splitting even starts, and we call that a one-blob file: a file reduced to one string, its structure gone. The plain text loader has a second effect, markup in the embedding: heading marks and table pipes stay in the text and get embedded with it unless a later step removes them.

Notebook 02b runs the same loaders on a different file, the LangChain README in data/sample/, and its counts differ from these. It also calls Unstructured's own partitioner directly, which returns Unstructured elements rather than LangChain Documents; converting them copies each element's text and metadata into a Document.

Across all 13 handbook files, element mode returns 78 elements:

Table 2: Element categories across the 13 Larkspur handbook files.

The two tables are where the mode matters most. Each Table element has a flat text form and an HTML form. The flat form of the server-size table starts "Size vCPUs Memory Included outbound traffic Small 2 4 GB 1 TB per month Medium 4 16 GB…", so once a splitter cuts it, "16 GB" no longer sits under a column name. Keep each table as one piece and give the model its HTML form; lesson 4 of T5.2 meets the same problem in a PDF.

The plain text loader is still the right choice in one case. A Markdown-aware splitter (T5.3) cuts on the heading marks and needs them in the text:

Table 3: Choosing a loader and mode for Markdown.

For the handbook, element mode does the most work at load time:

1. Load each file in element mode. The 13 files give 78 elements.
2. Fill section from parent_id. Each paragraph takes the text of its Title: the refunds rules carry "Refunds".
3. Keep the two Table elements whole, in their HTML form, so the server sizes keep their column names.
4. Hand the rest to the splitter (T5.3) with section attached, and put the section name into the chunk text as well, since metadata is not embedded.

Next: lesson 3 of T5.2 moves to sources that already come in records, CSV rows and JSON objects, where the choice is which record becomes one Document.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Single mode gives one Document per file. Element mode gives one per title, paragraph, list item or table, each with a category and a parent_id that links it to its heading.
• The plain text loader keeps Markdown marks in the text. That suits a heading-aware splitter, and puts markup into the embedding when nothing strips it.
• Keep each Table element as one piece in its HTML form, so values stay under their column names.