Learn

Custom loaders with lazy loading

Source: From the RAG & Agents Engineering course
By: Sourav Banerjee
Last updated: 9 Nov 2026

Lazy loading

What you'll learn

Explain the difference between loading a corpus as a list and streaming it from a generator.

Design a custom loader for a source no library covers, with the module's schema in every Document.

Diagnose an ingestion job that grows in memory and indexes nothing.

Apply a checkpoint so a crashed run resumes instead of starting over.

Lesson 5 of T5.2 loaded files and web sources with loaders that already exist. Larkspur's support tickets live behind its own paginated API. No library loader covers them, and we write one.

The first version builds a list of every ticket and returns it. On the million-ticket backfill, the job's memory grows for hours, the machine kills it, and the index holds nothing, because no Document could be split or embedded until the whole list was complete. We call this a load-all stall.

A loader has two methods. load() returns a list: the whole corpus sits in memory before anything else runs. lazy_load() is a generator: it yields one Document at a time, and the pipeline can split, embed and upsert in batches while the loader keeps reading.

The custom loader in this lesson, a ticket loader, implements only lazy_load(). It walks the API page by page from a since date and yields one Document per ticket, already in the module's schema:

Document field	Filled from
`page_content`	The ticket body
`source`	`ticket:` followed by the ticket id
`type`	The fixed value `ticket`
`date`	The ticket's `updated_at`
`tenant_id`	The customer's organisation, for access control (lesson 1 of T5.2)

Table 1: What the ticket loader puts in each Document.

BaseLoader derives load() from lazy_load(). Implementing the generator gives you both methods. The indexer pulls Documents one at a time and sends batches downstream:

The indexer calls next() on the loader.
lazy_load() reads one ticket from the API and yields it as a Document.
The indexer splits it and adds the chunks to the current batch.
When the batch is full, it is embedded and written to the index in one call.
The index returns the stored ids, and the indexer records the last one as its checkpoint.
The loop runs until the source is exhausted, so memory never holds more than one batch.

Figure 1: Lazy loading. Embedding starts with the first batch while the loader is still reading.

The memory difference is a matter of what is held at once. With an assumed average of 2 KB of text per ticket:

	`load()`	`lazy_load()`, batches of 100 chunks
Held in memory at the peak	All 1,000,000 tickets, about 2 GB of text before any Python overhead	One batch of 100 chunks, about 200 KB
First vectors reach the index	After the last ticket is read	After the first batch
Lost on a crash at ticket 900,000	Everything	The current batch, if a checkpoint was recorded

Table 2: Load all versus stream, for the million-ticket backfill (illustrative: 2 KB per ticket and per chunk are assumptions).

A failure part-way through only loses the current batch as long as you record a checkpoint: the last id or updated_at written to the index. The loader's since argument goes straight to the API, so a restart begins from the checkpoint instead of the first ticket.

In practice: the Larkspur ticket backfill

For the backfill and the nightly updates after it, the ticket loader runs in four steps:

Implement lazy_load() only, yielding one Document per ticket in the module's schema, with tenant_id filled.
Batch downstream: split each Document, collect 100 chunks, embed and upsert them.
Record a checkpoint after each batch is stored: the last updated_at written.
Pass the checkpoint as since on the next run. The nightly job then reads only tickets changed since the last one.

Next: T5.3 takes the Documents this module produced and decides where to cut them into chunks.

Run the notebooks

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

Notebook	GitHub	Colab
Custom loader	Open on GitHub	Open in Colab

Key Takeaways

load() holds the whole corpus in memory as a list, while lazy_load() yields one Document at a time so splitting and upserting run in batches alongside reading.
Implement lazy_load() in a custom BaseLoader. load() comes free from it.
Record a checkpoint, such as the last id or updated_at written, so a crash loses only the current batch and the next run can resume from there.

Check yourself

1. With lazy loading, why do the first vectors reach the index before the loader has finished reading the source?

lazy_load() is a generator that yields one Document at a time. The indexer splits each one, adds it to a batch, and sends each batch of 100 chunks to be embedded and upserted while the loader keeps reading. With load(), nothing else runs until the whole corpus sits in memory as a list.

2. The ticket loader ingests a million tickets and crashes part-way through. How much work is lost, and what lets you resume?

Only the current batch, as long as you record a checkpoint: the last id or updated_at written. The loader takes a since argument and passes it to the paginated API, so a restart can begin from the checkpoint instead of the first ticket.

3. A teammate writes a custom loader that overrides load and builds a list of every record. Which method should they implement instead?

lazy_load(), as a generator. BaseLoader derives load() from lazy_load(), so the generator gives them both methods. A list keeps the whole corpus in memory, which a million-record source cannot afford, and nothing reaches the index until it is complete: a load-all stall.