Page-image retrieval with ColPali

What you'll learn

• Explain late interaction: one vector per token and per patch, compared only at query time.
• Trace a Larkspur question through MaxSim, token by token, to a page score.
• Explain what page-level retrieval sends to the answer model, and what it skips.
• Weigh the storage, generation and citation costs of retrieving pages.

Lesson 1 of T6.5 ranked figures on a different scale from text; lesson 2 of T6.5 turned figures into text and trusted a vision model's reading forever. Both start by taking the page apart. ColPali keeps the page whole.

Retrieval so far has used two kinds of model. A single-vector retriever squeezes a chunk into one vector and a question into another, and compares them once. A cross-encoder (T6.3) reads question and chunk together, which is accurate and too slow to run over a whole index.

Late interaction sits between them. It encodes the document and the question separately, as with single vectors, so documents can be encoded ahead of time. It keeps one vector per token instead of one per text, and compares them only at the end. The comparison is called MaxSim:

Table 1: MaxSim, step by step.

ColBERT introduced this for text passages (Khattab and Zaharia, SIGIR 2020). ColPali applies it to page images (Faysse et al., ColPali: Efficient Document Retrieval with Vision Language Models, 2024): a vision language model reads the rendered page, and every image patch gets its own vector:

Figure 1: MaxSim on one page. The figures are from a real run.

Notebook 03a's model needs a GPU stack, so the Larkspur lab pages were scored with vidore/colSmol-256M, a small model of the same family, through the ColPali engine library. Each 850 by 1100 page image became 1,139 vectors of 128 dimensions, and each question 24 vectors: its own tokens plus 10 end-of-utterance tokens the processor appends, which count in the sum like any other. The library's multi-vector scoring computes Table 1's steps; a hand-written version matched it to within 2e-6.

Here are a few of the 24 per-token maxima for Which month missed the 99.95% availability target?:

Table 2: Per-token maxima and page scores (vidore/colSmol-256M, the module's lab pages).

month scores 0.65 and 0.61 on pages 1 and 2, so it barely separates them. availability and target do: each finds a strong match on page 1 and nothing better than 0.28 elsewhere. One vector per page would have pooled those signals before the question was known. We call that loss early pooling; late interaction keeps the signals apart until the question arrives.

In page-level retrieval the unit of retrieval is the page, and the answer model receives the page as an image. Nothing is extracted, split or captioned, so a chart, a table's grid and a scanned form arrive as they were printed.

03a does it in four calls with the byaldi library:

Table 3: The 03a pipeline in four calls.

Keeping the page images in the index is what lets search return the image with the hit, instead of a pointer to re-render.

The library's source (byaldi 0.0.7) explains why the course marks this lesson platform-specific:

• It defaults to a CUDA GPU. On a machine without CUDA, pass the Apple-silicon or CPU device instead. Weights load in bfloat16.
• PDFs go through a converter that needs the poppler tools installed on the machine.
• The Hugging Face token is read from a token argument or the HF_TOKEN environment variable. 03a sets a variable spelled HF_token, which the library never reads. We call this the misspelt token: it fails only when the model needs authentication.

In the interactive chart retrieval lab, choose Page images. ColSmol ranks the right page first for all three questions, by clear margins:

Table 4: Page scores for the three golden questions (vidore/colSmol-256M).

At k = 1 the model receives the page with the chart on it, so every answer is in what it reads. Some of page 1's lead comes from its paragraph, which says "availability" and "target"; that is fine, because the page brings its chart along.

Pages cost more than chunks in three places:

Table 5: Page retrieval against chunk retrieval.

Lower precision, pooled patch vectors, or a cheap first stage with MaxSim only on its shortlist keep a large collection affordable.

Interview tip: choose by where the meaning lives. Page images suit scans, slides, forms and dense tables, where layout carries meaning or there is no text layer. Text-heavy documents with a few charts are cheaper to serve with captions (lesson 2 of T6.5).

1. Use pages for status reports and scanned invoices, where charts and grids carry the answer.
2. Keep the handbook in text chunks, with captions for its few figures.
3. Shortlist first: a cheap single-vector search picks 20 pages, MaxSim reranks them.
4. Fix the token variable and pin the device before deploying 03a's code.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Late interaction keeps one vector per query token and per page patch, and scores a page as the sum of each token's best match (MaxSim). Pages are still encoded ahead of time, and no signal is pooled early.
• Page-level retrieval sends whole page images to a vision model, so charts, table grids and scans arrive as printed, with nothing extracted or captioned. On the Larkspur pages it ranks the right page first for all three questions.
• Pages cost more in storage (about 583 KB per page in float32), in generation and in citation granularity. Use them for scans, slides, forms and dense tables; caption text-heavy documents instead.