Caption-then-index with a vision LLM

What you'll learn

• Explain how captioning moves the vision work from query time to index time.
• Trace captions competing with text chunks on one scale.
• Diagnose a test question that never reaches a caption, and a caption that is found but cannot answer.
• Write a caption prompt that holds every value a question needs.

Lesson 1 of T6.5 found the right Larkspur chart and then ranked it behind every paragraph, because CLIP scores images and text on different scales. Captioning removes the second scale.

Captioning moves the vision work to index time. A vision model looks at each figure once and writes text about it, and from then on the figure is a text chunk: the same embedder, the same vector store and the same retriever as the paragraphs around it:

Figure 1: Caption-then-index. After indexing, a figure is a text chunk.

Notebook 02a builds it on the Attention paper:

Table 1: The captioning pipeline in 02a.

The cost lands once per image, at index time, and never per question. The index holds only text, so every tool from T5.4 to T6.3 still applies to captions: hybrid search, reranking, metadata filters.

02a's test question does not exercise the captions. PyMuPDF finds three raster images in the paper: Figure 1 on page 3 and the two images that make up Figure 2 on page 4. Figures 3 to 5 are not stored as images, so the extractor returns nothing for them.

The test question, What is the BLEU score of the Transformer (base model)?, is answered by Table 2 on page 8, which is text. The text extractor returns it with the rows flattened, one cell per line, and the training cost 3.3 × 10^18 comes out as "1018". The system prompt also says "Answer the question based upon your knowledge", which invites the model to answer from memory rather than from the retrieved chunk.

We call this the text-only test: a multimodal pipeline judged on a question that text retrieval already answers. To test captioning, ask about something only a figure shows.

The module's interactive chart retrieval lab embeds captions with all-MiniLM-L6-v2, the same model as the text chunks. The captions are representative, written the way a vision model following 02a's prompt would write them:

Table 2: Detailed captions among text chunks (the module's lab; captions illustrative, scores real).

The first misses at k = 1 because the paragraph repeats the question's words. Captions compete with text chunks on one scale, so the ranking is fair, and a k of 2 or 3 sends both.

02a's prompt asks for a summary "well optimised for retrieval". Retrieval needs less than the answer does. A short caption that names the chart and leaves out the values, such as "Bar chart of monthly availability in Q2 against a target line.", still ranks 2, 3 and 2 for the three lab questions. It is found, and it cannot answer: nothing in it says May or 99.91%. We call this the findable but empty caption.

Table 3: Two captions for Figure 1 (the module's lab; captions illustrative).

Write the caption prompt for the questions the figure must answer:

• Every printed value, with its label and unit: "April 99.98%, May 99.91%, June 99.97%".
• The title, axes and legend, and any reference line ("a dashed line marks the 99.95% target").
• The figure's place: document, page, figure number, so a citation can point at it.
• An explicit escape for anything unreadable, as T3.2 does with a not-extracted marker, so the captioner leaves a gap instead of guessing.

Pitfall: a misread caption, trusted forever. A caption is model output, written once and then trusted on every query. If the vision model reads 8 GB as 16 GB, retrieval finds the wrong number with full confidence.

02a already keeps each image's file name in its caption's metadata. That is enough to retrieve on the caption and then send the original image to a vision model at answer time, so the caption only has to be good enough to find the figure. Where a PDF has a real text layer, an element-level parser that returns each table as HTML (T5.2, lesson 4 of T5.2) keeps rows and columns that a caption would have to describe in prose.

1. Caption with a values-first prompt: every value with label and unit, axes, reference lines, figure number, and an escape.
2. Index captions beside text, with the image file in metadata.
3. Retrieve on captions, answer from images: send the original figure to the vision model with the question.
4. Test with chart-only questions, such as the May availability, never with a question the text answers.

Lesson 3 of T6.5 skips extraction altogether and retrieves the page as an image.

The code for this lesson is in these notebooks. Colab runs one in your browser: install the packages its first cell imports and set the API keys it reads. To run every notebook as written, clone the repository and use its uv environment.

• Captioning pays the vision cost once per image at index time. Captions then compete with text chunks on one scale, and hybrid search, reranking and filters all still apply.
• A caption that names the chart is found but cannot answer (findable but empty). Prompt for every printed value with its label and unit, the axes and reference lines, the figure's place, and an escape for unreadable parts.
• A caption is model output trusted on every query. Keep the image reference in metadata, send the original image at answer time, and test with questions only a figure can answer.