RAG & Agents Engineering

M20 lab, multimodal RAG

Four ways to index a chart.

Three Larkspur Cloud handbook pages, each with two short paragraphs and one figure that holds the numbers. Pick an index and a question, then move k: the items above the magenta rule are what the model receives.

Index
Golden question
Evidence rank
Figure reaches the model
Items sent
Questions answerable at this k

Ranking for this question

RankItemKindScoreGolden setWhat the model would read

Scores were recorded from real model runs (see the end of this page) and are ranked here. Each index scores on its own scale, so compare ranks across indexes, not scores.

Rank of the figure, every index

IndexQ1 MayQ2 refundQ3 memoryAnswerable

Green ranks fall inside the current k. Text only never indexes the figures, so it has no rank to give. Models: all-MiniLM-L6-v2 for text and captions, openai/clip-vit-base-patch32 for the joint space (the 01a notebook’s model), vidore/colSmol-256M for page images, a small model of the ColPali family. Captions are representative, written as a vision model following the 02a prompt would write them.