Multimodal RAG: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur moves its quarterly service report into the handbook as a PDF. Customers ask which month missed the availability target, and the assistant says the handbook does not say. The text pipeline has not changed. What happened, and how do you confirm it?

What a strong answer covers:

• Prints what the loader produced for that page: the paragraph that says "Figure 1 shows…" is there and the 99.91% is not, because the number exists only as pixels
• Checks the retrieved chunks: the figure note ranks first, so retrieval did its job on what it was given
• Counts how many golden questions need a figure, table or scan to answer, which sizes the fix
• Adds a figure route (captions or page images) and a figure-only question to the golden set so the regression stays caught

2. A teammate wants one CLIP index for the whole handbook, text and images together, because "one index is simpler". What do you expect to happen, and what would you build instead?

What a strong answer covers:

• CLIP's text-to-text scores sit far above its text-to-image scores (0.692 to 0.912 against 0.200 to 0.382 on the lab pages), so images lose every mixed ranking
• Retrieving with separate quotas, top text chunks plus the top image, fixes the crowding: on the lab pages the right figure is the top image for all three questions
• CLIP's text encoder truncates at 77 tokens and was trained on captions, not paragraphs, so it is a weak text retriever as well
• Captions embedded with the text model, or page images, avoid comparing scores across modalities at all

3. You add captioning. How do you measure whether figure questions improved, and how do you watch it in production?

What a strong answer covers:

• A golden set split by evidence type (paragraph, figure, table, scan), with the rank of the evidence and answer correctness per split
• Compares against the text-only baseline on the same questions, as in the T5.1 lab
• Logs which retrieved items were captions and whether the answer quoted a number from one
• Re-captions and re-scores when the vision model or the caption prompt changes, since every caption is model output

4. A customer is told the Medium server has 16 GB. The table says 8 GB. Retrieval returned the right caption. What went wrong, and how do you guard against it?

What a strong answer covers:

• The caption itself is the suspect: a vision model can misread a row or a column, and nothing checks it after indexing
• Keeps the image id in the caption's metadata and sends the original image to a vision model at answer time, using the caption only to find it
• Spot-checks captions that contain numbers against the source, or extracts tables with a table parser where the PDF has a text layer
• Adds the question to the golden set with its exact expected value

5. Compliance wants scanned contracts, with no text layer, searchable alongside the handbook. ColPali looks right. What does it cost, and how do you keep it practical?

What a strong answer covers:

• Page-level late interaction needs no OCR and keeps layout, which suits scans
• Storage grows with patches: ColSmol-256M stored 1,139 vectors of 128 dimensions per lab page, 145,792 floats or about 583 KB in float32, against one vector per chunk for a text index
• Reduces it with lower precision or pooled patch vectors, or uses a cheap first stage and MaxSim only to rerank its shortlist
• The answer model must read a whole page image per result, so k stays small and the generation cost per question rises