Framework Fluency: LlamaIndex: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. You port Larkspur's hybrid retriever to LlamaIndex: a FAISS IndexFlatL2 vector store and BM25, fused with QueryFusionRetriever(mode="relative_score"). The fused top result is now the least relevant handbook passage. What happened?

What a strong answer covers:

• Prints each retriever's raw scores before fusion. With IndexFlatL2, LlamaIndex's FAISS store returns squared L2 distances as score, so lower is better
• relative_score min-max scales each list with higher meaning better, so the nearest chunk gets 0 and the farthest gets the full weight
• Fixes the direction at the source: IndexFlatIP on normalised vectors (cosine similarity), or a retriever wrapper that flips distances
• Adds a golden-set check that compares fused results with each retriever alone, so a scale or direction bug shows up as a regression

2. The refund assistant answers "Can I get a refund after 45 days?" with "No, only within 30 days." The handbook prorates refunds after day 30. Do you raise window_size, raise similarity_top_k, or add a reranker?

What a strong answer covers:

• Checks the retrieved sentence first: it is the 30-day rule, and the prorated rule is the next sentence, ranked third
• A window of 1 sends both rules in 284 characters; similarity_top_k=2 adds "Monthly plans are not refundable" instead
• A cross-encoder over the top 10 also brings the prorated rule into the top 2, at the cost of a model call per candidate
• Weighs storage too: at window_size=3 the handbook's window metadata is 6.75 times its text, held on every node

3. The T6.6 lab asks you to port the T7 hybrid-plus-rerank pipeline to LlamaIndex and confirm the metrics match. They do not. How do you find out why?

What a strong answer covers:

• Compares retrieved node ids per golden question before comparing answer scores, so retrieval and generation differences separate
• Lists the default changes: chunk_size in tokens against LangChain's characters, metadata counted in the chunk budget and embedded with the text, and similarity_top_k=2
• Checks BM25 tokenisation: LlamaIndex's BM25Retriever stems and drops English stop words, LangChain's splits on whitespace
• Checks fusion: LlamaIndex's reciprocal_rerank ignores retriever_weights, while EnsembleRetriever applies weights to RRF

4. LLMRerank(top_n=5) sometimes returns one node, and on a few questions none. The answers that follow are thin or refuse. What is going on, and what do you change?

What a strong answer covers:

• The default prompt tells the LLM to leave out documents that are not relevant, so the output can be shorter than top_n
• Nodes the LLM omits are dropped, and the parser skips lines it cannot read without raising, so a reply in another format drops its whole batch of up to 10
• Logs the raw reranker reply and the count in and out per query, and alerts on empty results
• Falls back to the retrieval order when the reranker returns too few nodes, or uses a cross-encoder, which always scores every candidate

5. A refactor removed num_queries=1 from the fusion retriever. Retrieval quality looks the same on the golden set, but latency and the LLM bill went up. Why?

What a strong answer covers:

• QueryFusionRetriever defaults to num_queries=4: one LLM call writes 3 extra queries, and each query runs on every retriever
• With two retrievers that is 8 retrievals and one generation call before the answer step starts
• Treats default parameters as part of the pipeline's config: pin them explicitly and diff them in code review
• Measures per-stage latency and token use in traces, not only end-to-end quality, so a cost regression shows even when accuracy holds