Planning, Reflection & Reasoning Patterns: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur's cancellation assistant, a plan-and-execute agent, told a customer their data is kept for 12 months after cancelling. The handbook says 30 days. The trace shows the retention lookup ran. What went wrong, and what do you change?

What a strong answer covers:

• Reads the trace: the retention search returned the promotional-credits chunk ("expire 12 months"), and a fixed plan pops every step as done whatever it returned
• Adds a per-step evaluator that asks whether the result states the requested fact about the right subject, writes the verdict to state, and routes a failure to a replanner
• Passes only accepted results to the synthesizer and tells it to say a fact was not found rather than fill it in
• Reproduces the failure with a stub search that returns the wrong chunk on the first call, and adds the question to the golden set

2. The team is choosing between ReAct and plan-and-execute for two Larkspur assistants: one answers multi-part billing questions, the other troubleshoots a server that will not start. Which pattern goes where?

What a strong answer covers:

• Billing questions have lookups you can list up front, so a plan is cheaper (2 LLM calls) and inspectable before any search runs
• Troubleshooting is exploratory: the next check depends on the last result, which is what ReAct's per-step reasoning is for
• Names ReAct's failure from lesson 1 of T8.1: it can announce its next step as text and end, because nothing tracks what is left
• Suggests a middle ground for either: plan, execute, evaluate each step and replan, at N + 2 calls on a clean run

3. After adding replanning, the cost per question went up and the team wants to know whether it is paying for itself. What do you measure?

What a strong answer covers:

• Logs per run: planned steps, executed steps, rejected steps, replans, cap hits and LLM calls, from state rather than from prompts
• Tracks the replan rate and the share of rejected results that were really wrong, by sampling and labelling evaluator verdicts
• Compares answer completeness on the golden set with and without replanning, and the calls each extra complete answer cost
• Watches cap hits as their own alert, since a run that hits the cap returns a partial answer

4. A customer question needs six lookups, and the replanning agent runs with max_iterations = 6. On a day when one search returns the wrong chunk, one part of the answer goes missing. Why, and how should the cap be set?

What a strong answer covers:

• The cap counts executed steps, and a rejected step plus its retry is two executions: six lookups with one retry need seven
• The router checks the cap before it checks the verdict, so the run goes to the synthesizer with one lookup unanswered
• Sets the cap from the plan: planned lookups plus the retries you will pay for, or a per-step retry limit
• Keeps the honest fallback: the synthesizer reports the missing fact, and the response says the answer is partial

5. You add chain-of-verification to the RAG answer step, as the T8.1 lab brief asks. Unsupported claims fall on the golden set, but a reviewer finds one correct claim was removed. How do you evaluate and guard this?

What a strong answer covers:

• Scores before and after in code against labelled claims, counting removed correct claims as well as removed unsupported ones
• Expects precision to rise and recall not to: CoVe checks the claims that were made and adds none
• Answers verification questions from retrieved handbook chunks rather than the model's memory, and checks the reviser's output instead of trusting its list of corrections
• Reports the cost of 1 + 1 + N + 1 calls per answer and batches the N independent calls to hold latency