Prompt Engineering Patterns: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur's relevance grader labels the refund-timing chunk "no" for "How long does a refund take?", so the assistant answers "I don't know". The grader is a few-shot prompt. How do you debug it?

What a strong answer covers:

• Reproduces the call with the exact rendered prompt, model and temperature, and reads what the grader saw
• Checks the examples first: do they cover a chunk that answers the question with different words ("Refunds" against "refund")? Are the yes and no examples balanced?
• Adds the case to a labelled grading set and measures agreement before and after the fix, so the change is not tuned to one question
• Considers a stricter output contract (one word, or a schema) so a hedged answer is not parsed as "no"

2. The team wants self-consistency on the grader: five samples and a vote on every chunk. What do you say?

What a strong answer covers:

• A yes/no grade has one wrong answer, so a vote amplifies the majority: it helps only when one sample is already right more than half the time
• Measures one-sample accuracy on labelled pairs before deciding; below 0.5 the vote makes the grader worse (0.40 becomes 0.317 at five samples in the lab)
• Counts the cost: five calls per chunk times k chunks per question, plus an aggregator call if an LLM merges the paths as in notebook 02a
• Prefers an odd number of samples, since an even split on two labels is a coin flip and adds a call for nothing

3. You are moving the support assistant from gpt-4o-mini to a cheaper open model. How do you decide whether the prompts still work?

What a strong answer covers:

• Freezes the prompt set and the labelled cases, and runs both models at temperature 0 through the same wrapper
• Scores each task with a metric that fits it: exact match for labels and JSON validity for extraction, an LLM or human judge for free text
• Compares per task, not on average: a model can match on classification and still fail the JSON format or the five-line limit
• Pins the model id in one config value, because provider ids are retired (Groq shut down the llama-3.2-90b-vision-preview id used in two T3.1 notebooks on 14 April 2025)

4. The ReAct support agent sometimes loops: it searches the handbook, gets the same chunks, and searches again. How do you find and stop this?

What a strong answer covers:

• Logs every Thought, tool call and Observation per run, and counts tool calls per question
• Sets a hard limit on iterations (max_iterations on AgentExecutor, a recursion limit in LangGraph) and a fallback answer when it is hit
• Checks whether the Observation can answer the question at all: repeated identical searches usually mean the evidence is not in the handbook, which should end in an "I don't know"
• Tracks the share of runs that hit the limit as a metric, alongside answer quality on the golden set

5. A product manager asks for one prompt that summarises a ticket, classifies it, detects the language and drafts a reply in that language. When is one prompt right, and when do you split it?

What a strong answer covers:

• One prompt with an output contract (named JSON fields, as in notebook 04a's IT support task) is cheaper and keeps the parts consistent with each other
• Splits it when a part needs its own model, its own evaluation or a human check, or when one field's failure should not discard the others
• Validates the output against a schema and retries or repairs on a parse failure, instead of trusting free-form JSON
• Labels a sample of tickets per field, so a regression in classification is visible even when the reply text still looks fine