Short-Term Memory & Conversation State: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. After a refactor, Larkspur's assistant answers first questions well, but follow-ups like "How long do I have?" now cite billing chunks. Nothing in the retriever changed. How do you find the cause?

What a strong answer covers:

• Logs the query the retriever actually received on each turn, next to the user's message, before reading any prompt
• Checks that history reaches the rewrite step: the session id in the config, the history_messages_key matching the placeholder, and an empty chat_history (which makes create_history_aware_retriever skip the rewrite and search the raw text)
• Replays a follow-up golden set and compares hit rate on turns 2 and 3 with the last good release
• Adds those follow-ups to the regression suite so the next refactor is caught by a number, not a customer

2. The rewrite step adds an LLM call, and its latency, to every follow-up. A teammate wants to drop it and append the chat history to the query instead. What do you tell them?

What a strong answer covers:

• Appending is free and does recover elliptical turns: in the lab, turn 2 hits 5 of 6 either way
• It fails on topic changes, because earlier questions outweigh the new one: 1 of 6 turn-3 hits at k = 1, against 5 of 6 with a rewrite
• A larger k hides the problem by spending context slots on the old topic, which costs tokens and adds distractors
• Middle ground: skip the rewrite on turn 1 and when a cheap check says the question is already standalone, use a small fast model for the rewrite, and measure latency against hit rate

3. How do you know, week by week, whether conversational retrieval is getting better or worse in production?

What a strong answer covers:

• Logs per turn: turn number, raw message, rewritten query, retrieved chunk ids and scores, session id
• Tracks retrieval quality split by turn number, since first turns hide follow-up regressions in an overall average
• Samples rewrites for an LLM judge or human check: is the rewrite faithful to the history, and does it drop or invent constraints
• Re-runs a frozen set of multi-turn conversations on every prompt, model or index change, as the T10.1 lab brief describes

4. A product manager wants the conversation id in the URL so customers can share a chat with a colleague. What is the risk, and how would you build it?

What a strong answer covers:

• The thread_id or session_id is the only thing separating one customer's history from another's; whoever can choose it can read that history
• Derives the id on the server from the authenticated user (or checks ownership before loading), never from request text
• Builds sharing as an explicit, read-only copy or an access grant, with an audit entry
• Remembers that a persistent checkpointer keeps the thread until it is deleted, so retention and deletion rules apply to it like any customer data

5. Support tickets now run to 60 turns, and prompt cost per turn keeps climbing. Which memory strategy do you pick, and what do you check before shipping it?

What a strong answer covers:

• Full history sends every earlier message on every turn, so cost per turn grows without limit
• A window or trim on read flattens the prompt but loses early facts such as the plan or the region; a running summary keeps them for one extra call per fold
• Prefers trim on read or a summary over deleting messages, so the full thread stays available for audit and replay
• Tests recall of an early fact on a late turn, and watches the summary channel: with no reducer, writing summary again on a later turn overwrites the accumulated summary