Chunking Strategies: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Your support assistant tells a customer "Annual plans are refundable within 30 days" and the customer gets a refund denied: they had used 300 GB of traffic. The retrieved chunk is correct as far as it goes. What happened, and what do you change?

What a strong answer covers:

• Recognises a boundary cut: the condition ("less than 100 GB") sits in the next chunk
• Confirms it by printing both chunks and their scores for this question, rather than guessing
• Fixes it at index time: structure-aware or larger chunks for policy text, enough overlap to carry the condition, or propositions that keep each condition with its rule
• Adds the case to the golden set, with a check that the answer mentions the condition

2. A teammate set chunk_size=2000 "so nothing gets cut". Hit rate went up, but answers got worse and the token bill doubled. Explain why, and propose a fix that keeps the hit rate.

What a strong answer covers:

• Large chunks raise hit rate because almost everything is "in" some chunk, but each one mixes topics, diluting the embedding and adding distractors
• Measures faithfulness and correctness, not only hit rate, plus tokens per answer
• Proposes small chunks for matching with larger parents (or neighbour expansion) for context, or a reranker to cut what reaches the model
• Validates with a sweep on the golden set, not a single comparison

3. You are indexing a 50,000-page internal wiki in Markdown and a 2,000-page set of scanned legal contracts. Would you use the same chunking strategy for both?

What a strong answer covers:

• No: the wiki has markup, so header-based splitting with header metadata, capped by size
• Contracts have long clauses where qualifiers matter, and OCR noise; clause-aware splitting, overlap, and possibly propositions for key obligations, with grading
• Budgets the cost: propositions over 2,000 pages may be affordable; over 50,000 they probably are not
• Uses separate golden sets per corpus, because the right settings differ

4. You switch from recursive to semantic chunking. How do you know in production that it helped, and how would you notice if it silently got worse after a content change?

What a strong answer covers:

• Compares against the recursive baseline on the same golden set: hit rate, faithfulness, latency, index size
• Monitors the chunk length distribution; semantic chunks can balloon on uniform text or fragment on noisy text
• Watches the extra embedding cost at index time, and re-runs the golden set when the embedding model changes, because breakpoints depend on it
• Spot-checks boundaries on newly ingested document types

5. Proposition chunking improved retrieval precision in your offline eval. Before shipping, what failure mode worries you most, and how do you guard against it?

What a strong answer covers:

• The LLM rewriting step can drop qualifiers, merge facts or hallucinate, and the index would then serve text the source never said
• Grades every proposition against its source chunk (accuracy, completeness) and drops failures
• Keeps the source chunk id so answers cite and show the original text, not only the proposition
• Samples graded propositions for human review, and tracks the cost of regeneration when the source changes