Reliability, Feedback Loops & Diversity: overview

A RAG answer can read well and still be wrong in three separate ways. It can rest on a chunk that has nothing to do with the question, it can state something no chunk says, and it can keep failing after users have said so. This module adds a check for each: graders before and after generation, quoted sources the user can verify, a feedback loop that re-weights retrieval, and a selector that stops duplicate chunks from crowding out the evidence. The T9.5 notebooks build on four ideas: catching unsupported answers, proving each quote, learning from user signals, and choosing diverse chunks.

The running example is the Larkspur Cloud handbook from T5.1: the same twelve chunks and the same TF-IDF index as the T5.1 top-k lab. The golden set keeps T5.1's three questions and adds two: Is there a credit if my server plan has an outage?, whose evidence ranks third, and Can I still use my server after I cancel the plan?, whose top answer carries an unrelated chunk along with the right one. The refund question, How long does a refund take?, runs through the whole module, because its answer sits in a chunk the retriever never returns.

• Lesson 1 of T9.5 wraps generation in two LLM graders. One answers yes or no for each retrieved chunk before generation, the other for whether the answer is grounded in the kept chunks. Grading removes noise but cannot add a chunk retrieval missed, and with source highlighting the three checks add 6 model calls per answer at k = 4.
• Source highlighting asks the model for verbatim quotes of the chunks it used. The parser checks only their shape, so find every quote in its chunk in code and fall back to citing the whole chunk when it is missing.
• The feedback re-ranker in lesson 2 of T9.5 multiplies each chunk's score by its average rating over 3, which keeps it between 0.333 and 1.667. In the lab, 100 rated answers lift mean recall@2 on the golden set from 0.50 to 0.70, the order stops changing after event 8, and 40 of the 41 wrong answers go to two questions whose evidence the retriever never returns.
• A rating on the whole answer lands on every chunk in it. In the lab an unrelated chunk rides a good answer up to 1.667, and an evidence chunk is pushed down to 0.333 next to the distractor it was served with.
• With the handbook stored five times, plain top-3 for the export question returns the export chunk three times. Dartboard selection returns export, retention, export at its default oversampling of 3, and export, retention, cancel at 4, once the pool holds three distinct chunks.

The checks in the top row act on one answer and can only remove: a grader drops a chunk or rejects an answer, and neither can add the chunk that was missing. The loop in the bottom row acts across many answers and can only reorder the chunks retrieval already returns. A missing chunk is a retrieval problem in both rows, and the fixes for it sit in T5.4 to T6.3.

The feedback flywheel: open the file in a browser. Pick Is there a credit if my server plan has an outage? and move the slider from 3 to 4 events: one thumbs-down drops sizing and reboot to 0.333 and the evidence chunk sla moves into the answer. Then slide to 100 and pick How long does a refund take?: 20 wrong answers, and no rating reaches refund-timing. Last, pick Can I still use my server after I cancel the plan? and switch between the two signals to watch sizing go from 1.667 to 0.333.

• Before: T5.1 supplies the golden set and the baseline this module's lab reports against. T9.1 introduced the document grader that 32.1 reuses, and T9.2 the groundedness check. T6.1 taught MMR, the older answer to the problem Dartboard solves, and T6.3 the cross-encoder reranker, a second way to reorder the same candidates.
• Next: T10.1 adds conversation memory, where feedback and grading have to handle follow-up questions that only make sense with the turns before them.

• Graders filter and verify; they never add evidence. When every chunk is graded irrelevant or the answer is not grounded, rewrite the query or abstain instead of answering anyway.
• Check source highlights in code: every segment must be found in the chunk it claims to come from.
• Feedback re-ranking moves only chunks that were shown. Read its result per question, and send questions it cannot fix to retrieval work.
• Remove duplicate chunks at indexing time, then use a diversity-aware selector with a candidate pool larger than the number of copies.