Feedback re-ranks only what retrieval returns.
Five Larkspur Cloud golden questions take turns. Each answer uses the top two chunks, and a simulated user rates it: thumbs up (5) when every evidence chunk is in the answer, thumbs down (1) otherwise. The re-ranker is the notebook’s rule: relevance_score starts at 1.0 and is multiplied by the average rating over 3.
- Recall@2 before feedback
- Recall@2 now
- Wrong answers served, this question
- Wrong answers served, all five
Candidates after re-ranking
Top four by TF-IDF, then sorted by relevance_score
| Rank | Chunk | TF-IDF | Ratings | relevance_score | Golden set |
|---|
The golden set at this point
| Golden question | Before | Now | Wrong |
|---|---|---|---|
| Mean recall@2, and wrong answers in total |
The notebook asks gpt-4o whether each stored feedback record is relevant to the current query and chunk. The lab uses a fixed rule in its place: a record counts when it was given on the same question and its answer used that chunk. Ties keep the TF-IDF order, as Python’s stable sort does.