Architecture Synthesis: Choosing a System Design: overview

T5.1 to T13.3 built more than a dozen ways to answer a question from documents, from one retrieval and one model call up to self-correcting loops, teams of agents and graph indexes. This module puts them on one map and asks which one a requirement needs. The T15.1 notebooks build on four ideas: choosing an architecture against a simpler alternative, the budget each extra component spends, late interaction as a retrieval upgrade that adds no model calls, and evaluating the whole pipeline end to end.

The running example is the Larkspur Cloud handbook from T5.1. Six questions carry the comparison, and an earlier module's lab has already run each one through several architectures: Can I export my data after I cancel my plan? (T7.2, T9.1, T9.2), What happens if my server goes down? (T6.1, T7.2, T9.1, T9.2), My server went down last night. Do I get service credits? (T11.1), Does Larkspur offer GPU servers? (T9.2), How does service continuity work? (T13.1) and Which plans support data export in the EU? (T13.2). This module reuses their counts and adds no new measurements.

• Start with plain RAG, one retrieval and one model call, and add one component at a time, each justified by a failure on the golden set. Every step up is paid in model calls: in the course's labs a router costs 2, a supervisor 3 to 5, CRAG 4 to 7 plus a web search and Self-RAG 7 to 20.
• Fix the stage that fails. On What happens if my server goes down? TF-IDF retrieves neither evidence chunk, and dense retrieval ranks both first and second (T6.1 lab) for the same single call. The T9.1 retry loop needs 10 calls to get there, and Self-RAG spends 14 without finding them.
• Two kinds of question need a different index. A theme across the handbook needs summaries built at index time (T13.1), and "available in the EU" needs an edge to check (T13.2). No retrieval setting stands in for either.
• Late interaction (ColBERT) keeps one vector per token and scores with MaxSim. A passage of up to 180 tokens at 128 dimensions is up to 23,040 numbers before compression, against 1,536 for one text-embedding-3-small vector, and it still makes one retrieval and one model call.
• Evaluate the pipeline on one golden set with a metric per stage: context recall for retrieval, faithfulness for generation, answer correctness end to end. In this course's locked environment import ragas fails, so the notebook's RAGAS cells do not run as shipped.

Every architecture in the course is plain RAG plus components: better retrieval, a router, a grader, a loop, more agents, a different index. Each component has a price in model calls, latency and parts to test, so it stays only when the golden set shows the failure it fixes. The simplest architecture that meets the requirement is the one this loop stops at.

The simplest architecture that works: open the file in a browser. It opens on the outage question with one handbook index, a budget of 5 calls and a cheap wrong answer: dense retrieval wins at 1 call, and the ladder shows TF-IDF and the hybrid blend falling short. Set the wrong answer to cost a support ticket and the only counted rung that has a check and answers is the T9.1 retry loop at 10 calls, so nothing fits until the budget reaches 10. Then pick the service-credits question and switch between one index and three knowledge bases: decomposition at 5 calls against a swarm at 3, with the router's 2 calls never asking billing.

• Before: every module. The baseline and the golden set come from T5.1; retrieval upgrades from T5.4 to T6.3; the self-correcting loops from T9.1, T9.2 and T9.5; routing from T9.3 and T11.1; tool-based retrieval from T7.2, T8.1 and T9.4; the RAG or fine-tune decision from T14.1; structural indexes from T13.1, T13.2 and T13.3.
• Next: the capstone projects. Each ships a working system, an evaluation report on a golden question set and a demo, and the T15.1 lab brief asks for the design document behind it: every component justified against a simpler alternative.

• Build plain RAG first and record its row on a frozen golden set; every later component has to beat that row on the questions it targets.
• Name the failing stage before adding anything. A retrieval miss is fixed in retrieval, at no extra model calls; a loop on top of it pays repeatedly for the same missing chunk.
• Count what each component costs in model calls, retrievals and index-time work, and set a budget per question before choosing; loops get a cap in code.
• Change the data when the question needs it: summaries for themes, explicit edges for constraints, separate knowledge bases only when routing to them buys something.