Inference and GPU infrastructure: request walkthrough and lab
Part of the Staff AI System Design course. Everything runs in your browser and makes no network requests. Every figure is illustrative.
Request walkthrough
- Responsible component
- Input
- Output and state
- Possible failure
Lab: Change Memory Assumptions Without Claiming Speed
What fits in the assumed single device?
Hypothetical dense 8B model: 32 layers, 8 KV heads, head dimension 128; KV stays FP16. Fixed four-GiB runtime reserve. Raw estimates omit real allocator/quantisation overhead; no model, GPU or throughput measurement.