Inference and GPU infrastructure: validate and carry forward

1. What does KV memory depend on here? K/V count, layers, KV heads, head dimension, bytes, total cached tokens and active sequences.
2. Why is weight quantisation separate from KV? Reducing weight bits does not automatically change cached activation representation.
3. What must be ready before routing? The approved model/engine and its supported envelope, not merely a running container.
4. When is another provider safe after timeout? Only under explicit approved data/behaviour and known-outcome/reconciliation semantics.
5. What remains untested? Actual inference/GPU performance, deployment, tokenisation, network privacy and production cost.

The 60-second card: approved route; current access; stable operation; bound context/output; budget tokens/slots; separate prefill/decode; count weights/KV/runtime; profile before claiming speed; drain without erasing uncertainty. Review after about one, seven and 21 days.

• Explain the serving path and memory terms without notes in two minutes.
• Draw protected Gateway, budget guard, engine/replica, device and result boundaries.
• Defend hosted/self-hosted, one-device replica and sharding alternatives at their assigned depths.
• Diagnose queue, prefill, decode, memory and unknown-outcome failures using concrete signals.
• For build depth Gateway, run/modify the prior-stage integration and final-write tests. For design depth, complete the memory/deployment design exercise. For explain depth topics, explain purpose/boundary/trade-off and stop; learner completion remains unassessed.

ADR-009 and the snapshot preserve actual evidence.

Stop after T18.10. A future T18.11 considers fine-tuning/customisation only when behaviour requirements justify it, retaining approved data, release and serving boundaries. No training, GPU provisioning or production endpoint has been performed here.

• vLLM paged-attention design: one concrete KV-management implementation, not a benchmark used by this model.
• vLLM parallelism/scaling: serving-replica and sharding choices.
• Transformers inference optimisation: implementation examples of cache/compilation/optimisation concerns; current main documentation is not a pinned runtime here.
• Kubernetes GPU scheduling: device resources and placement vocabulary, not a deployed cluster.

Sources checked 2 November 2026. All hardware shapes, timings and costs in the worked examples are explicit assumptions. Validation distinguishes memory/control arithmetic from real hardware/inference measurements.

• KV memory depends on the K and V count, layers, KV heads, head dimension, bytes per element, total cached tokens and active sequences. Reducing weight bits does not change the cached representation.
• Route work only when the approved model, engine and supported envelope are ready, not merely when a container runs. Drain on replacement without erasing unknown outcomes.
• Another provider is safe after a timeout only under explicitly approved data and behaviour rules with known-outcome or reconciliation semantics.
• Memory and control arithmetic leave actual inference and GPU performance, deployment, tokenisation, network privacy and production cost untested.