Inference and GPU infrastructure: think like a staff engineer

Record approved bundle/model/engine identity, route type, queue wait, input/output tokens, TTFT, ITL, total completion, memory occupancy/eviction and bounded error reasons. Correlate with policy/audit, provider grants and durable result state. Keep raw context and credentials out of routine telemetry. GPU utilisation without queue/token/phase context is not a serving SLO.

When TTFT rises, distinguish admission wait, cold loading, prefill and network setup. When token streaming stalls, inspect decode scheduling, active sequences, memory pressure and communication rather than only app CPU. When memory errors occur after weights load, compare peak retained tokens/concurrency and actual overhead. A paging/eviction policy can protect memory while harming latency; measure both.

Quality and safety evaluation remain required for dtype, quantisation, engine, prompt or model changes. Continuous batching, prefix reuse and sharding can alter timing/reproducibility and operational isolation. Roll them out as versioned changes through T18.9 rather than untracked “performance-only” flags. Reject or degrade within approved policy when the supported envelope cannot fit.

A production test matrix needs the actual approved model/engine/hardware, context/output distributions, concurrent tenants/classes, startup/drain, failure/cancellation, token budgets, privacy/egress and recovery. No local memory calculator or fixture transport validates that matrix. The serving option is concrete enough to evaluate, but not secretly deployed by this lesson.

Prompt: Every app replica could call whichever endpoint looks cheapest. Why centralise the logical policy?

Spoken answer: “The gateway binds approved model/release identity, tenant data path, current access, context/output limits and shared budgets before dispatch. It normalises provider outcomes while preserving stable operation identity. That logical role may be implemented with replicated processes, but its authority must not become independent per-app quotas or user-selected destinations. Cheap routing does not justify bypassing policy or retrying unknown effects.”

Trade-off: A shared control boundary adds configuration and dependency management. Changed constraint: If different tenants approve different model/network routes, explain a trusted registry and rollout policy.

Prompt: Raw weights consume 14.9 GiB and the device budget is 24 GiB. Is deployment ready?

Spoken answer: “Not yet. I need peak KV for total cached input/output tokens and active sequences, plus real runtime/workspace/allocator overhead. In our hypothetical layout, eight 4,096-token sequences add four GiB of FP16 KV, and the assumed reserve brings total near 22.9. That arithmetic fit has little margin and proves no latency or throughput. I would profile the exact approved engine and workload.”

Trade-off: Conservative memory bounds reduce concurrency. Changed constraint: Doubling context length makes the example exceed 24 GiB; compare bounded context, active sequences, dtype and sharding without promising equal quality/speed.

Prompt: Two devices are available. Should the model always use tensor parallelism?

Spoken answer: “No. If a full model and peak request state fit one device, independent replicas may better serve separate work. Sharding can enable a larger model or another execution profile, but introduces communication/topology and coordinated failure/readiness. Actual weight and KV partitioning depends on model and engine support. I would decide from fit and measured workload, not divide all memory and latency by two.”

Trade-off: Memory fit versus communication/operations. Changed constraint: If the devices are on different hosts with a slow link, what additional evidence would you require?

Prompt: A request completed on the old model, then the candidate becomes stable. What should same-key replay return?

Spoken answer: “The original committed result and bundle, after the required current access checks on its original evidence. A release pointer changes future operations, not prior facts or duplicate identity. If the original source access is revoked under this grounded-history policy, deny replay rather than substitute unrelated allowed evidence. An incomplete operation remains inspectable; it is not permission to call the new route again.”

Trade-off: Conservative replay/access policy can withhold previously visible text. Changed constraint: If product requirements allow immutable historical access, define that separate auditable policy rather than weakening current source checks accidentally.

• Record bundle, model and engine identity, queue wait, input and output tokens, TTFT, ITL, total completion and memory occupancy. GPU utilisation without queue, token and phase context is not a serving SLO.
• When TTFT rises, separate admission wait, cold loading, prefill and network setup. When streaming stalls, inspect decode scheduling, active sequences, memory pressure and communication rather than app CPU.
• If a full model and its peak request state fit one device, independent replicas may serve separate work better than tensor parallelism. Sharding adds communication and coordinated readiness, so never divide all memory and latency by the device count.
• Same-key replay after a release change returns the original committed result and bundle, after current access checks on its original evidence. If that access is revoked, deny replay rather than substitute other evidence.