Inference and GPU infrastructure: observe and experiment

The offline walkthrough compares an approved dispatch, a forbidden/unready route and a lost provider response. Read the static boundary table before advancing the interactive steps.

Sequence diagram. No branch creates a second provider call after an uncertain outcome. Identity, route approval, budget reservation and result commit remain distinct evidence boundaries.

Assume a hypothetical dense decoder with 8 billion parameters, 32 layers, 8 KV heads, head dimension 128 and FP16 KV elements. These are authored architecture inputs, not a named model recommendation. Let cached tokens mean the total retained input plus generated tokens at the evaluated instant.

Raw FP16 weights require 8,000,000,000 × 2 = 16,000,000,000 bytes, about 14.90 GiB. For this KV layout, bytes per cached token per sequence are 2 (K and V) × 32 layers × 8 KV heads × 128 dimensions × 2 bytes = 131,072 bytes, or 128 KiB.

At 4,096 cached tokens and eight active sequences, KV memory is 131,072 × 4,096 × 8 = 4 GiB. Add an assumed four-GiB runtime/workspace reserve: total is about 14.90 + 4 + 4 = 22.90 GiB. That is within an assumed 24-GiB budget with little unmodelled headroom. This reserve is not measured and does not guarantee an allocation will succeed.

At 8,192 cached tokens and the same eight active sequences, KV becomes eight GiB and total about 26.90 GiB, exceeding 24. Raw four-bit weights would be about 3.73 GiB, but the worksheet leaves KV in FP16. Total then becomes about 11.73 GiB at the original 4,096/eight setting. Actual quantisation metadata, workspaces, padding, allocator fragmentation and engine overhead can differ; changing weight bits does not automatically quantise KV or preserve answer quality.

For a latency worksheet, if queue/verification/retrieval total an assumed 300 ms, prefill 200 ms, and decode 300 tokens at an assumed 30 ms/token, total before final persistence is about 9.5 seconds. A first token near 500 ms does not establish a five-second saved-answer SLO. These are arithmetic assumptions, not a throughput benchmark or a prediction for the memory example's hardware.

Open the single-device memory experiment. Four controls vary total cached tokens/sequence, active sequences, raw weight bits (16 or 4) and assumed VRAM. KV stays FP16 and the runtime reserve stays four GiB. No GPU, kernel, scheduling engine or real model is invoked.

A “within” decision means only that this arithmetic sum is below the entered capacity. It is not ready-for-production status, a token-rate prediction, a quality check or proof of memory isolation. The static table remains useful without JavaScript.

Run this stage's Python lab in your terminal to see each idea in the stage hold. It needs Python 3.10 or later and nothing else: the standard library and SQLite only.

From the root of your local clone of the repository, run:

python3 10_AI_System_Design/stages/09-inference-gpu/labs/test_inference.py -v

-v prints one line per test, and each test name states one behaviour. Read a test, change one value in the lab code, predict which test fails, and run it again.

inference.py independently computes memory. gateway.py composes actual prior Security, Gate and Releases fixtures plus a durable operation table. Twelve tests check approved dispatch, readiness/private constraints, registry/provider denial, context/output bounds, quota before transport, unknown replay without retry, original bundle preservation across release changes, current revocation on saved replay, input conflicts, memory arithmetic, final-write failure without completed acknowledgement or a second call, and partial revocation of original multi-source evidence.

For build depth gateway practice, fail transport after a fresh grant, change the release pointer, and retry the same key. The result must remain unknown/original rather than silently invoke a new model. Fail the final result update after fixture generation: replay must expose incomplete dispatching evidence, not a completed answer. For design depth serving/orchestration practice, write a deployment/readiness/drain checklist and a representative load-test plan. Explain depth GPU topics stop at explaining memory, placement and the relevant trade-off.

For hosted inference, expected request cost is input tokens times the input price plus output tokens times the output price, using the provider's units and actual usage. For self-hosting, allocate GPU/runtime cost, idle/reserved headroom, storage/network, maintenance and operator effort across successfully served approved requests. Include quality/refusal and reliability differences rather than counting every generated token as useful work.

As a deliberately dimensionless example, an all-in serving replica costing C=10 units/hour with an assumed busy-capacity q=20 successful requests/second and effective utilisation u=0.5 has infrastructure allocation C/(3,600qu) ≈ 0.000278 units/request, before any costs omitted from C. Neither C nor q is a current price or measured throughput here. If traffic is bursty and effective utilisation drops, self-hosted idle cost per success rises. Do not make a purchase decision from this worksheet.

A permitted hosted request times out after dispatch. An available self-hosted route has spare memory. May the gateway automatically retry there with a new operation identity?

• In the worked example, 8 billion FP16 parameters need about 14.90 GiB and each cached token costs 128 KiB of KV per sequence. Eight sequences at 4,096 tokens add 4 GiB, and a 4 GiB reserve brings the total to about 22.90 GiB against 24.
• At 8,192 cached tokens KV reaches 8 GiB and the total about 26.90 GiB, over budget. Four-bit weights shrink weights to about 3.73 GiB but leave KV in FP16, and a "within" result proves no speed or quality.
• With 300 ms of queue and retrieval, 200 ms of prefill and 300 tokens at 30 ms each, the answer takes about 9.5 seconds before persistence. A first token near 500 ms does not establish a five-second saved-answer SLO.
• After a dispatched request times out, the gateway never retries on a spare self-hosted route under a new operation identity. It preserves the unknown outcome and uses a documented reconciliation contract.