AI System for an Air-Gapped Environment

Anchor the Load Envelope to Customer Reality

• Candidates often slip on the first pass. They picture an on-prem service that analyses documents, guess at a few dozen active users, and settle on one model replica behind a queue. That holds up on an average day. Then the customer names a deadline, their busiest day or a date by which the backlog must be gone, and the plan falls over.
• An enclave gives you little or no room to scale out later. Get the capacity plan right before anything ships.
• Start from how the customer really works. The case fixes four facts: 1,000 users, 20 QPS (queries per second) at peak, a corpus of 50 million pages, and a GPU pool that cannot grow.
  • Those figures pick the architecture for you. The options are batch processing, online interactive extraction, or a split where indexing runs asynchronously under a smaller interactive layer.
  • They also decide what failure looks like: a "slow response", or a "missed mission deadline".
• Keep four figures apart, since each answers its own question:
  • Average load → describes ordinary daily behaviour.
  • Peak load → guards you when bursts pile up in the queue.
  • Growth → tells you whether the design lasts through a year of adoption.
  • Headroom → stops you running on the brink of collapse when a large document lands.

Storage Sketch

• Model weights and runtime packages, kept as one primary release and two versions you can roll back to.
• Vector or search indexes covering 50 million pages, with room for metadata and sharding overhead.
• Logs, audit trails and operational metrics, held for as long as the customer's local policy requires.
• Short-lived staging space for signed updates, validation artifacts and the packages behind a canary rollout.
• Real sizes hinge on the model you pick, how you chunk, the embedding dimension, the retention policy and compression. Skip false precision, but show that storage is never "just the model".
• Keep those rollback copies. Inside an enclave, reverting can be slow, or the approval process can hold it up, so a bad release may stay live for a long time.

Memory and Throughput per Replica

• Work out two figures for each replica: the memory it needs and the work it can serve.
  • Memory has to fit model weights, the KV (key–value) cache (or an equivalent state), framework overhead and batch buffers.
  • Throughput normally means tokens per second for one replica, given the model, quantisation and batch size you settled on.
  • Quote GPU count with no throughput and you miss the bottleneck. Quote memory with no throughput and you miss the placement constraint.
• A memory estimate at interview scale (my own illustrative figures; none come from real hardware):
  • Weights after quantisation: 12 GB.
  • Runtime and framework overhead in the serving stack: 2 GB.
  • KV cache plus transient activations at the target context length: 6 GB.
  • Batch buffers, with slack for fragmentation: 2 GB.
  • Sum: about 22 GB of GPU memory for each replica.
  • A 24 GB card leaves too thin a margin for a noisy batch or a context longer than planned. You would probably step up to a 32 GB GPU, or else pick a smaller model, cap context harder, or quantise further.
  • Treat the estimate as a way to decide, not an exact promise. Its job is to show whether today's class of hardware can carry the workload at all.

The Replica Sizing Formula

• One equation drives the sizing:

$$ Replicas = \left\lceil \frac{QPS \times Tokens_{request}}{TokensPerSecond_{replica} \times UtilizationTarget} \right\rceil $$

• Terms: QPS is how fast requests arrive. Tokens_request is the mean token work in one request. TokensPerSecond_replica is what a single replica can sustain. UtilizationTarget is the share of peak capacity you will spend before latency gets too bad.
• The top of the fraction turns request rate into token demand. The bottom turns one replica into the capacity it can really offer once headroom is set aside.
• Suppose the answer is larger than the fixed GPU pool. Hoping harder is not a plan. Cut the token work in each request, use a smaller model, batch harder, relax the SLO (service level objective), shift work into offline preprocessing, or agree to a smaller interactive feature set.

Worked Example: 1,000 Users, 20 QPS

• The case supplies 1,000 users who drive a peak of 20 QPS in a surge window. Everything else is my assumption. A typical request costs 2,000 tokens of model work, covering retrieval augmentation and the generated output. At our chosen batch shape, a replica holds 120 tokens/sec. I'd aim for 70% utilisation:

$$ Replicas = \left\lceil \frac{20 \times 2000}{120 \times 0.70} \right\rceil = \left\lceil \frac{40000}{84} \right\rceil = 477 $$

• The result is meant to feel wrong. It says the assumptions are likely wrong for a government deployment behind an air gap, where the GPU pool is fixed.
• Do not defend 477. Question the request profile:
  • Not all of the 20 QPS may be LLM (large language model) inference. Much of it could be retrieval or metadata lookups.
  • Chunking, caching or form-based extraction might shrink the average model work behind each request.
  • The customer's workflow could really be batch-first. Then a queue with a nightly SLA (service level agreement) is the better fit.
• This is why you estimate at all: the arithmetic forces the design towards something real.

🎯 Interview Pointer: Interviewers often choose numbers that give a ridiculous replica count on purpose. They want to see whether you cling to the result or question the assumed request mix. State the formula, reach the awkward number, then move to how the workload shape could change. Do not ask for more hardware.

Sensitivity Analysis: Baseline vs. 10x Growth

• Strong answers give ranges instead of fake exactness. A sensitivity view beats one heroic number because it shows which design lever matters most.

• Direction matters more than the exact figures. In an enclave, ten times the load is not fixed by buying more of the same. The fixed GPU pool, the storage and the operational process cannot stretch on demand.

What Changes First at 10x Growth

• More users? Identity, audit and queueing feel the strain before raw inference does.
• More pages? The index grows, and its rebuild time or storage bandwidth may become the limit.
• Longer documents? The bottleneck moves to token count and context length.
• Stricter SLOs? You need more headroom, and the fixed GPU pool runs short.
• State uncertainty plainly: "These numbers are illustrative. I'd build the first deployment around the 20 QPS peak, plus a growth factor and some rollback margin. If the real request mix leans towards batch, the interactive pool gets smaller. If the average request is longer, we either add replicas or narrow what each request may ask for."
• I expect per-request token demand to steer component choice and partitioning more than anything else:
  • Light requests → one general-purpose inference tier with retrieval may do the job.
  • Heavy requests → the design tends to split three ways: cheap preprocessing up front, a retrieval/rules layer in the middle, and a small set of expensive inference workers.
  • Inside an enclave, each added replica eats scarce offline capacity. Each added dependency makes updates and audits heavier.

SLIs Tied to the Workflow

• Pick SLIs (service level indicators) whose movement a customer would notice:
  • Availability: share of working hours in which the system takes jobs and hands back results.
  • Latency: how soon a submitted document yields its first useful result; separately, how long a whole batch takes.
  • Freshness: time from the arrival of approved content until it can be searched or used by the model.
  • Quality: accuracy of extraction, relevance of retrieval, or the rate at which analysts accept output.
  • Security: whether the network, identity, signing and audit boundaries hold.
  • Cost: GPU hours, storage footprint, support load and what a rollback costs.
• Fit objectives to the workflow. If analysts review results interactively, the interactive latency budget is the SLO that matters most. If ingestion runs nightly, freshness and completion time outrank latency per request.
• A latency budget breaks the end-to-end target into parts you can control: upload, preprocessing, retrieval, inference, validation and rendering the response.
  • When inference eats most of that budget, you likely need a smaller model, a shorter context or retrieval done ahead of time.
• What to say in the interview: work out size from the peak request rate, the average token work and the throughput of the fixed GPU pool. Turn that into a replica count, a storage footprint and headroom. Keep average load apart from peak load. Build SLOs around the customer's workflow. Use sensitivity ranges to find the assumption that moves the design most. Reshape the workload before you ask for unlimited hardware.

Two Worlds Separated by a Controlled Update Path

• The clearest way to defend this design is to walk one request through it twice. On the first pass everything works. On the second, something critical has gone missing.
• Picture the scenario. Inside the restricted or classified network, a user submits a packet of documents. They want it classified, want fields pulled out of it, or want it redacted. The answer has to be useful and auditable, and producing it must not rely on the internet at all.
• One hidden constraint shapes everything else. Software, models and metadata reach the offline side through a single controlled update path and no other way. So we deliberately split the design into two worlds: an online side where things get built, and an offline side where they run.

Nine Components in Dependency Order

• Present the components in the order they depend on one another, not the order they would sit on a slide:
  1. Connected build environment — gathers source code, pinned dependencies and model artifacts ahead of export.
  2. Artifact signing service — signs whatever bundle crosses the boundary and issues attestations that anyone can check.
  3. Transfer staging zone — holds each bundle in quarantine for inspection until the offline side admits it.
  4. Offline registry and package mirror — the one trusted local store for every approved model package, container image and library.
  5. Local orchestrator — runs canary rollouts, applies deployment policy and decides where services run.
  6. Model server — hosts the local model, or the inference runtime, that the document pipeline calls.
  7. Document pipeline — takes documents in, runs OCR/parsing, chunks them, retrieves context, scores, redacts and builds the final result.
  8. Local identity provider — confirms who each user and service is, without leaving the boundary.
  9. Local telemetry stack — collects audit events, logs, metrics and traces, and exports none of it.
• The main trust boundary separates the build environment, the only connected part, from everything else.
• Boundaries inside the enclave count too: orchestrator versus workloads, identity versus services, document pipeline versus telemetry store. Cutting the network link does not make untrusted code vanish. It moves the threat model away from internet exposure and towards admission control, least privilege and a strong audit trail.

Top-Down Architecture Diagram

• Each box earns its place by meeting a requirement. None is there just to fill out the picture:
  • Connected build environment → inputs that can be rebuilt identically.
  • Signing service → evidence of exactly what crossed the boundary.
  • Transfer zone → unvetted material never lands straight on the offline side.
  • Registry and mirror → deployments stay deterministic.
  • Orchestrator → keeps the runtime under management.
  • Model server and document pipeline → do the actual work.
  • Identity and telemetry stacks → every access is authenticated and every operation can be audited.

System of Record vs. Cache vs. Queue

• Getting this distinction right is what sets a strong answer apart from an architecture that merely looks good:
  • System of record: approved packages and images live in the offline registry. User and service identities live in the identity provider. The audit log records each deployment and its time.
  • Cache: model weights kept locally in memory, retrieval indexes or frequently used document fragments, and any response cache that saves repeating costly computation.
  • Queue: jobs that ingest documents, tasks that scan, attest and export, and background jobs for enrichment or post-processing.
  • External dependency: end-user traffic has none. The single thing from "outside" is the controlled build pipeline that produces the offline bundle.
• A frequent error is to let a cache act as if it were the source of truth. Air gaps make this worse. Nothing online refreshes it, so stale entries can linger unnoticed for a long while.
  • When the registry marks an image as approved, the runtime must not quietly fall back to an older copy it has cached.
  • A service credential revoked in the identity provider must be revoked everywhere, consistently, by the local control plane.

🎯 Interview Pointer: Interviewers love to probe the split between system of record, cache and queue. For every component you draw, be ready to say which bucket it belongs to and what breaks once its data goes stale.

End-to-End Happy Path

• Tell the happy path as a sequence of explicit state changes:
  1. Build from pinned dependencies. On the connected side, pin code, system packages and model package versions to known revisions.
  2. Scan, attest, sign, and export bundle. The bundle is checked for policy violations, given a provenance attestation, signed, then packaged ready for export.
  3. Inspect in the transfer zone. Before anything moves inward, the zone confirms integrity, policy conformance and chain-of-custody.
  4. Verify signatures offline. This proves that an authorised signer produced the bundle and that nothing altered it on the way.
  5. Import to local registry. The offline runtime can then pull approved model files, images and libraries from the local mirror.
  6. Deploy canary. The orchestrator runs a small share of model-serving and document-pipeline capacity against harmless test data.
  7. Run acceptance tests. They check authentication, ingestion of documents, latency, the quality of outputs, logging, and whether rollback is ready.
  8. Promote or rollback. A passing canary gets more traffic. A failing one is reverted, and its bundle goes into quarantine for review.
• Building, scanning, attesting, exporting and inspecting are all admin work, and they run asynchronously. In the live system, deployment, request handling and the inference users see are all synchronous.
• State it plainly to the interviewer. Document requests need a synchronous answer from the offline runtime. Updates and the supply chain run in the background, so a user never waits on them.

Sequence Diagram of the Happy Path

Dependency-Failure Drill: Fail Closed

• The fail-closed drill you must be able to run: one transitive package is absent.

• When the package mirror lacks a dependency, the offline system must fail closed. It must never improvise by reaching across the boundary for the missing piece.
• The rule: any unresolved dependency blocks promotion. The bundle stays quarantined in staging or the mirror until someone repairs the dependency and it is approved again.

Backpressure and Flow Control

• Without backpressure, a burst of document uploads can melt an offline service. With it, the service stays stable.
  • At ingestion: cap the number of concurrent uploads, check file size and type as early as possible, and queue oversized jobs rather than letting them grab parser resources straight away.
  • At the pipeline: shard each document's work on a fixed key: case ID, tenant or document ID. Related state then lands in one place and retries behave deterministically.
  • At the model server: limit concurrency, put a ceiling on batch size, and reject or defer work once GPU/CPU saturation passes a safe threshold.
  • At the orchestrator: use rollout limits so the canary sees only a small share of traffic until acceptance tests pass.
• The choice of partitioning key matters. It keeps related work local and stops state from one document corrupting another. With document ID as the key, a retry or a partial failure is much simpler to reason through than when requests land on random workers. That matters most when a document's context, its redaction choices and its audit trail must agree.

Component Responsibility Table

MVP vs. Later Evolution

• Keep the MVP (minimum viable product) small and opinionated. It needs one each of offline registry, orchestrator, identity provider, telemetry stack, model server and document pipeline, plus a clear canary process.
• Signing, verification and the transfer zone are not optional extras. They ship in the first release, because without them offline updates cannot be defended.
• What comes later: smarter scheduling, routing across several models, better retrieval, reindexing in the background and richer policy engines. These are follow-on features, not prerequisites.
• The signal for the interviewer: first make the offline path dependable and auditable. Only then tune throughput and make life easier for operators.
• Failure overlay to describe out loud: a bundle arrives, but the offline mirror lacks one transitive package. Promotion stalls in the orchestrator, which leaves the canary isolated. The telemetry stack logs the failed rollout. The registry holds the bundle in quarantine. An operator repairs the package on the connected build side and retries.

Control Plane vs. Data Plane

• A diagram only helps if you can talk through how data, identity, state and failure move across it.
• Control plane: signing, verification, deployment, identity and policy.
• Data plane: inference and live document requests.
• Be clear about which workflows are synchronous and which are asynchronous, and about which system owns every durable state transition.
• Why it matters in the job market: this split lets you explain one architecture to customer stakeholders and engineering stakeholders alike.
  • What the customer hears: "The system stays inside our boundary, and we can still take controlled updates."
  • What engineering hears: "State has a known home, failures have a known edge, and promotion holds no surprises."
• 90-second interview summary: "I'd keep the connected build side apart from the offline runtime, with signing and staging as the only bridge. Inside the enclave sit a local registry, an orchestrator, a model server, a document pipeline, an identity provider and a telemetry stack. Requests are synchronous. Updates are asynchronous and tightly controlled. Deployable artifacts have one system of record: the offline registry. Local telemetry and the identity provider hold the audit trail. My biggest trade-off is security versus how hard updates become. The first production gate is simple: no bundle enters the enclave until it has proved, offline, that it is intact, complete and compatible. After that, a canary proves signature provenance, complete dependencies, working authentication and a rollback path before wider promotion."

Three Governable Records

• ReleaseManifest(version, artifact_hashes, signatures, dependencies) — the promise a bundle makes, and so the contract you deploy against.
  • Primary key: version. Operators need one stable name they can cite, rebuild against and promote.
  • Lifecycle: written in the connected build environment, then signed, carried into the offline side, validated and staged. Finally it is promoted or turned away.
  • Retention: keep the manifest, its signature material and enough dependency detail to show later why the release passed or failed. Audit needs and local policy set the period; no universal standard applies.
• ModelDeployment(model_id, version, hardware, status) — the runtime truth about which model is live, and where.
  • Preferred primary key: deployment_id. When one model family can sit in several release states across different targets, add a unique constraint over (model_id, version, hardware).
  • Without a dedicated deployment identifier, the composite (model_id, version, hardware) is the natural key. Pick one on purpose.
  • Lifecycle states: staged, validated, active, draining, retired.
  • Retention: hold enough history to answer "Which model served this document when we analysed it?"
• OfflineAuditEvent(actor, action, artifact_hash, time) — which actor took which action, on which artifact, at what moment.
  • Primary key: usually an event ID. The (actor, action, artifact_hash, time) tuple serves as the business key.
  • Lifecycle: append-only. Nobody rewrites an audit row once it exists.
  • Retention: follow the customer's audit policy. Purge too eagerly and trust goes; hoard too much and you may break local retention rules.
• Ownership boundaries: the build team holds signed release material up to the handoff into offline validation. The offline runtime holds deployment state. Security/operations holds the audit trail. This split stops a government network from turning into one huge mutable blob.

API Contracts

• POST /v1/documents/analyze — the data-plane call that users hit.
  • Request: an authenticated document payload, plus optional metadata such as an idempotency key or an analysis profile.
  • Response: a typed analysis result with a pointer to the model version and a request identifier.
  • Authentication stays inside the boundary, through an offline identity provider or a token minted on the network itself. "Just use OAuth" is not an answer.
  • Idempotency: if one idempotency key arrives twice with an identical body, the caller gets the same result, or a stable "already processed" reply.
  • Errors separate bad input, unauthorised access, no free model capacity and a policy rejection.
• GET /v1/models/status — the read path for operators.
  • It reports each model's deployment state, the live version and whether each hardware target is ready.
  • Repeated calls change nothing, so they are harmless. Callers still authenticate, because status data can reveal operational details.
  • Its errors tell apart "unknown model", "not deployed" and "temporarily unavailable".
• POST /v1/admin/releases/validate — the gate in the control plane.
  • It takes a pointer to a bundle or the uploaded bundle itself. It checks the manifest signature, compares artifact hashes and confirms every declared dependency is on hand. It returns a structured pass/fail result or a validation report.
  • It is idempotent per release identifier. Two runs on one bundle yield a single logical validation record, unless the content changed in between.
• POST /v1/admin/releases/{id}/promote — moves a release from validated to active.
  • Three guards protect it: authorisation, policy checks and optimistic concurrency.
  • It refuses when the release cannot be promoted from its current state, when the target deployment moved since the client last read it, or when the live release is not the one the client expected.
  • Request and response shapes carry versions, so a later change to the manifest or to validation output cannot break older tooling overnight.
• Versioning does not stop at public APIs. The bundle format, the manifest schema and the audit event schema need it too. When nobody can update the offline environment at will, each breaking change must be planned as a migration, never treated as a light refactor.

Release Verification: The Highest-Risk Code Path

• No component carries more risk than release verification. If the system cannot check a bundle offline, the rest is theatre. Hence the first production gate: no bundle enters the enclave until it has proved, offline, that it is intact, complete and compatible.

from __future__ import annotations

import hashlib
import hmac
import json
from dataclasses import dataclass
from typing import Any, Mapping


class IntegrityError(Exception):
    pass


class ManifestError(Exception):
    pass


class SignatureError(Exception):
    pass


class ValidationError(Exception):
    pass


@dataclass(frozen=True)
class VerifiedManifest:
    version: str
    artifacts: tuple[dict[str, str], ...]
    dependencies: tuple[str, ...]


def canonical_json(obj: Any) -> bytes:
    return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode("utf-8")


def sha256_hex(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def verify_signature(trusted_public_key: bytes, payload: bytes, signature: bytes) -> None:
    # Interview sketch: replace with a real signature primitive such as Ed25519 verification.
    expected = hmac.new(trusted_public_key, payload, hashlib.sha256).digest()
    if not hmac.compare_digest(expected, signature):
        raise SignatureError("manifest signature verification failed")


def _require_str(mapping: Mapping[str, Any], key: str) -> str:
    value = mapping.get(key)
    if not isinstance(value, str) or not value.strip():
        raise ManifestError(f"missing or invalid string field: {key}")
    return value


def _require_list(mapping: Mapping[str, Any], key: str) -> list[Any]:
    value = mapping.get(key)
    if not isinstance(value, list):
        raise ManifestError(f"missing or invalid list field: {key}")
    return value


def verify_bundle(bundle, trusted_public_key: bytes) -> VerifiedManifest:
    manifest_raw = bundle.read_bytes("manifest.json")
    signature = bundle.read_bytes("manifest.sig")

    try:
        manifest = json.loads(manifest_raw)
    except json.JSONDecodeError as exc:
        raise ManifestError("manifest.json is not valid JSON") from exc

    if not isinstance(manifest, dict):
        raise ManifestError("manifest root must be an object")

    version = _require_str(manifest, "version")
    artifacts = _require_list(manifest, "artifacts")
    dependencies = _require_list(manifest, "dependencies")

    verify_signature(trusted_public_key, canonical_json(manifest), signature)

    normalized_artifacts: list[dict[str, str]] = []
    for idx, item in enumerate(artifacts):
        if not isinstance(item, dict):
            raise ManifestError(f"artifacts[{idx}] must be an object")
        path = _require_str(item, "path")
        expected_hash = _require_str(item, "sha256")
        if len(expected_hash) != 64 or any(c not in "0123456789abcdef" for c in expected_hash.lower()):
            raise ManifestError(f"artifacts[{idx}].sha256 is not a valid hex digest")
        actual_hash = sha256_hex(bundle.read_bytes(path))
        if not hmac.compare_digest(actual_hash, expected_hash):
            raise IntegrityError(path)
        normalized_artifacts.append({"path": path, "sha256": expected_hash})

    normalized_deps: list[str] = []
    for idx, dep in enumerate(dependencies):
        if not isinstance(dep, str) or not dep.strip():
            raise ManifestError(f"dependencies[{idx}] must be a non-empty string")
        normalized_deps.append(dep)

    return VerifiedManifest(
        version=version,
        artifacts=tuple(normalized_artifacts),
        dependencies=tuple(normalized_deps),
    )

• Walk through it piece by piece:
  • canonical_json turns the manifest into deterministic bytes before the signature check, so one logical manifest always hashes the same way.
  • sha256_hex produces a stable digest to compare artifacts against.
  • verify_signature is labelled a sketch on purpose. Production code needs a genuine asymmetric primitive where this HMAC (hash-based message authentication code) stand-in sits.
  • Three exception classes tell the caller what broke: the signature, the shape of the manifest or the integrity of the content. Each calls for a different operator response.
  • VerifiedManifest is frozen, so nothing can quietly alter a successful validation result afterwards.
  • _require_str/_require_list check types at the boundary. Malformed input fails early, before any state moves.
  • verify_bundle reads the manifest and signature, parses the JSON defensively and checks the required fields. It verifies the signature over the canonical form, then compares each artifact hash with hmac.compare_digest to avoid timing leaks. It returns a normalised, typed result that a later promote step can rely on.

Verify-Bundle Flow

What the Snippet Omits on Purpose

• The sketch leaves out concurrency, retries and observability. They belong to the production layer around the core path.
• Concurrency: promotion should rely on optimistic concurrency in the control plane. Each release object holds a version/revision token that the promote call sends back. If another operator has already moved the deployment state on, the server refuses the write. Two people cannot then push incompatible releases live at once.
• Retries: retry safe reads and validation calls automatically, and nothing else by default. Retry a promotion only when it was built to be idempotent and a write token guards it. Document analysis requests should take an idempotency key, so a client retry creates neither duplicate work nor duplicate audit rows.
• Observability: surround the path with logs, counters and structured audit events. Each validation pass should record who validated which bundle, which hash it checked and, on failure, the reason. An air gap has no external vendor console, so these records are the only practical way to rebuild an incident.

Failure Behaviour to State Explicitly

• Duplicate requests reveal whether a design truly handles idempotency. Say an operator submits one validated bundle twice because the first response timed out. With a matching idempotency key and bundle hash, the second call returns the same validation outcome and does not move state forward again.
• If a promote request is resent after a transient fault inside the boundary, the server either returns the state already reached or rejects the repeat with a clear, stable error: this version is not waiting for promotion any more.
• The key drill is a missing transitive package. When a manifest names a dependency the offline artifact store lacks, validation must fail before installation, not at runtime. That is why dependencies sits in the manifest rather than in someone's head.

Contract and Failure-Injection Tests

• Full coverage is not the goal. The tests only need to show the candidate knows how the deployment gate acts on a pass and on a failure.

class BundleStub:
    def __init__(self, files):
        self.files = files

    def read_bytes(self, path):
        return self.files[path]


def test_verify_bundle_accepts_known_good_bundle():
    manifest = {
        "version": "2024.10.1",
        "artifacts": [{"path": "model.bin", "sha256": ""}],
        "dependencies": ["tokenizer.json"],
    }
    files = {
        "model.bin": b"weights-bytes",
        "tokenizer.json": b"vocab-bytes",
    }
    manifest["artifacts"][0]["sha256"] = sha256_hex(files["model.bin"])
    manifest_bytes = canonical_json(manifest)
    key = b"trusted-key"
    sig = hmac.new(key, manifest_bytes, hashlib.sha256).digest()
    bundle = BundleStub({
        "manifest.json": manifest_bytes,
        "manifest.sig": sig,
        **files,
    })

    verified = verify_bundle(bundle, key)

    assert verified.version == "2024.10.1"
    assert verified.artifacts[0]["path"] == "model.bin"
    assert verified.dependencies == ("tokenizer.json",)


def test_verify_bundle_rejects_corrupted_artifact_hash():
    manifest = {
        "version": "2024.10.1",
        "artifacts": [{"path": "model.bin", "sha256": "0" * 64}],
        "dependencies": ["tokenizer.json"],
    }
    manifest_bytes = canonical_json(manifest)
    key = b"trusted-key"
    sig = hmac.new(key, manifest_bytes, hashlib.sha256).digest()
    bundle = BundleStub({
        "manifest.json": manifest_bytes,
        "manifest.sig": sig,
        "model.bin": b"tampered-weights",
        "tokenizer.json": b"vocab-bytes",
    })

    try:
        verify_bundle(bundle, key)
        raise AssertionError("expected IntegrityError")
    except IntegrityError as exc:
        assert str(exc) == "model.bin"

• Test 1 (contract test): with a valid manifest, a valid signature and matching artifact hashes, the verifier returns a normalised VerifiedManifest holding the expected version and dependency list.
• Test 2 (failure-injection test): it fakes corruption with a wrong hash and altered artifact bytes, then checks that IntegrityError names the exact path that failed.
• A real repository would add a third test that damages manifest.sig and expects SignatureError, since a bad signature and a bad artifact need different operational responses. For a whiteboard sketch, one contract test and one failure-injection test are enough.

Why This Is a Strong FDE Answer

• An FDE (Forward Deployed Engineer) does more than draw the architecture. The job is to turn it into production-grade detail a customer can trust.
• The lesson: an answer earns credibility once its state changes, API contracts and failure-safe code are spelled out.
• 90-second implementation summary: I'd ship each release as a signed manifest that lists every artifact hash and declares every dependency. Deployment state lives apart from audit state. Versioned endpoints cover analysis, status, validation and promotion. Every write path is idempotent and uses optimistic concurrency. Release verification is the riskiest part, so I'd build and test it first: typed checks on the manifest, a signature check, a hash check per artifact, and clearly structured failure modes. The result is maintainable AI that runs locally and never crosses the enclave's network, artifact, identity or audit boundaries.

Lead With the Failure, Not the Model

• Open the way a security and operations reviewer would: with the incident that hurts most. The update package lands, yet one transitive package never made it into the offline mirror.
• "Just retry" is the wrong reflex. With no network, a retry cannot conjure an artifact nobody copied in.
• A good answer does something else. It halts the promotion, keeps the manifest and the evidence intact, and shrinks the damage to the smallest slice of the system it can.

Threat-Model the Offline Control Plane

• Inference is only one thing you defend. The local AI capability depends on a chain of signing keys, model artifacts, index files, package repositories, privileged admin actions and update workflows. Any link can be attacked.
• Control 1 — pin and attest every dependency. Nothing crosses into the air-gapped network without it. The release manifest lists each exact version and hash. The installer checks signatures against offline trust roots. Anything outside the known-good set is turned away at deployment.
• Control 2 — avoid telemetry egress assumptions. In a connected product, teams switch on cloud diagnostics, remote metrics and crash reporting without a second thought. Inside the enclave that habit is wrong. Each observability path has to be local-first and approved on purpose. Metrics flow to an internal collector. Support bundles leave only through a controlled operator workflow. A library that silently calls home is a leak, not monitoring.
• Control 3 — treat privileged administration as a high-risk interface. Admin commands live apart from the everyday user flow. Demand local role-based approval for them. Key rotation, index rebuilds, trust-root updates and rollback promotion should each run as a deliberate, explicit operation that leaves an audit record.

🎯 Interview Pointer: Telemetry egress is where candidates slip. Interviewers wait to hear "we'll send logs to our SaaS dashboard", said out of habit. Say plainly that every observability path stays local-first and approved.

Failure Policy: Fail Open, Fail Closed, Degrade, Queue

• A strong FDE (Forward Deployed Engineer) answer sorts each failure into a bucket: fail open, fail closed, degrade, queue, or hand to a human. For example:
  • Authentication and signature checks fail closed.
  • When the downstream classifier is down, document ingestion can queue, but only inside a bounded local buffer.
  • Search on a stale index can degrade, as long as users see a freshness warning.
  • If memory runs short, model serving can drop to a smaller approved model, provided that fallback was verified in advance.
  • Promoting a bundle or updating a trust root needs a human to approve it.
• This split is what separates a system that is simply down from one that is dangerous.

Five-Condition Decision Table

Blast Radius

• Spell out the blast radius per workflow, per tenant, per dependency and per region (enclave/site).
• On an air-gapped government network, a "region" is often a physical enclave or site, not a cloud geography.
• Where tenants or workspaces can be isolated, one bad index rebuild must not knock out document search across the whole enclave.
• A rollout of a model that fails must leave the audit service valid.
• If policy lets reads continue, a certificate fault in the admin plane must not stop read-only document lookup.
• Give every dependency a containment boundary of its own.

Critical Incident Drill: Missing Transitive Package

• This drill is mandatory because it tests whether a candidate grasps what offline really means. The bundle carries a valid signature, yet operations finds that the installer cannot pull one transitive dependency from the local mirror.
• The tempting mistake is to let the install go ahead "just this once". That is exactly what not to do.
• Containment (immediate):
  1. Halt the promotion before it writes any partial state.
  2. Capture the failing dependency graph, plus its exact bundle hash and manifest version.
  3. Save signer metadata, logs and the artifact itself as evidence.
  4. Check that the release now in service has not been touched.
  5. Let the operator know this is not a runtime defect: the mirrored set is incomplete.
• Recovery: use the source of truth to rebuild the offline mirror, prove the dependency closure is whole, and verify hashes and signatures again.
• Prevention: run a dependency-closure preflight before the bundle ever reaches the enclave. A release process that cannot prove completeness must never hand the bundle to the installer.

Other Failure Paths

• Artifact corruption during transfer: check cryptographic hashes and signatures at every hop, not only where the bundle was built. A bundle that verifies at its origin but fails at the enclave boundary is hostile or damaged until shown otherwise. Set the bad artifact aside for investigation; do not overwrite it.
• Model exceeds GPU memory: this is a capacity and packaging problem, and it should never surprise you at runtime. Send the request to a smaller approved model, hold it until a compatible worker frees up, or reject it with a clear operational error. Never let the node crash in the hope that the scheduler sorts it out. Where you can, spot memory pressure before admission. Keep an explicit compatibility map that ties model size, quantisation format and hardware profile together.
• Local certificate expiring: at first this is about availability and administration. It turns into a security problem the moment the system skips checks just to stay up. Issue a short-lived operational certificate, watch its expiry, renew it locally from an offline trust anchor, and keep an escalation path for overdue rotation. Privileged actions should halt, while read-only workflows may keep going in degraded form if policy says so.
• Update breaks stored-index compatibility: give the index format a version that moves with the model and the reader code. The three ship as one unit. They ship only when the new reader can read the old index, or can rebuild it deterministically. If the new release cannot read the current index, pause promotion and let the old path keep serving. A human should step in if nobody is sure what the compatibility matrix says, or if rebuilding would take longer than the recovery objective allows.

Code: Proving One Security Invariant

• Interview code should prove a security invariant rather than walk a toy happy path. The invariant here: a tampered bundle is refused before it is deployed.

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
import hashlib
import hmac
import json
import tempfile
from typing import Mapping


class IntegrityError(RuntimeError):
    pass


@dataclass(frozen=True)
class TrustedKey:
    name: str
    secret_key: bytes


@dataclass(frozen=True)
class Bundle:
    root: Path

    def replace(self, relative_path: str, content: bytes) -> None:
        target = self.root / relative_path
        if not target.is_file():
            raise FileNotFoundError(relative_path)
        target.write_bytes(content)


@dataclass(frozen=True)
class AuditEvent:
    action: str
    detail: str


def _sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def _load_manifest(bundle: Bundle) -> Mapping[str, object]:
    manifest_path = bundle.root / "manifest.json"
    if not manifest_path.is_file():
        raise IntegrityError("missing manifest")
    try:
        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    except json.JSONDecodeError as exc:
        raise IntegrityError("invalid manifest") from exc
    if not isinstance(manifest, dict):
        raise IntegrityError("invalid manifest")
    return manifest


def _audit(log: list[AuditEvent], action: str, detail: str) -> None:
    log.append(AuditEvent(action=action, detail=detail))


def verify_bundle(bundle: Bundle, trusted_key: TrustedKey, audit_log: list[AuditEvent] | None = None) -> None:
    audit_log = audit_log if audit_log is not None else []
    manifest = _load_manifest(bundle)

    if manifest.get("signer") != trusted_key.name:
        _audit(audit_log, "verify_bundle", "untrusted signer")
        raise IntegrityError("untrusted signer")

    signature = manifest.get("signature")
    if not isinstance(signature, str) or not signature:
        _audit(audit_log, "verify_bundle", "missing signature")
        raise IntegrityError("missing signature")

    expected = manifest.get("artifacts")
    if not isinstance(expected, dict):
        _audit(audit_log, "verify_bundle", "invalid artifact list")
        raise IntegrityError("invalid artifact list")

    # Validate the closure of the bundle before any promotion occurs.
    for relative_path, expected_hash in expected.items():
        if not isinstance(relative_path, str) or not isinstance(expected_hash, str):
            _audit(audit_log, "verify_bundle", "invalid artifact entry")
            raise IntegrityError("invalid artifact entry")
        artifact_path = bundle.root / relative_path
        if not artifact_path.is_file():
            _audit(audit_log, "verify_bundle", f"missing artifact: {relative_path}")
            raise IntegrityError(f"missing artifact: {relative_path}")
        actual_hash = _sha256_file(artifact_path)
        if not hmac.compare_digest(actual_hash, expected_hash):
            _audit(audit_log, "verify_bundle", f"hash mismatch: {relative_path}")
            raise IntegrityError(f"hash mismatch: {relative_path}")

    # Offline trust-root check: the manifest is authenticated with a pinned local key.
    material = json.dumps(
        {"signer": manifest["signer"], "artifacts": manifest["artifacts"]},
        sort_keys=True,
        separators=(",", ":"),
    ).encode("utf-8")
    expected_sig = hmac.new(trusted_key.secret_key, material, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(signature, expected_sig):
        _audit(audit_log, "verify_bundle", "signature verification failed")
        raise IntegrityError("signature verification failed")

    _audit(audit_log, "verify_bundle", "bundle accepted")


def _write_bundle(root: Path, signer: str, secret_key: bytes, model_bytes: bytes) -> Bundle:
    root.mkdir(parents=True, exist_ok=True)
    (root / "model.bin").write_bytes(model_bytes)
    artifact_hash = hashlib.sha256(model_bytes).hexdigest()
    manifest_body = {"signer": signer, "artifacts": {"model.bin": artifact_hash}}
    material = json.dumps(manifest_body, sort_keys=True, separators=(",", ":")).encode("utf-8")
    signature = hmac.new(secret_key, material, hashlib.sha256).hexdigest()
    manifest = {**manifest_body, "signature": signature}
    (root / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
    return Bundle(root=root)


def test_tampered_model_is_rejected():
    with tempfile.TemporaryDirectory() as tmpdir:
        bundle_root = Path(tmpdir) / "bundle"
        trusted_key = TrustedKey(name="offline-root-1", secret_key=b"local-trust-root-secret")
        bundle = _write_bundle(
            bundle_root,
            signer=trusted_key.name,
            secret_key=trusted_key.secret_key,
            model_bytes=b"approved-model-bytes",
        )

        bundle.replace("model.bin", b"tampered")
        audit_log: list[AuditEvent] = []

        try:
            verify_bundle(bundle, trusted_key, audit_log=audit_log)
        except IntegrityError:
            assert any(event.action == "verify_bundle" for event in audit_log)
            assert (bundle_root / "model.bin").read_bytes() == b"tampered"
            return
        raise AssertionError("tampered bundle should be rejected")

• The sketch is sized for an interview on purpose. For production, keep this control flow but swap the teaching scaffold for a hardened release verifier. Check the manifest schema explicitly, write structured audit events to local storage, and add tests for the negative cases: signer mismatch, a malformed manifest, a missing file and a broken dependency closure.
• Running cleanly is not what the test is for. It shows that the system will not install a bundle whose integrity it cannot establish.

Evidence, Runbooks, and Operational Readiness

• Runbooks must exist before go-live for node replacement, index rebuild, rollback, certificate renewal, trust-root rotation and verification failure.
• Every runbook names who may run it, which evidence it keeps, which systems it pauses and how success is judged.
• The audit trail captures the operator's identity and approval record, the signed manifest with its hash list, the trust-root fingerprint, installation logs and any rollback decision.
• Later, a customer may ask why a release was blocked. The evidence by itself should be enough to rebuild the answer.
• Skip this discipline and each incident turns into forensic guesswork. Keep it, and operations can show that a release was rejected for a real integrity reason rather than a flaky installer.

Why This Matters for the Job

• A strong FDE goes past the happy-path diagram. They show the design survives stale state, partial failure, dependency outages, hostile input, retries that change nothing, and political pressure to "just make it work."
• They know which failures should queue, which should degrade, which must fail closed and which need a person. They can explain the blast radius, hold on to evidence and back a safety invariant with a test.
• 90-second interview summary: "The offline release pipeline is a high-trust control plane, and I guard it that way. Every dependency is pinned and attested. Signatures are checked against offline trust roots. Telemetry never assumes it can leave the network. Privileged admin is kept apart from normal use. The riskiest failure is a bundle that is incomplete or corrupted. So the system fails closed, keeps the evidence and leaves the running release alone. Runtime problems each get a clear policy: some queue, some degrade to an approved fallback, some need a human. My first production gate: no bundle enters the enclave until it has proved, offline, that it is intact, complete and compatible."

🎯 Interview Pointer: When someone asks for your first production gate, give the same sentence every time, in every section, and learn it by heart: no bundle enters the enclave until it has proved, offline, that it is intact, complete and compatible.

From Prototype to Trusted Production

• A working demo is only the start. The milestone is the customer seriously asking when they can rely on it in production.
• Once that question lands, the interview moves past architecture into delivery. You now need a phased rollout, gates you can measure, named owners and a support model that holds up when the first incident hits.
• Inside an air gap, delivery belongs to the design itself. If the team must improvise to move, verify, install, watch or roll back the software, what they have is an encouraging demo, not something they can deploy.
• The FDE's job is to turn the prototype into an operating path: safe enough for the enclave, dull enough for operators, valuable enough for users.

Four Phases That Move Risk Out of the Dark

1. Stand up a staging environment on identical hardware.
   • Owner: infrastructure or platform engineering, with a security review by the enclave team.
   • Exit criteria: trust roots, OS image, GPU/CPU profile, storage layout and node class all match production, so install and runtime results are representative.
   • Why it matters: a staging setup built for convenience masks the problems that bite later, such as package incompatibility, driver mismatch, disk layout surprises and painful upgrades.
2. Rehearse the signed transfer.
   • Owner: release engineering runs it; security and operations watch the drill together.
   • Exit criteria: the team carries a release bundle along the controlled transfer path, checks its signatures offline and installs it with no call beyond the boundary.
   • Why it matters: real systems often fail at the transfer step. A plan of "someone will hand-carry it" falls short unless the handoff is written down, repeatable and auditable.
3. Run a canary on a single node.
   • Owner: operations runs it, while the product owner and an incident responder stay on call.
   • Exit criteria: a single production node takes the new release, handles real traffic and stays within the agreed health thresholds for a set observation window.
   • Why it matters: the canary contains the risk. No cloud-scale rollback shortcut exists on a closed network. So the first live node must show that configuration, dependencies, model and bundle behave well together.
4. Keep a tested rollback bundle and its runbooks.
   • Owner: operations carries out a rollback, release engineering keeps the artifacts, and the technical lead approves.
   • Exit criteria: the last known-good bundle sits stored, tested and ready for immediate use, along with its escalation path, operator checklist, rollback steps and signatures.
   • Why it matters: treat rollback as a prepared artifact rather than an idea. When a release goes wrong, the team needs a trusted route back, not a scramble to rebuild confidence under pressure.

• Each phase gets an owner, a gate and a stopping point before damage spreads. That separates a staged rollout from deployment by hope.

Technical Health Metrics

• Offline install success rate
  • Calculation: offline installs that succeed divided by offline installs attempted, both counted over the same release window.
  • Data from: installer logs, release logs and node boot records.
  • Owner: release engineering.
  • Alert threshold: a rate under baseline, or failures on more than one staging or production node in one release cycle, halts expansion. Check bundle integrity, dependency drift and environment mismatch.
• Signature verification failures
  • Calculation: how many signature checks on bundles, artifacts or manifests fail in transfer or at install.
  • Data from: transfer logs, plus the verification logs written at install time.
  • Owner: security engineering or release engineering, whichever the local operating model assigns.
  • Alert threshold: treat any unexpected failure as a go/no-go blocker until the root cause is clear.
• Model throughput and latency
  • Calculation: documents or pages handled per unit of time, together with request latency at the chosen percentile on user-facing paths.
  • Data from: local performance counters and service telemetry.
  • Owner: platform or ML (machine learning) operations.
  • Alert threshold: when latency leaves the band users accept, or throughput drops under the demand forecast, look at queue limits, hardware saturation, model quantisation or batching changes.
• Capacity saturation
  • Calculation: how much of the safe operating headroom is in use, across token budget, queue depth, disk I/O, memory, GPU or CPU.
  • Data from: service queues and host metrics.
  • Owner: infrastructure or SRE (site reliability engineering).
  • Alert threshold: if the system runs close to its ceiling for a sustained period, scale up, shed load or throttle workflows before it turns unstable.
• Mean time to repair
  • Calculation: the average time between declaring an incident and restoring service.
  • Data from: incident timelines.
  • Owner: incident management together with the primary on-call team.
  • Alert threshold: if repair times climb from one release to the next, the team probably lacks documentation, automation or spare capacity.
• Release age
  • Calculation: time elapsed since the approved bundle now running was signed off and deployed.
  • Data from: deployment records and the release registry.
  • Owner: release manager.
  • Alert threshold: a release that is too old has drifted away from the latest security fixes and operational lessons. Releases that change too fast get too little soak time.

User and Business Metrics

• These are distinct from technical health. A perfectly healthy system can still go unused.
• Adoption metrics: repeat use, active users, documents submitted, and the share of workflows done with no manual workaround.
• Business outcome metrics: time saved on each case, a lighter manual review load, more analyst throughput, and fewer missed deadlines or rework cycles.
• Outcome owner: normally the business sponsor or the operations manager, backed by product and the FDE.
• Tie the layers together. Faster throughput means little if analysts distrust the output. And if users adopt it but still correct results by hand, the workflow gain is smaller than the demo implied.

A Dashboard That Tells One Story

• Avoid a wall of green lights. The dashboard should trace how value for users depends on each layer of the stack.
• Top row: user outcome.
  • Documents handled in each shift.
  • Mean turnaround time per case.
  • Proportion of cases finished without any manual fallback.
• Middle row: service behaviour.
  • Offline install success rate for each release.
  • Signature verification failures for each transfer event.
  • Model latency and throughput per node.
  • Indicators of queue depth and saturation.
• Bottom row: operational control.
  • Age of the release now running.
  • Whether a rollback bundle is available.
  • Mean time to repair, alongside open incidents.
  • The most recent successful signed transfer.
• Laid out this way, the dashboard settles the question that counts most in a closed network. Is the user's problem getting better because the system is healthy? Or is the system just staying alive while the workflow stays painful?

🎯 Interview Pointer: "How would you dashboard this?" comes up often. Answer with three rows (user outcome, service behaviour, operational control) rather than a flat list of metrics. It shows you reason in causal layers instead of collecting green lights.

Ownership, Gates, and Rollback Triggers

• Product or business owner: sets the case-review workflow and decides what success means.
• Release engineering: packages the artifacts, signs the bundles and keeps the release records.
• Security: signs off on trust-root handling, audit evidence and the transfer process.
• Operations or SRE: monitors health metrics, runs the canary and carries out rollback.
• ML or platform engineering: answers for model performance, runtime efficiency and compatibility.
• Help desk or support lead: sorts user problems, training gaps and documentation defects.
• Go/no-go gates: the first two apply the rule that no bundle enters the enclave until it has proved, offline, that it is intact, complete and compatible.
  • Offline verification of the bundle passes.
  • The bundle is complete and fits the target node class.
  • On hardware close enough to production, the staging run shows the expected behaviour.
  • Latency, throughput and saturation on the canary node stay within thresholds.
  • Someone has actually tested the rollback bundle; archiving it is not enough.
• Rollback triggers: signature failures that repeat, an install failure nobody can explain, a latency regression on the canary, a climbing error rate, or any sign that the release puts enclave integrity at risk. In an air-gapped setting, "we'll debug it live" is no plan. Saying it means the team never had a working way back.

Standardisation Strategy

• This sorts what should be the same for every customer from what should stay local:
  • Configuration: approval thresholds, retention rules, document types, queue limits, which approved offline model to use, and the rules that route workflows.
  • Adapter: importers that read the customer's document sources, connectors that export to case systems, and translation layers that map local formats into the analysis pipeline.
  • Shared service: the release registry, audit logging, bundle validation, signature verification and model serving primitives. These are strong candidates to reuse across enclaves.
  • Core product: the document analysis engine, the health-check framework, the offline package manager and the discipline of rollback. Improve these and every deployment benefits.
• This split shows product thinking, not only implementation. The FDE should not hand-build one-off integrations forever. The goal is to turn repeated local pain into reusable product capability while respecting every customer boundary.

Risk Register

• Risk: the offline bundle lacks a transitive package.
  • Owner: release engineering.
  • Mitigation: dependency lockfiles, a preflight check that validates the bundle, and an install rehearsed in staging.
  • Trigger: an install fails, or an import cannot be resolved at boot.
• Risk: signatures do not match during transfer.
  • Owner: security engineering.
  • Mitigation: validating the trust root offline, checking signed manifests, and transfer steps that need dual control.
  • Trigger: any verification failure nobody expected.
• Risk: on one node, model latency goes beyond what users will tolerate.
  • Owner: platform or ML operations.
  • Mitigation: a smaller variant of the model, adjusted batching, or moving the work to other hardware.
  • Trigger: the canary shows a latency regression or queues keep growing.
• Risk: operators need tribal knowledge to support the workflow.
  • Owner: support lead and technical lead.
  • Mitigation: incident drills, runbooks, training, and explanations attached to alerts.
  • Trigger: the same questions keep coming, escalations stall, or people handle cases inconsistently.

The Delivery Story the Interviewer Wants to Hear

• The prototype runs, and the customer wants to know when it will be production-ready. "After we tune the model some more" is the wrong reply.
• The right reply takes four steps from prototype to staged rollout. Build staging on hardware identical to production, rehearse the signed transfer, run a canary on one node, and keep a tested rollback bundle with runbooks.
• Track release age, mean time to repair, saturation, latency, throughput, signature verification failures and offline install success rate.
• Track business outcome, adoption, model quality and technical health as separate layers. Only then can you tell whether the product really makes the workflow better.
• Name owners and set gates, so a bad release halts before it turns into an incident.
• Measurable customer impact statement: the system counts as a success only when analysts use it, documents are handled faster and more consistently, and the operating team keeps it running as maintainable AI that runs locally and never crosses the enclave's network, artifact, identity or audit boundaries.

Minute Zero: Open With the Customer Outcome

• Lead with what the customer gains, phrased the way they would say it: "We want local AI we can maintain inside an air-gapped government network. Updates have to be controlled, nothing can lean on the internet, and we need an audit trail that stands up to scrutiny."
• It shows you heard the operating constraint, and it keeps you out of a stock chatbot design.
• Before you draw anything, get the interviewer to confirm which boundary carries the most risk:
  • What does the air gap actually protect?
  • Is the isolation total, or do governed transfer points exist for media, packages and logs?
  • Are we deploying into one secure enclave, several disconnected sites, or a staged network split into separate trust zones?
  • Should analysts share one service, or do especially sensitive teams need their own isolated workstations?
• Spend minutes by risk, not by diagram size. Software ingress, provenance, identity and rollback deserve the most. Cosmetic model debates and over-tuned UI features deserve the least.
• Announce the plan out loud: "First I'll cover how software gets into the enclave and how we show it's safe. After that, we can walk the request path and how the team runs it."

A 50-Minute Pacing Plan

• 0–5 minutes: discovery and constraints
  • Give the customer outcome back in a single sentence.
  • Pin down the air-gap boundary, how updates happen and what auditors expect.
  • Ask who the users are, which document types matter and whether success means search, extraction, classification, summarisation, or all of them.
  • Check where it runs: centrally, on analyst workstations, or both.
  • Flag your riskiest assumption and promise to return to it.
• 5–10 minutes: scale and success criteria
  • Estimate daily documents, typical size, peak concurrent analysts and freshness needs.
  • Without given latency targets, state them in words: interactive queries feel quick; batch jobs may run slower.
  • Track user adoption, offline install success, signature verification, release age and incident recovery time.
  • Treat every capacity figure as illustrative until the customer confirms the workload.
• 10–18 minutes: architecture overview
  • Sketch the trust zones: outside sits the build environment; a transfer gate links it to the enclave, which holds the admin plane, audit store, document pipeline, inference service, model registry and package repository.
  • Keep the data plane apart from the control plane.
  • Documents move inward; results and logs reach only approved destinations; nothing relies on cloud telemetry.
  • No path ever breaches the identity, artifact or audit boundary.
• 18–28 minutes: trade-offs and system design decisions
  • Bigger-model quality against hardware fit.
  • Containers or virtual appliances.
  • Update cadence against accreditation cost.
  • A central cluster or workstations.
  • Justify each by customer value and running cost, never taste.
• 28–35 minutes: security, provenance, and failure handling
  • Cover signed artifacts, pinned dependencies and repeatable builds.
  • Show a patch passing a controlled import gate and checks before promotion.
  • Explain offline debugging with reproducible replay, health endpoints, structured event traces and local logs.
  • Run the drill: new model weights, old index. What happens?
• 35–42 minutes: rollout and operations
  • Walk through staging, the canary, rollback and user training.
  • Say who owns each area: the platform, security, support and the analyst community.
  • Show release age and approval gates lowering risk without freezing progress.
  • Explain reuse in similar enclaves.
• 42–50 minutes: recap and follow-up defence
  • Give the ninety-second version of the architecture.
  • Point out the biggest trade-off.
  • State the first production gate.
  • Take questions; tie each answer back to boundaries, provenance or rollback.

Trade-Off: Model Size vs. Hardware Fit

• This is usually the first tension interviewers probe.
• Extraction and summaries may improve with a bigger model. The cost: it may not fit in local GPU memory, it answers slower, patching gets harder and the customer may have to buy or borrow hardware.
• A smaller model fits easily and is simpler to support, yet may struggle with noisy documents, specialised vocabulary or long-context reasoning.
• "Always biggest" and "always smallest" are both wrong. Optimise for the mission:
  • For triage or structured extraction, a reliable, well-tuned small model can beat a brittle large one that is costly to maintain.
  • Subtle legal or technical interpretation can justify bigger inference hardware, but only if enclave operators can sustain and safely patch it.
• Interview line: "I'd pick the smallest model that clears the quality bar on the customer's own documents. Each step up in size competes for enclave hardware, adds latency and adds accreditation work."

Trade-Off: Containers vs. Virtual Appliances

• Containers give portability, repeatable deployment and cleaner separation of concerns. They fit an enclave with an existing platform team and components refreshed one at a time.
• A virtual appliance seals the stack in a fixed boundary with fewer moving parts, which some security teams find easier to review.
• The real tension is operational complexity against governance simplicity:
  • Pick containers if efficient scaling, controlled patching and reusable components rank highest.
  • Pick an appliance if the customer prefers one tight, inspectable unit and slow, deliberate change.
• Let the organisation decide, not purity. An existing hardened virtualisation standard favours the appliance. An existing secure container platform favours containers for upkeep and reuse.

Trade-Off: Update Frequency vs. Accreditation Cost

• Frequent updates cut vulnerability exposure, lift model quality and speed feedback. Yet each one must be packaged, signed, transferred, verified, staged, tested and approved, and may trigger re-accreditation.
• "Ship weekly because we're agile" is wrong. Size the release train to the governance load the customer can carry:
  • Keep urgent security patches on a separate track from new features.
  • Prebuild trusted update bundles.
  • Promote from dev to staging to the enclave.
  • Make each high-friction change smaller, more predictable and easier to audit, so fewer are needed.
• Strong phrasing: "I'd favour fewer releases that we trust more. In an air-gapped setting, change costs you in engineering and in governance."

Trade-Off: Central Cluster vs. Workstation Deployment

• One central cluster is easier to manage and observe, and every analyst reaches the same models. It suits many analysts on one service, where a managed platform is allowed.
• Workstations help on a heavily segmented network, at low volume, or when the capability must survive limited central infrastructure.
• Decide on user count, concurrency, isolation needs and how much central administration the customer can sustain.
  • Workstations reduce reliance on shared infrastructure, at the price of harder patch distribution, more support and weaker consistency.
  • A central service raises reuse and control, but becomes one operational asset to protect and monitor closely.
• If the enclave can host it, start central: governance and updates are easier there. Fall back to workstations for isolated teams or cut-off sub-enclaves.

🎯 Interview Pointer: The four trade-offs follow one pattern. Lay out both options, name the deciding factor in terms of the customer's real constraints, then commit to a default and say when you would leave it. Use the same pattern live for trade-offs this section does not list.

Four Hard Follow-Up Questions

• "How do patches enter the network?"
  • Via controlled import, never casual file copying. Builds happen outside the enclave; each patch is signed, scanned and packaged as an immutable release artifact.
  • It crosses by a customer-approved transfer mechanism into a quarantine or staging repository inside the air gap.
  • Before promotion, the enclave checks the signature and dependency manifests and runs regression tests.
  • If pushed, split emergency security updates from planned releases. Emergencies still need provenance and validation but can use a faster approval path with pre-agreed criteria. Every crossing carries a named owner, a checksum or signature, and a promotion record.
• "With no cloud telemetry, how do you find faults?"
  • On site, by design, with enough instrumentation that nobody guesses. The enclave keeps its own audit events, local metrics, health checks, trace correlation IDs and structured logs.
  • Replayable fixtures from representative documents let you reproduce failures offline.
  • Support flow: once an analyst flags a fault, record the index revision, document hash, model version, request context and involved service logs. You can then reproduce it with no data leaving the network. If policy bars full payloads, keep the smallest safe subset or a redacted reproduction bundle.
  • Principle: disciplined internal observability plus rich release metadata replace external observability.
• "The weights changed but the index didn't. Now what?"
  • It probes whether you see coupled artifacts. Version model and retrieval index together as a compatible pair.
  • New weights that alter embeddings, tokenisation or ranking behaviour may force an index rebuild or revalidation.
  • Index schema, embedding pipeline, prompts, tokeniser and model package form a single compatibility surface.
  • A pure runtime gain behind the same interface may keep the index. A representation change needs a rebuild, or a parallel index staged to compare outputs before traffic moves.
  • You show the unwritten contract between retrieval and generation, and a safe upgrade path rather than just "updating the model."
• "How do you prove build provenance?"
  • As chain of custody: where the artifact came from, which code built it, which dependencies went in and who approved its entry to the enclave.
  • The evidence: artifact hashes, reproducible build steps, pinned dependency manifests, tagged releases or signed commits, plus a verification record at every hand-off.
  • For detail, a build system emits attestations and policy checks verify them before install.
  • Never claim absolute proof. You get strong evidence and a control you can defend, nothing magical. It ends the guesswork when an auditor or operator asks, "What exactly is running here?"

Common Weak Answers and Repairs

• "Let's run our usual cloud stack on-prem." → Repair: name the cloud dependencies that would breach the air-gap boundary, and the enclave-local services that replace them.
• "Pick the largest model we can get." → Repair: tie the choice of model to what the hardware holds, the latency users accept and the cost of upkeep.
• "Security can handle the rest." → Repair: show that you own the artifact flow, identity, logs and rollback.
• "We'll patch monthly." → Repair: explain release trains, the emergency-fix path and the approval cost every update brings.
• "Logs can wait; we'll look when something breaks." → Repair: say what you must capture from day one so offline debugging is possible.
• "Model and index are independent." → Repair: admit that embedding behaviour, retrieval quality and index freshness have to stay compatible.
• To sound senior, drop the generic answer and say: "That works in plenty of environments. In an air-gapped enclave, though, the hard part isn't technical feasibility. It's managing change safely."

Self-Scoring Rubric

• Discovery: Did you surface the customer's true outcome, where the boundaries sit and which unknowns are most dangerous?
• Estimation: Did you convey a sensible scale and capacity, and flag numbers as illustrative where needed?
• Architecture: Were the update flow, the trust boundaries, and the data and control planes each kept separate?
• Depth: Did you dig into the riskiest areas rather than give every box the same airtime?
• Security: Did you cover provenance, identity, transfer controls, logging and rollback, while admitting no control makes risk vanish?
• Delivery: Did you show the path from prototype through staged rollout to production support?
• Communication: Were you crisp and structured, and did you welcome redirection?
• Winning answers show judgement, not box count: what matters, what can wait, what needs control and how the customer gains.

The Final 90-Second Summary

• "I'd run document analysis inside the air gap as an enclave service under central management. Signed releases come in through a controlled import path. The model and index ship together as versioned bundles. Local observability lets us support it offline. On model size, I'd go as small as approved hardware allows while still passing the customer's quality bar on their documents. Accuracy matters, but so do hardware fit, update cost and accreditation friction. If the customer already runs a secure platform, I'd use containers. If governance simplicity comes first, I'd switch to a virtual appliance. My biggest risk is the tension between how often we update and how much review each update needs. A staged release train, signed artifacts and bundles we can roll back keep that in check. The first production gate: no bundle enters the enclave until it has proved, offline, that it is intact, complete and compatible. After that, an in-enclave canary has to show verified provenance, steady latency and a support runbook we've already tested."
• It works because it ties the design to the customer's goal, maintainable AI that runs locally and never crosses the enclave's network, artifact, identity or audit boundaries, and proves you can hold the design under pressure.

Practice Assignments

• Solo exercise: Without looking at this page, write your own 90-second answer to this scenario. Then cut it again under one rule: a sentence stays only if it names a production risk, defends a trade-off or makes a boundary clearer.
• Pair mock: One person plays a sceptical security reviewer whose only questions are follow-ups on patches, provenance and logging. The candidate may say "secure," "robust" or "scalable" only if they define the mechanism behind the word straight away.
• Implementation exercise: Map the life of a single enclave update: commit the code, build, sign, stage, import, validate, promote and roll back. Flag each step that alters the audit log, the index or the model artifact. Then pick the exact step where you would halt the rollout if the weights changed and retrieval quality dropped.
• What to remember under interview pressure: when lost, come back to what the customer needs, where trust ends, how releases travel and how you undo one. Great FDE (Forward Deployed Engineer) answers are structured, carry believable numbers, stay safe without theatre, keep the customer in view and name trade-offs openly.