Secure Multi-Tenant AI Platform

Multi-tenant AI is one platform, many customers. Cheaper than a stack per customer. The failure is Acme seeing Globex’s docs, in a cache, a log, or a prompt.

G07 covers one slice: identity → tenant stamp that never changes → every store and model call respects it. Not “put tenant_id on a row and hope.”

End to end, as Acme user Maya asking her assistant:

1. SSO (single sign-on) proves Maya is Acme. That tenant id is frozen on the request.
2. Quotas and region — maybe Frankfurt only, including logs.
3. Retrieval, cache, and the model see only Acme bytes.
4. If Globex is noisy, fair queues protect Maya; we don’t leak by sharing a hot cache key.
5. Spend is metered before the model call, so Acme cannot burn Globex’s budget.
6. If Acme leaves, we prove keys and copies are gone.

That’s it: verify tenant once → enforce everywhere → meter and isolate. Arbitrary tenant plugins stay out.

The promise is one affordable AI platform for many customers without a cross-tenant read, write, cache hit, or model context leak. A tenant ID on a row is insufficient. Derive tenant context from verified identity once, keep it immutable, and enforce it independently at every storage and inference boundary.

G07 is the source study’s anchor for the shared platform, per-tenant budget, noisy-neighbor, and cross-tenant retrieval cases.

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Onboard tenants and configure policy.
2. Stamp immutable tenant context on every request from verified identity.
3. Apply per-tenant quotas, keys, regions, and retention.
4. Scope data, cache, queues, retrieval, and inference to that tenant.
5. Make administration auditable.
6. Offer a dedicated tier for justified exceptions.
7. Coordinate export and deletion when a tenant leaves.

Non-functional requirements — how well / under what constraints

Unit economics: C_tenant = C_fixed/N + C_usage + C_isolation. Share when safe; pay isolation for residency, risk, or performance.

First release

No arbitrary tenant plugins in the core, no universal custom runtime, no cross-region active-active writes for everyone, no unlimited model choice at launch. Model selection is an approved platform capability, not tenant-supplied execution.

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, derive tenant once and enforce it at every store and model call. Non-functionally, zero cross-tenant leaks, per-tenant p95 and cost, fail closed, and isolation you can prove to an auditor.”

This condenses the source architecture, §4. The platform hosts LLM (large language model) inference, but the model receives only tenant-scoped inputs and never decides tenant membership or authorisation.

Step-by-step architecture

• Step 1. Validate the identity token and map the principal through the tenant directory; ignore tenant IDs supplied in a request body.
• Step 2. Create immutable actor, tenant, role, and region context, then evaluate policy for the specific action and resource.
• Step 3. Route only to an allowed region and shared or dedicated tier; select tenant-specific keys and retention policy.
• Step 4. Meter tokens, concurrency, and budget before inference; a tenant over quota queues or is rejected without stalling neighbors.
• Step 5. Tenant-aware services read and write scoped DB rows, object namespaces, cache keys, queue envelopes, and vector indexes. A missing scope fails closed.
• Step 6. Apply tenant filtering at vector ingestion and query, then independently check request.tenant_id == chunk.doc_tenant_id before building the LLM prompt.
• Step 7. Route the permitted request to fair shared or reserved LLM inference. Validate typed model output before any persistence or downstream action.
• Step 8. Emit tenant-scoped audit and usage events; return the answer only within the verified context.

Agent role: this is a multi-tenant AI platform, not one universal autonomous agent. It can host RAG (retrieval-augmented generation) assistants and bounded agents. Every model prompt, retrieval, tool call, cache hit, output, and trace inherits the same immutable tenant context; agent plans are proposals checked by platform policy.

Default to shared infrastructure with enforced tenant rows for the common case. A separate namespace adds a stronger boundary without a whole deployment; a dedicated cluster/region/key set is for requirements the shared tier cannot safely or economically meet. Least privilege applies to every service identity. The blast radius is measured by tenant, region, workflow, and dependency.

Rate limiting caps a tenant’s own usage; fair queuing decides order under contention. Use token, concurrency, and burst quotas to cap each tenant, and weighted fair queuing across per-tenant queues to share the worker pool; reserve capacity for contractual tiers. Diagnose unusually high usage by API key, user, feature, time, model route, tokens, batch jobs, and agent steps. Support legitimate adoption; stop loops or abuse. Do not rely on autoscaling alone.

Define policy centrally and enforce it regionally when residency requires. Data, traces, logs, backups, and derived indexes follow placement and retention. During a control-plane outage, continue only operations whose local verified policy and keys remain sufficient; block sensitive changes, queue where safe, and reconcile on recovery.

Export and deletion need a tenant inventory across rows, blobs, embeddings, caches, queues, logs, and backups, plus verified completion and retention exceptions. A database delete alone is not offboarding.

Release gate: an adversarial isolation suite for cross-tenant object reads, stale cache, replayed tokens, malformed filters, namespace collisions, missing predicates, and vector retrieval. Any unexplained boundary failure stops promotion; confirmed cross-tenant incidents target zero. Track per-tenant p95, quota rejection, cost, and regional failover against agreed SLOs.

Roll out internal tenants → negative isolation suite → small shared tenants with conservative quotas → dedicated tier by policy and economics. Canary, rollback, operator runbooks, support training, and audited admin actions are part of delivery.

“I would default to a shared platform for 500 tenants, but derive tenant context from verified identity once and make it immutable. The control plane sets membership, policy, region, keys, and quotas; the data plane enforces scope independently in rows, objects, caches, queues, logs, indexes, and keys. I would meter before LLM inference and use fair queues so a large tenant cannot starve the rest. A missing tenant predicate or cross-tenant retrieved chunk fails closed. I would prove that with an adversarial suite before external rollout, then offer dedicated capacity only when the customer’s risk, residency, or SLO requires it.”

Use the Deep Dive for the leak drill and isolation mechanics and the Cheat Sheet for last-minute recall.

The Main guide is the spoken answer. This is technical backup for the unchanged source study, especially §§3–14.

Validate token signature, issuer, audience, and expiry with the customer IdP before authorisation. The tenant directory maps the verified principal to membership, allowed region, and deployment posture; customer-owned groups may be synced, so renamed groups and mapping drift need alerts. Just-in-time provisioning grants the minimum local role for that tenant. A request-body tenant_id is untrusted, and so is any header the client sets rather than a trusted gateway. Establish context once, make it immutable, and propagate it to services, database sessions, cache helpers, queue messages, vector filters, inference, and audit.

Tenant(id, region, tier, key_ref, retention_policy) owns placement and data policy. Membership(user_id, tenant_id, role, status) owns authority. UsageLedger(tenant_id, period, tokens, requests, cost) owns metering. AuditEvent(tenant_id, actor, action, resource, decision) makes the control provable. Writes use idempotency keys and optimistic concurrency. Model output is parsed into a typed schema and policy-checked before persistence.

The row, object, cache, queue, log, vector, and key layers enforce independently. Bind database row security to verified session context, with the app connecting as a role that is not the table owner, a superuser or BYPASSRLS (or use FORCE ROW LEVEL SECURITY); a query with no tenant predicate then returns no data, rather than all data. Partitioning helps only behind per-tenant credentials. Objects use tenant namespaces. Cache keys include tenant, permission signature, and source/policy version. Queue and dead-letter envelopes carry tenant context. Logs omit raw prompts and secrets. Vector chunks receive tenant metadata at ingest, queries compile a tenant filter, and a final check compares request and chunk tenant before prompt assembly. Tenant-scoped data keys mean raw storage access yields only ciphertext, one leaked key opens one tenant rather than all, and destroying a key crypto-shreds that tenant.

Three tenancy tiers trade cost for separation: shared rows with database enforcement (default), separate schema/namespace, and dedicated cluster/region/keys. Escalate on residency, strict latency, unacceptable contention, or a policy demanding harder separation, using documented thresholds rather than customer size alone. G01’s pre-filter plus authoritative post-check applies to tenant retrieval, but tenant isolation must extend to every non-retrieval path too.

The source’s exercise is 500 tenants, 50k active users, 200 QPS (queries per second) peak, 20× skew. Average QPS per tenant is 0.4, which cannot size peak concurrency, retries, failover, and long-tail inference. At 10× the planning case is 5,000 tenants and 2,000 QPS; expect cache churn, noisy neighbors, and more regional partitions before merely adding one bigger inference pool.

C_tenant = C_fixed/N + C_usage + C_isolation. Attribute tokens, storage, retrieval, egress, and reserved capacity per tenant. Enforce token, concurrency, burst, and monetary budgets before inference; a post-call bill is not a budget gate. Rate limit admission with a token bucket per tenant, and use weighted fair queuing across per-tenant queues to share the workers proportionally. Premium reserved lanes protect contractual performance without forcing every tenant into a dedicated cluster. Quotas too tight create artificial failures; quotas too loose shift one tenant’s burst onto its neighbors.

For one high-usage tenant, segment by key, user, feature, time, prompt version, tokens/request, batch jobs, and agent steps. Determine whether this is healthy adoption, a loop, abuse, or missing quota. Support the first with capacity planning; throttle or pause the others, then add contract-aligned budgets and alerts.

Define policy centrally and enforce regionally when feasible. A tenant’s residency rule includes data, prompts, outputs, traces, logs, backups, and derived indexes. Retention attaches to the tenant record or a versioned policy, not scattered service flags. A regional controller outage may leave safe reads serving from valid local policy and preloaded keys; sensitive changes fail closed, and queued work is reconciled after recovery. Decide RPO/RTO before replication and failover routing.

Export and delete require an inventory of every tenant-owned surface: relational rows, objects, embeddings, caches, queues, logs, and derived artifacts. A verified deletion job records completion and retention exceptions. Backup erasure follows restore and retention semantics; do not claim an immediate surgical delete where the backup design cannot provide one.

The source §14 records an Acme answer citing a Globex renewal addendum. The request was tenant_id=acme; retrieved chunks included doc_tenant_id=globex. Filter version v3 applied only visibility in ['public','internal'], omitting tenant_id == 'acme'. The ACL (access control list) enforcer then skipped its check under pre_authorized_connector_flag=true. The response evaluator detected a critical violation after response_served=true. A rollout had reached 15% of Acme traffic, so the live filter version and canary cohort matter.

Trace request → actual filter → chunk tenants → ACL decision → prompt/citation. Search all traces on v3 for tenant mismatch; compare with v2 controls rather than relying on a UI reproduction. Contain by disabling v3, blocking any mismatched chunk before context, purging suspect answer caches, holding affected contract answers for human review, and notifying the customer. Root cause is two supposedly independent layers trusting a connector shortcut. Prevent with a non-bypassable server-side tenant predicate, hard post-retrieval invariant, removal of connector-level ACL bypass, negative release tests, and cross_tenant_retrieval_count by tenant, connector, and filter version. This is a confidentiality incident, not a hallucination.

The negative suite tries wrong-tenant IDs, stale-cache hits, replayed tokens, malformed filters, namespace collisions, missing predicates, and cross-tenant vectors. One unexplained boundary failure halts promotion. Retain access logs with identity-derived context, role reviews, namespace checks, key-rotation procedures, and incident runbooks. Track per-tenant p95, quota rejects, cost band, isolation-test pass rate, confirmed incident count, and regional failover time separately.

Internal tenants establish tracing and scope → adversarial suite proves negative cases → a conservative shared-tenant canary tests real load → dedicated tier opens only under policy and economics. Roll back bad config by tenant and region without contaminating caches or data. Platform engineering owns runtime and predicates, security owns isolation gates, SRE (site reliability engineering) owns failover, product owns tiering, and support owns first-response triage.

Use the Main guide for the spoken design, the Deep Dive for mechanics, and the unchanged source for the full case.

Derive tenant context from verified identity once; enforce it independently everywhere, and meter before LLM (large language model) inference.

Data classes? Residency for data/logs/backups? Shared versus dedicated exceptions? Peak skew and per-tenant SLO (service level objective)? Federated identity and keys? Contractual budgets? Verified export/deletion? Audit evidence?

Verified IdP → tenant directory → immutable context → policy + allowed region → pre-call quota → tenant-scoped stores/retrieval → chunk-tenant post-check → LLM inference → typed output check → audit/usage.

Agent role: this platform hosts assistants or bounded agents; their plans, prompts, tools, caches, and traces inherit tenant context. Model output cannot establish identity or authority.

Row · object · cache · queue · log · vector index · key. The store refuses an unscoped read (RLS (row-level security), with an app role that is not owner, superuser or BYPASSRLS). Vector scoping happens at ingest, query, and final context assembly. Cache keys include tenant + permission signature + version.

Illustrative sizing: 500 tenants, 50k active users, 200 QPS (queries per second) peak, 20× skew; mean 0.4 QPS/tenant hides concentration. C_tenant = C_fixed/N + C_usage + C_isolation. Shared enforced rows by default → separate namespace → dedicated region/cluster/keys when risk, residency, SLO, or contention justifies it.

Acme request cited Globex document: v3 filter dropped tenant equality and connector flag skipped ACL (access control list). Roll back v3, purge caches, search mismatch traces, notify customer; add non-bypassable predicate, post-check, and negative release gate. It is an isolation failure, not a model hallucination.

Internal tenants → adversarial isolation suite → small shared tenants → policy-justified dedicated tier. Any unexplained boundary failure blocks promotion; confirmed cross-tenant incident target is zero. Track p95 and cost per tenant.

“The control plane decides tenant policy, region, tier, and budgets; the data plane enforces scope at every store and inference boundary.”

One application serving 500 tenants: shared enough to be affordable, isolated enough that a missing predicate is impossible rather than unlikely.

The dangerous failure is not a bad answer. It is a query, cache lookup, queue message or export that runs without a tenant predicate and reads whatever matches. This page consolidates the group's prompts into one read for the day before. It holds the chapter 2 worksheet and answer key, the tutorial's high-yield sections, the two cost scenarios and the cross-tenant leak incident. The two-layer enforcement mechanics are owned by G01 section 6 and referenced here rather than repeated.

Open with the outcome, not the topology. The feature asked for is one AI application for 500 enterprise tenants. The business result is a cost-efficient shared platform with defensible isolation, plus a dedicated path for the exceptions. Opening with Kubernetes, a vector database and a model gateway signals feature-first thinking, and that is the tell an FDE (Forward Deployed Engineer) interviewer listens for.

"Support 500 enterprise tenants preserving isolation, predictable performance and regional controls. I'd clarify shared versus dedicated tenants, cross-region data limits, and success criteria per stakeholder. Default: a shared control plane with tenant-aware data and inference paths, strong authorisation at every boundary, and an escape hatch for premium tenants that need dedicated infrastructure."

The headline ask hides a disagreement between four stakeholders about what isolation and success mean. Map the people before the boxes.

Only about six questions fit a forty-five-minute interview, so ask the ones that change the architecture. Each should produce scope, assumptions, risks and owners.

The three answer tiers show what the opening buys. The weak answer uses microservices and Kubernetes, puts a tenant_id column on every row and adds row-level security. It is solution-first, names no customer outcome, and treats one row label as the whole enforcement stack. The average answer builds a shared platform with per-tenant authorisation, quotas, regional routing, audit logging and per-tenant latency monitoring. It never says where tenant context originates, never makes every isolation layer agree, and never defines when a customer graduates to dedicated infrastructure. The strong answer defaults to shared because pooled economics are the point across 500 tenants. It derives tenant context once from verified identity, never from the request body, and makes it immutable. The control plane decides policy, region, quota and key; the data plane enforces at the gateway, the services and every store. Seven layers enforce independently, a missing predicate fails closed, and an adversarial isolation suite gates every external tenant.

State assumptions if the interviewer withholds answers, and choose the ones that protect the most dangerous failure. Assume tenants bring their own identity provider. Assume regional restrictions apply to stored data and inference logs. Assume premium customers may pay for dedicated capacity. Assume model selection is already approved. Have the line for interviewer silence ready:

"If we do not know tenant variability yet, I will assume the shared tier must support most customers, but I will design an explicit dedicated path for customers with stricter residency or performance needs."

Split the conversation into four buckets, functional requirements, non-functional requirements, explicit exclusions and hard constraints, because the trap here is premature certainty. The interviewer only answers half the questions.

The functional requirements are ordered by value, and the order matters: the control plane is foundational and the dedicated tier is an escape hatch, not a default.

1. Tenant-aware control plane. Onboard tenants, assign policies, provision configuration, manage keys, record admin actions.
2. Isolated data-plane request path. Every request carries tenant identity, policy context and authorisation through the whole call chain.
3. Per-tenant policy, quotas, keys and configuration without bespoke code paths per customer.
4. Regional routing and retention controls. Direct requests and artifacts to the correct region; retain or delete per tenant policy.
5. Auditable administration. Answer who changed what, when, under which tenant context.
6. Dedicated deployment tier where justified. An explicit escape hatch for exceptional customers.

The first five are must-haves, because each protects a highest-risk failure mode. Richer quota tuning and broader automation are should-haves. Advanced tenant analytics and convenience workflows are could-haves, and a could-have never expands the core design or delays launch. If a tenant's must-have cannot be satisfied safely in the shared tier, that is the justification for the dedicated path.

Exclude these from the MVP unless the interviewer pushes, because each expands the trust surface or the support burden:

• arbitrary tenant-managed plugin execution inside the core platform
• cross-region active-active writes for every tenant
• fully custom per-tenant runtime stacks
• unlimited per-request model swapping
• manual exception handling for every onboarding request
• ad hoc shared-secret administration outside the control plane

The safety goals are stated as measurable constraints, and so are the operating constraints.

Every requirement then needs the component that enforces it. If a requirement has no component it is probably not real yet, and if a component has no requirement it is probably scope creep. Say that out loud when presenting the table.

A design that is plausible at average traffic fails exactly when customers feel pain. Size for average, peak, growth and skew, with round numbers whose only job is to force decisions: 500 tenants, 50,000 active users, 200 QPS peak, 20x workload skew.

At 200 QPS across 500 tenants the average tenant looks trivial at 0.4 QPS. With 20x skew a few large tenants drive a material share of peak. So no uniform-tenant assumption is safe for queue sizing, cache partitioning, rate limiting, retry policy or noisy-neighbour protection. The most consequential estimate is usually the shape of peak concurrency after retries, fan-out and long-tail latency, not raw QPS. Targeting exactly the naive peak is already broken. Reserve headroom for tenant concentration, retried requests, failover overhead, deploy-time capacity loss and growth before the next tuning cycle.

Pair pooled capacity with four per-tenant controls. A token quota caps monthly or hourly usage to protect spend and fairness. A concurrency quota stops one tenant saturating worker pools or model backends. A burst quota allows short spikes without immediate throttling. Reserved capacity protects contractual performance for premium tenants. Name the trade-off: quotas too tight produce artificial customer-visible failures, quotas too loose let one tenant consume the shared budget.

Anchor SLOs (service level objectives) in the customer workflow rather than in what the platform team prefers to measure:

• availability as successful requests or job completions over a window
• latency as end-to-end response time or completion time
• freshness as time from source update to model-visible update
• quality as task success rate and grounded-answer rate
• security indicators as authorisation failures, policy denials and blocked cross-tenant attempts
• cost per request, per tenant and per successful task

Tie one to a workflow aloud. If this powers support agents, the latency budget must preserve the agent's interaction loop, or a technically successful response arrives functionally useless.

The unit economics settle the shared-versus-dedicated argument.

C_tenant = C_fixed / N + C_usage + C_isolation

The first term is shared platform cost spread across N tenants. The second is the variable cost a tenant directly drives in tokens, storage, retrieval, egress and compute. The third is the premium for stronger separation: dedicated pools, stricter network boundaries, regional duplication, customer-specific encryption, extra compliance controls. Pooled economics win when most tenants can share safely; pay the isolation premium only where the requirement demands it.

What changes at 10x is not the table but the dominant risk. Noisy neighbours, cache churn and operational complexity shift the architecture from one shared tier with guardrails to tiered tenancy with explicit partitioning. Define the per-region recovery objective before the happy path too, since RPO (recovery point objective) and RTO (recovery time objective) decide replication, failover routing and queue durability. And define the missing-tenant-predicate drill up front: can a request, query, export, cache lookup or admin action cross a tenant boundary if a filter is omitted? A design without that test is incomplete.

"I will size for average, peak, and growth; reserve headroom for skew and failover; give each tenant token and concurrency quotas; and tie latency and availability targets to the customer's workflow. I prefer pooled economics for the common case, but I will pay for stronger isolation when the risk, performance, or regional requirement justifies it."

One diagram carries the design. The organising split is control plane against data plane. The control plane decides who may do what, where and under which limits. Its changes are slower, more privileged and more auditable. The data plane does the work. A tenant moving to the dedicated tier is a controlled configuration change in the control plane, not an ad hoc code path.

 ╔═══════════════════ CONTROL PLANE — decides who may do what, where, under which limits ═══════════════════╗
 ║  identity federation (customer IdP) ─> tenant directory ─> policy decision point ─┬─> regional scheduler   ║
 ║  quota configuration · per-tenant key manager · tenant registry + provisioning    └─> key selection       ║
 ║  central policy definition · regional enforcement · SCIM group sync · retention policy per tenant         ║
 ╚═══════════════════════════════════════╤═══════════════════════════════════════════════════════════════════╝
                                         │ immutable tenant context (never re-derived below this line)
 ╔═══════════════════ DATA PLANE — enforces, per region, shared by default ═══════════════════════════════════╗
 ║                                                                                                            ║
 ║  REQUEST PATH — synchronous, strict dependency order                                                       ║
 ║   user ─> API GATEWAY ─> AUTHORIZE ─> ROUTE ─> ADMIT ─> TENANT-AWARE SERVICE ─> STORES ─> AUDIT ─> answer  ║
 ║           admit,       PDP: role,   region +  quota:    executes only on      row · object ·   tenant-   ║
 ║           shape,       tenant,      tier      concur-   immutable context     cache · queue ·  scoped    ║
 ║           reject early action,                rency,                          vector · key     event     ║
 ║                        region                 tokens,        │ missing tenant predicate                   ║
 ║                                               budget         v                                            ║
 ║                                                           FAIL CLOSED (reject / 404 / no result)          ║
 ║                                                                                                            ║
 ║  STORES — tenant key in the physical layout AND in every predicate                                         ║
 ║   partitioned DB (tenant_id) · object store (namespace per tenant) · cache (tenant in key)                 ║
 ║   queues (tenant envelope, per-tenant dead-letter) · vector index (scoped at ingest AND query)             ║
 ║   per-tenant data-encryption keys (envelope) · inference workers (fair queue, reserved lanes)              ║
 ║                                                                                                            ║
 ║  REGIONS — shared control plane, regional data planes; residency decided by the scheduler, not the app     ║
 ║                                                                                                            ║
 ║  OBSERVABILITY — per tenant                                                                                ║
 ║   audit pipeline (append-only) ─> usage ledger (cost attribution) ─> dashboards: p95 by tenant, quota     ║
 ║   rejections, cost/tenant, cross_tenant_retrieval_count, isolation-test pass rate, failover time           ║
 ╚════════════════════════════════════════════════════════════════════════════════════════════════════════════╝

The same flow as a rendered diagram:

The answer key's own diagram is the compressed form, and it is the one to draw first if time is short:

Read the components in dependency order, because that is the order they must exist and the order they fail.

Three boundaries are worth pointing at while the diagram is up. The sync/async boundary sits after audit: authentication, authorisation, quota admission and region selection are synchronous, and reporting, logging and offline enrichment may be asynchronous. Anything that could cross a trust boundary or consume scarce capacity is never deferred blindly to a background task. The systems of record are the tenant directory for membership, the partitioned database for tenant data, the quota service for usage and the audit pipeline for traceability. A boundary mistake among them is a security incident. Caches improve latency and are never the source of truth for membership or authorisation. Queues carry tenant identifiers and policy context explicitly, because backpressure is only safe when the worker knows which tenant it is slowing down.

The MVP (minimum viable product) is one policy service, one gateway, one tenant-aware application tier, one partitioned data store, one quota service, one regional control plane and one audit stream. That is enough to prove isolation, performance shaping and regional routing. Dedicated customer clusters, per-service SLOs (service level objectives), richer workload classification and a more granular policy language are extensions, not prerequisites.

Tenant context comes from verified identity, never from the request body. A signed token, verified claim or server-side session can be authenticated and logged. A header or JSON field the client sets can be spoofed, replayed or forwarded across tenants by accident, unless a trusted gateway sets it. That one choice governs everything downstream, because row filters, object ownership, cache keys, queue partitions, log redaction and vector-index lookup must all consume the same verified context.

The dependency order is strict, and each step has a reason. Identity must come first. Context must become immutable early. Policy must run before expensive work. Region is a control-plane decision, not a late optimisation. Audit happens after the action but close to it.

1. Identity federation authenticates the human or workload through the customer's IdP.
2. Tenant directory maps the principal to tenant membership and allowed deployment posture.
3. Policy decision point evaluates principal, action, resource and region together.
4. API gateway admits, shapes and routes into the correct service tier.
5. Tenant-aware services execute only after receiving immutable context.
6. Partitioned databases and indexes carry the tenant key in physical layout and in every predicate.
7. Quota service tracks concurrency, token and budget limits.
8. Per-tenant key manager selects the encryption context for tenant and tier.
9. Regional scheduler decides which region and cluster class may serve the request.
10. Audit pipeline records tenant-scoped activity.

Narrate the happy path and tie each step to a customer reason. Authenticate through federation, because the customer keeps their own identity source. Derive immutable tenant context, because one company may have several subsidiaries. Authorise against policy, because authorisation must be explainable and centrally governed. Route to the allowed region and tier, because location control is part of the product promise. Apply quota before inference, because shared cannot mean noisy neighbour. Read and write only tenant-scoped resources. Emit a tenant-scoped audit event, because enterprise buyers need traceability.

If the missing-predicate case is not obvious from the diagram, the design is still too hand-wavy. That is the diagnostic test for the diagram itself. The strongest signal in the round is not the drawing but the narration of how identity, data, state and failure move through it.

Production identity has three problems in front of the policy engine, and none of them is authorisation. Token validation checks the signature against the IdP's published keys, the issuer, the expiry, and that the token was issued for this application, before any permission logic runs. Group mapping is owned by the customer. Their Okta or Microsoft Entra ID (formerly Azure AD) groups are the source of truth, mapped into the platform's role vocabulary. That mapping is configuration that drifts silently when a group is renamed. Just-in-time provisioning creates the first local record from the token's claims, scoped to the customer and never with more access than intended by default.

One row label is one defence layer. It is easy to omit in application code, ad hoc queries, background jobs and admin tools. It covers nothing in backups, object storage, search indexes, caches, analytics pipelines or service-to-service auth. So seven layers enforce independently, and one missed check is not a full breach.

The tenant predicate is the mechanism this group owns. The rule is that the store itself refuses to return a row unless the request is already scoped to one tenant. Row-level security bound to the session does that, as long as the app connects as a role that is not the table owner, a superuser or BYPASSRLS, or the table uses FORCE ROW LEVEL SECURITY. Partitioning helps only if each tenant's partition sits behind its own credentials. Isolation then never depends on every engineer remembering WHERE tenant_id = …. A future API that forgets the clause returns nothing rather than another company's data. The same predicate is compiled into the vector-search filter and into the cache key, keyed on tenant, permission signature and version. The two-layer pattern that makes a pre-filter cheap and a post-check authoritative is G01's section 6, and it applies here unchanged. The tenant predicate is layer 1 in the search. The post-retrieval invariant request.tenant_id == chunk.doc_tenant_id is the layer 2 check, and section 14's incident shows it being skipped.

Pick a tenancy level and justify it, because the interviewer will ask how strongly tenants are isolated.

Start at level 1 and make the default honest with two follow-ups. The database enforces the tenant rather than the application. And there is a named escalation path for regulated customers, from a tagged row to their own keys, region or deployment. Per-tenant keys answer the question "what happens if someone gets raw read access to the storage layer?" With per-tenant data keys wrapped by a master key, storage access alone gives them only ciphertext. A leaked tenant key opens one tenant, not the fleet, and destroying that key crypto-shreds the tenant's data. With one platform-wide key, a single leaked key exposes every customer at once. The master key still guards everyone, so it lives in the KMS (key management service) and never leaves it.

Two more principles belong on the board. Least privilege: each service sees only the tenant scope it needs, only for the operation it performs. Blast radius as the design unit: by tenant, by region, by workflow, by dependency. And negative isolation tests matter more than positive ones, because the platform should actively prove the wrong tenant cannot read, infer, cache, dequeue or search another tenant's data.

"I'd start with shared infrastructure and a tenant id on every record, that's the right default. The filter has to live in the data layer, not in application code: the store should refuse to return a row unless the request is tenant-scoped. For regulated customers I'd escalate to a separate schema or a dedicated region and keys, not just a different tag in the same table."

Apply quota and budget before inference, storage growth or batch fan-out, never after. A budget checked after the call is a bill, not a control, and budgets here are contractual. The quota service enforces concurrency, token and burst limits per tenant or tier. The usage ledger attributes every request to a tenant. Quota alerts fire before the limit rather than at it.

Rate limiting and fairness are different problems, and a real platform needs both. A rate limit answers whether this request may proceed at all, a ceiling on one tenant's own allotment. A fair queue answers in what order requests are served under load. A large tenant's burst then cannot make a small, well-behaved tenant wait longer for its turn, even though the small tenant never exceeded any limit. The mechanism is weighted fair queuing across per-tenant queues feeding a shared worker pool, so throughput is shared proportionally under contention rather than first-come-first-served; a token bucket per tenant caps each tenant's rate on top.

Contain a noisy neighbour in two steps. First the per-tenant quotas, rate limits, concurrency caps and workload classification. Then isolate the expensive parts: admission control before inference, queue partitioning, reserved capacity for premium tenants, circuit breakers past the envelope. Noise caused by a bug gets fast detection and a kill switch. Legitimate burst demand gets fairness, not punishment.

Move a customer to dedicated infrastructure when shared tenancy no longer satisfies risk, performance or operational requirements at acceptable cost. The triggers are regulatory residency constraints, unusually strict latency SLOs, high-value workloads with low tolerance for contention, and a customer policy demanding a harder boundary. Avoid "big customers get dedicated"; define measurable thresholds and an exception review path.

Regional controls reshape routing, storage, backup and logging, so settle early whether they are hard requirements or customer exceptions. The balanced answer is central policy definition with regional enforcement, unless residency or availability forces local control. A central control plane keeps policy consistent but can become a latency and residency problem; a fully regional one improves locality but multiplies duplication, rollout complexity and policy-drift risk.

The regional scheduler places work in an allowed region and tier as a control-plane decision. The same rule covers artifacts. Logs, backups and inference traces follow the tenant's residency policy, not the platform's convenience. Retention is part of the data model, attached to the tenant record or a versioned policy it references, so no service rediscovers policy from brittle configuration.

A control-plane outage is not always a full outage, but it is always serious. Policy distribution, deployment coordination and config updates may be impaired while requests still flow. Monitor control-plane health separately from the data plane and alert when the local authority source stops refreshing. If local auth, cached policy and preloaded keys are sufficient, keep serving verified-safe operations only. Otherwise fail closed for sensitive changes, queue non-critical writes, or reroute. Never silently continue with stale control state for irreversible actions. Recover by restoring the regional controller, reconciling queued changes and confirming sync with the authoritative policy source. Prevent by designing for stale-control tolerance, caching minimum policy state locally, rehearsing regional failover, and keeping distinct emergency access paths.

Export and delete across regions deserve their own answer. A tenant inventory and a deletion workflow span every store: relational rows, blobs, logs, embeddings, caches, queues and derived artifacts. Export assembles a complete package with clear ownership and time bounds. Deletion is a coordinated job with verification, retention exceptions and an auditable completion record. Backup deletion semantics follow retention windows and restore controls, not ad hoc surgical erasure.

Memorise the pattern, not the table. Security defects fail closed. Capacity pressure degrades. Metadata leaks quarantine and escalate to a human. Control-plane loss depends on whether local enforcement still holds.

Each of the first five has a rehearsed drill in the tutorial, and they share a shape: detect, refuse and log, contain, recover, prevent. For the missing predicate, detection is code review plus automated tests plus runtime alerts on any query reaching a repository without tenant scope; it never depends on a customer ticket. Refusal is a security error, never a silently substituted default tenant, with identity, trace, operation and the missing-scope fact logged. Containment blocks the release in pre-production. In production it disables the feature path, rotates credentials and freezes logs and traces. In-flight retries keep failing closed rather than retrying into a different scope. Recovery re-runs affected requests through the corrected path and confirms idempotency keys and replay logs cannot bypass the fixed check. Prevention is a negative test that fails if any unscoped repository method is reachable, plus static query-shape lint, so the unscoped path is hard to call rather than merely undesirable.

The cache drill is the one candidates underrate. A cache bug looks harmless in isolation, because the database stays protected, but a shared cache replays the wrong tenant's object if the key misses tenant scope. Detect by instrumenting key components in metrics and traces. Contain by disabling the cache path or namespace, invalidating suspect entries and falling back to the source of truth. Recover by rebuilding from correctly scoped reads and re-checking responses served in the window. Prevent by requiring tenant id in every cache-key helper, centralising key construction, and a negative test asserting a cross-tenant lookup is a miss. If the cache holds auth-sensitive material, treat it as a security incident, not a performance bug.

Production failure behaviour is the difference between a toy and a system. Timeouts tight enough to avoid pile-ups of stale work. Retries only for transient failures, never for authorisation failures or malformed requests. Idempotency on writes, billing events and workflow triggers. Circuit breakers on dependencies returning repeated failures. Dead-letter queues that hold poison messages for inspection rather than endless reprocessing. Human escalation for policy corruption, suspected isolation failure or ambiguous tenant ownership. The honest system says what it can recover automatically and what needs a human with authority.

Say what breaks first at 10x. Noisy-neighbour risk and cache churn become dominant, so pooled-plus-quotas becomes pooled-plus-partitioning, with more dedicated pools and more explicit regional controls.

The central promise is proven by a suite that tries to break the boundary, and a single unexplained boundary failure halts promotion. The goal is not to prove perfection but to prove the system fails safely and visibly under attack.

Keep technical health, model quality, adoption and business outcome as separate metric layers, because blurring them loses the ability to tell an engineering problem from product fit from customer behaviour. Trace a user outcome back to component telemetry in one chain. "Document processing feels slow" becomes request arrival, quota admission, queue depth, inference duration, retrieval latency, cache hit ratio and regional routing.

The interview-sized isolation test proves one invariant at the API level, not only in the database. A secret created under one tenant is invisible to another. The other tenant does not even learn that the object exists.

import pytest

class TenantSession:
    def __init__(self, tenant_id, store, next_id):
        self._tenant_id = tenant_id
        self._store = store
        self._next_id = next_id

    async def create_secret(self, value):
        secret_id = str(self._next_id())
        self._store[(self._tenant_id, secret_id)] = value
        return type("Secret", (), {"id": secret_id})()

    async def get_secret(self, secret_id):
        if (self._tenant_id, secret_id) not in self._store:
            return type("Response", (), {"status_code": 404})()
        return type("Response", (), {"status_code": 200, "value": self._store[(self._tenant_id, secret_id)]})()

class FakeApi:
    def __init__(self):
        self._store = {}
        self._next_id_value = 1

    def _next_id(self):
        current = self._next_id_value
        self._next_id_value += 1
        return current

    def as_tenant(self, tenant_id):
        if not tenant_id:
            raise ValueError("tenant context required")
        return TenantSession(tenant_id, self._store, self._next_id)

@pytest.mark.asyncio
async def test_cross_tenant_ids_are_invisible():
    api = FakeApi()
    created = await api.as_tenant("alpha").create_secret("A")
    response = await api.as_tenant("beta").get_secret(created.id)
    assert response.status_code == 404

It deliberately omits authentication middleware, request signing, database transactions, observability hooks and retry wrappers, and says so. Before launch, hold the evidence that the controls exist and are exercised:

• access logs showing tenant-derived context
• the negative isolation tests
• role and permission reviews
• queue and cache namespace checks
• key-rotation procedures
• incident runbooks for suspected boundary failures

The operational question is not "do we have security?" It is "can we prove the system fails in the right direction, and recover without widening the blast radius?" Every external dependency and irreversible action needs an explicit failure and recovery policy. The absence of a policy is itself a policy, and it usually fails open.

The data model carries four ownership boundaries rather than four tables. Tenant(id, region, tier, key_ref, retention_policy) is data ownership. Membership(user_id, tenant_id, role, status) is authority. UsageLedger(tenant_id, period, tokens, requests, cost) is commercial control. AuditEvent(tenant_id, actor, action, resource, decision) is defensibility. Every write boundary needs an idempotency key, which stops duplicate writes from retries, and optimistic concurrency, which stops lost updates from concurrent writers. Model output never flows raw into the system; it is parsed into a typed schema and policy-checked before persistence.

The platform is done only when users adopt it, the workflow improves and the operating team can support it. A staged rollout with measurable gates, named owners and rollback paths is the answer to "when can this be trusted in production?"

In weeks: 0-1 name stakeholders, the dangerous failure and explicit exclusions. 1-2 internal tenants. 2-3 the isolation suite. 3-4 hold promotion until it passes. 5 low-risk shared tenants under conservative quotas. 6-8 validate boundaries, latency and operator understanding. After the pilot, the dedicated tier on measurable thresholds rather than for whoever asks loudest.

Rollout is more than code promotion. A canary takes a tiny slice of low-risk traffic first. Rollback returns to the prior configuration with no data loss and no cross-tenant contamination. Migration moves tenant metadata, quotas and routing with validation at each step. Training makes support and customer-facing teams fluent in the guarantees and non-guarantees. Documentation covers setup, limits, escalation paths and what a quota error means. Ownership is named: platform engineering owns runtime, security owns boundary validation, support owns first-response triage, product owns tiering policy, SRE owns service health and failover rehearsals.

Keep four kinds of thing apart, because anything re-implemented per customer is the symptom of a platform becoming a bundle of one-off projects. Configuration is what varies per customer without code: quotas, region preferences, feature flags, retention, routing policy. Adapters absorb external variance: identity providers, storage systems, logging sinks, approval workflows. Shared services are expensive to rebuild and central to leverage: tenant registry, policy enforcement, metering, audit, failover orchestration. Core product defines the platform: isolation, admission, data access boundaries, observability primitives, internal APIs.

Name the failure conditions explicitly. Cost reduced but users do not adopt: not succeeded. Users adopt but support cannot operate it: not succeeded. Fast but unable to prove tenant boundaries: not succeeded.

Spend time in proportion to risk. Isolation, export and delete, quotas, regional controls and rollback deserve the depth; a row of generic services does not.

The sample exchange at minute zero: the interviewer says "start from minute zero, what do you say?" and the candidate frames the outcome and the cheapest-safe-shared assumption. The probe follows: "what if your cheapest-shared assumption is wrong?" The answer is to change the control boundary, not just the capacity. Test whether any customer needs strict residency or dedicated inference, and bias the architecture toward stronger isolation if so.

The ninety-second architecture summary:

"I'd default to a shared platform because it gives the best cost and operational leverage across 500 enterprise customers. Each request carries authenticated tenant context; that context is enforced at the application boundary, in the data layer, and in storage and queue paths. I'd use namespace-level isolation plus network policy and quotas for the common case, with reserved inference capacity for premium or burst-sensitive customers and dedicated clusters or databases only when risk or performance requires it. The control plane would manage policy and rollout, while regional enforcement would preserve residency and latency constraints. Defense in depth is the point, one control alone is not enough. I'd verify tenant isolation with adversarial tests, monitor for cross-tenant access attempts, and define a clear export/delete workflow that spans all tenant-owned data stores."

The two-minute spoken answer:

I would not start with the model. I would start by separating the feature they asked for, one AI application for 500 tenants, from the business result, which is a cost-efficient shared platform with defensible isolation plus a dedicated path for the exceptions. Four stakeholders mean four definitions of success, so I would clarify which data classes are in scope, whether regional controls are hard requirements, what predictable performance means, and which failure is most expensive. My default is shared-by-default with tiered isolation. The non-negotiable rule is that tenant context comes from verified identity, never the request body, is established once, and becomes immutable. The control plane federates identity, resolves the tenant directory, evaluates policy, schedules the region, and selects keys; the data plane admits at the gateway, applies quotas before expensive inference, and reads and writes only tenant-scoped resources. Seven isolation layers enforce independently, row, object, cache, queue, log, vector index, and key, so a single missed check is not a breach, and a missing tenant predicate fails closed rather than reading whatever matches. I would size for load shape rather than the average, since 20x skew means a few tenants drive peak. I would gate the rollout on an adversarial isolation suite, then internal tenants, small shared tenants, and finally a dedicated tier justified by measurable thresholds. Success is not elegance; it is adoption, an improved workflow, and an operations team that can support it.

The delivery close, in ninety seconds:

"I would deliver this platform in four steps: internal tenants, adversarial isolation testing, small shared tenants, then a dedicated tier for customers whose policy or economics justify it. I would track isolation-test pass rate, cross-tenant incidents, per-tenant p95 latency, quota rejections, cost per tenant, and regional failover time, while separating technical health from adoption and business outcomes. The rollout would have named owners, explicit go/no-go gates, rollback triggers, training, support, and documentation. The system is successful only when users adopt it, the workflow improves, and the operating team can support it."

Defend four trade-off pairs, each with a balanced verdict.

The follow-ups arrive in a predictable order.

Repair the weak answers on the spot. "I'd just put tenant_id on every query" becomes the enforcement stack and the failure mode of a missed predicate. "I'd isolate everything" becomes a question about what everything costs, then the shared default with an exception path. "The control plane can be centralised for all cases" becomes policy management separated from residency and regional execution. "We'll autoscale if latency rises" becomes capacity planning, queue control, fairness and reserved tiers as different things. "Deletion is just a database delete" becomes every place tenant data can exist, with retention and backup constraints. "Noisy neighbours are rare" becomes a containment plan before the incident.

The lines that carry the round:

1. "Tenant context comes from verified identity, never the request body, is established once, and becomes immutable."
2. "The control plane decides; the data plane enforces."
3. "Seven layers enforce independently, so one missed check is not a breach."
4. "A missing tenant predicate fails closed. It never reads whatever matches."
5. "The store refuses to return a row unless the request is already tenant-scoped."
6. "Security defects fail closed; capacity pressure degrades; metadata leaks quarantine."
7. "Size for the load shape, not the average. 20x skew means a few tenants drive peak."
8. "Pay the isolation premium only where a requirement demands it."
9. "A cache hit across tenants is a security incident, not a performance bug."
10. "Blast radius is the design unit."

Practise it three ways:

• ten minutes answering aloud, then cut every sentence that does not change a design decision
• a pair mock where the partner interrupts on "secure", "scale" and "optimised" until the mechanism is named
• one enforcement point (row-level authorisation, tenant-scoped export and delete, or per-tenant rate limiting) written with tests, failure handling and a rollback plan

The interviewer's pivot after a good design is budgets: a SaaS company wants GenAI features across customers with strict tenant isolation and budget controls. Answer it in the same sitting with the same architecture, because the budget check is already in the request path.

The sixty-second line: make tenant_id and permission_signature first-class in every service, and because budgets are contractual, meter before the call, not after.

The second pivot is the §15 scenario where one tenant has unusually high usage. Distinguish healthy adoption from runaway automation first, because the response differs. Ask whether it is legitimate adoption, abuse, a loop, a batch job or an integration bug. The causes are an automated integration loop, heavy document processing, a power-user workflow or a missing quota. Segment usage by API key, user, feature, time of day and prompt version. Watch cost per tenant, cost per user, request source, tokens per request, batch jobs and agent steps. Now: a tenant rate limit, a conversation with the customer, and a pause on the suspicious workflow if necessary. Later: tenant budgets, a usage dashboard, anomaly alerts and integration safeguards. Prevent with contract-aligned quotas and notifications.

"I would first distinguish healthy adoption from runaway automation. The response differs: support scale-up for healthy usage, throttle loops or abuse."

Every strong cost answer is generated by four verbs in order. Measure, by tracing and attributing first. Route, matching model and path to risk. Bound, with limits on steps, tokens, top-k, timeouts and budgets. Cache safely, with tenant, permission and version in the key. Deliver it in six moves: frame the business impact, decompose the path, name the largest measured driver, fix safely, prove with before and after, prevent recurrence.

The scenario's incident is the design's failure mode observed in production, and it tests whether debugging goes beyond "retrieval quality". Frame it as a multi-tenant isolation failure, never as a hallucination.

Acme uses an enterprise support copilot over customer contracts, escalation policies and entitlements. The vector database stores chunks from several enterprise customers in one collection, with tenant_id, doc_tenant_id, visibility, source_connector, acl_hash and metadata_filter_version on each chunk. The intended invariant is that tenant_id on the request matches doc_tenant_id on every retrieved chunk, and the ACL (access control list) service authorises each chunk before it reaches the prompt. On 2026-07-08 a support agent asked whether Acme customers could receive premium onboarding under the 2026 renewal terms. The assistant said yes and cited a "Premium Success Addendum" that belonged to Globex.

2026-07-08T09:12:44.129Z level=warn service=rag-api env=prod region=eu-central-1
  trace_id=trc_7f91c2 request_id=req_acme_88421 query_id=q_20260708_091244_19
  user_id=u_acme_support_044 tenant_id=acme user_role=support_agent
  workflow=contract_entitlement_answer retriever_version=2026-07-07
  metadata_filter_version=v3 filter_expression="visibility in ['public','internal']"
  expected_filter_expression="tenant_id == 'acme' AND visibility in ['public','internal']"

2026-07-08T09:12:44.388Z level=info service=vector-retriever
  trace_id=trc_7f91c2 collection=enterprise_contract_chunks index=contracts_hnsw_v22
  retrieval_k=12 returned_k=12 top_score=0.81
  retrieved_chunk_ids=[chk_globex_9921,chk_acme_1710,chk_acme_1709,chk_globex_9922]
  retrieved_doc_ids=[doc_globex_renewal_2026,doc_acme_msa_2025]
  doc_tenant_ids=[globex,acme,acme,globex]

2026-07-08T09:12:44.421Z level=error service=acl-enforcer
  trace_id=trc_7f91c2 acl_check=skipped reason="pre_authorized_connector_flag=true"
  connector=sharepoint connector_scope=tenant_shared acl_cache_key=acl:v2:sharepoint:contracts
  request_tenant_id=acme doc_tenant_id=globex policy_decision=not_evaluated

2026-07-08T09:12:45.221Z level=error service=response-evaluator
  trace_id=trc_7f91c2 eval=tenant_isolation_violation severity=critical
  response_served=true citation_doc_id=doc_globex_renewal_2026 citation_chunk_id=chk_globex_9921
  tenant_id=acme doc_tenant_id=globex human_review=false

Detect. Start from the known bad answer and open the full trace. Compare tenant_id on the request with doc_tenant_id on the retrieved chunks, read the filter expression the retriever actually used, and check whether ACL was evaluated per chunk. The telemetry already says it. The filter dropped the tenant equality. Two Globex chunks came back. The ACL check was skipped because of a connector flag. The response evaluator flagged a critical isolation violation after the response was served. Then query every trace on metadata_filter_version=v3 for tenant_id != doc_tenant_id, and compare the canary cohort with the control cohort still on v2. Do not re-run the question in the UI; it may not reproduce the canary route, the cache state or the retrieval order. The path is trace-first: request, filter, retrieved chunks, ACL decision, prompt context, citation.

Contain. Disable metadata_filter_version=v3. Purge affected answer-cache entries. Block every response where any retrieved chunk has tenant_id != doc_tenant_id. Force human review for contract-entitlement answers generated in the incident window. Notify the customer, because a leaked clause can expose pricing, support commitments or regulated data.

Root cause. A retrieval deploy on 2026-07-07 introduced filter version v3 to support global product documentation. The filter builder treated connector-level "pre-authorised" documents as safe and omitted the tenant equality predicate for one SharePoint collection. The ACL service then skipped chunk-level checks because it trusted the connector-level scope. The rollout reached 15% of Acme traffic at 08:45 on 2026-07-08. Two layers failed together, which is exactly what section 6's independence rule exists to prevent.

Prevent. Make tenant isolation a non-bypassable server-side predicate. Add a hard post-retrieval invariant check before context assembly. Remove connector-level ACL shortcutting. Add an automated eval that fails deployment if cross-tenant chunks appear in retrieved context. Add a dashboard panel for cross_tenant_retrieval_count by tenant, connector, retriever version and filter version. Say why the enforcement lives outside the LLM (large language model) and outside prompt instructions: a malicious user could deliberately query for rare phrases that retrieve neighbouring tenant documents if filters are weak.

"I would treat this as a critical data isolation incident. First I would disable the new filter version and stop serving answers from suspicious traces. Then I would inspect the trace for request tenant, retrieved chunk tenant, filter expression, and ACL decision. The telemetry already shows tenant_id=acme, doc_tenant_id=globex, and acl_check=skipped, so the root cause is likely the filter/ACL path, not the model. The fix is to enforce tenant equality as a hard backend invariant before context assembly, add post-retrieval validation, purge affected cache, and add a deployment gate that fails on cross-tenant retrieval."

On this design the levels separate on how many boundaries enforce the tenant scope on their own, and who proves it, not on how the model is served.

• The feature is one application for 500 tenants; the business result is a cost-efficient shared platform with defensible isolation and a dedicated path for exceptions.
• Requirements are stated so a test can fail them: five must-haves, three measurable safety goals, an exclusion list, and a component owner for every constraint.
• Sizing is for load shape, not average: 200 QPS at 20x skew means a few tenants drive peak, and the unit-economics equation decides where the isolation premium is paid.
• One diagram splits the control plane that decides from the data plane that enforces, with the fail-closed branch drawn on it.
• Tenant context is derived once from verified identity, never the request body, and becomes immutable before any expensive work.
• Seven isolation layers enforce independently, the store refuses unscoped reads, and the tenancy level is chosen and justified with a named escalation path.
• Quotas and budgets are applied before the call, and fairness is a separate mechanism from rate limits.
• Policy is defined centrally and enforced regionally, and a control-plane outage never continues stale state into an irreversible action.
• Security defects fail closed, capacity pressure degrades, metadata leaks quarantine.
• The release gate is an adversarial isolation suite, with cross-tenant incidents at zero and per-tenant p95 measured separately.
• Rollout goes internal tenants, isolation suite, small shared tenants, then a dedicated tier by policy, each phase with an owner and a go/no-go.
• The fifty minutes are spent in proportion to risk, with four trade-off pairs and the row-tag follow-up ready.
• The cost pivot is answered with the same architecture: meter before the call, attribute per tenant, key caches on tenant and permission.
• The cross-tenant leak is debugged trace-first, contained by rollback and cache purge, and prevented by a non-bypassable predicate plus a deployment gate.

1. Why is tenant_id on every row insufficient? It is one defence layer that covers nothing in caches, blob stores, indexes, background jobs, analytics or admin tooling, and it fails the first time a developer omits the predicate. Enforcement has to exist independently in several layers.
2. Where does tenant context come from, and why not the request body? From verified identity, because a signed token, verified claim or server-side session can be authenticated and logged, while a client-set header or JSON field can be spoofed, replayed or forwarded across tenants.
3. What is the fail-closed rule for a missing predicate? Reject with a security error and never substitute a default tenant; log identity, trace, operation and the missing-scope fact; keep in-flight retries failing closed.
4. A cache key omits the tenant id and the database is still protected. What kind of incident is it? A security incident, because a shared cache can replay another tenant's object; disable the path, invalidate, rebuild from scoped reads, and add a negative test asserting a cross-tenant lookup is a miss.
5. What is the difference between a rate limit and a fair queue? A rate limit is a ceiling on one tenant's own allotment; a fair queue decides the order of service under load so a large tenant's burst cannot delay a small tenant who never exceeded any limit.
6. What does the unit-economics equation say about when to pay for isolation? C_tenant = C_fixed/N + C_usage + C_isolation: pooled economics win when most tenants can share safely, and the isolation premium is paid only where a requirement demands it.
7. What are the three tenancy levels and the honest default? Shared infra with a tenant id on every row, a separate schema or namespace per tenant, and a dedicated deployment per tenant. Start at level 1 with the database enforcing the tenant and a named escalation path for regulated customers.
8. In the incident, which two layers failed and why did that matter? The filter builder dropped the tenant predicate for pre-authorised connectors and the ACL service skipped chunk checks because it trusted connector scope. Two layers failing together is exactly what independent enforcement is meant to make impossible.
9. What is the sixty-second cost answer? Make tenant_id and permission_signature first-class in every service, and meter before the call because budgets are contractual.

Fact-check sources (checked 27 Sep 2026)

• OWASP Authorization Cheat Sheet — trust only verified tokens, claims or sessions, never client-set fields
• New name for Azure Active Directory (Microsoft Learn) — Azure AD is now Microsoft Entra ID
• PostgreSQL docs: Row Security Policies — owners, superusers and BYPASSRLS roles skip RLS (row-level security) unless FORCE ROW LEVEL SECURITY is set
• HHS: Guidance on HIPAA & Cloud Computing — HIPAA (Health Insurance Portability and Accountability Act) does not require single-tenant hosting
• OCC Bulletin 2023-17: Interagency Guidance on Third-Party Relationships — bank rules are risk-based, not a single-tenant mandate
• AWS KMS cryptography essentials (envelope encryption) — what per-tenant data keys wrapped by a master key protect
• Demers, Keshav, Shenker 1989: Analysis and Simulation of a Fair Queueing Algorithm — fair queuing shares a saturated resource proportionally
• RFC 2697: A Single Rate Three Color Marker — a token bucket meters rate; it does not schedule fairly
• ICO: Right to erasure — backups can be kept "beyond use" until overwritten on schedule
• GDPR Article 17: Right to erasure — the erasure duty itself
• OWASP GenAI LLM08:2025 Vector and Embedding Weaknesses — cross-tenant leaks from missing vector access controls