Document Review Copilot

Document review is: a packet arrives, a human decides yes/no against a rule (payer, law, playbook). Reading is cheap. Signing is expensive.

G08 covers one slice: extract evidence, compare to the right rule version, cite the exact span, draft. A named human still decides.

End to end, as prior auth for a procedure:

1. Packet and payer rule ingest. If page 7’s table is unreadable, we stop ingest.
2. Clinician opens the case with patient scope.
3. We pull the rule that was in force on the procedure date.
4. We extract fields and the supporting sentences.
5. We draft “meets criteria” only where the span actually says that. Bad citation → no green draft.
6. Clinician approves or rejects. We never send to the payer alone.

That’s it: evidence → rule → cited draft → human. Autonomous clinical or legal yes is out.

Extraction is cheap; judgement is expensive. The copilot extracts evidence, compares it with the governing rule, cites the exact supporting span, and drafts. A named clinician, compliance reviewer, or lawyer makes the material decision. Missing evidence, a broken citation, or uncertain access stops the path.

The source study treats three cases as variations of one spine:

The easiest way to frame requirements in an interview is:

Functional = what the system does. Non-functional = how well it does it and what constraints it must satisfy.

Functional requirements — what the system must do

1. Retrieve the governing rule effective for the case.
2. Extract fields and exact source spans.
3. Compare evidence with criteria — flag missing, stale, or conflicting facts.
4. Classify risk.
5. Draft with a span citation for every claim — editable.
6. Verify each span supports the claim and the user may view it.
7. Route material or uncertain cases to a named human.
8. Audit evidence IDs, rule/prompt/model versions, edits, and approval.

Non-functional requirements — how well / under what constraints

First release

One document type, one rule source, internal drafts, every material case approved. No autonomous clinical, legal, or regulatory decision and no external submission. Unmapped ACL or critical parser coverage failure excludes the source until repaired.

Interview shortcut

If asked “What are the requirements?”, say:

“Functionally, extract evidence, match the right rule version, cite the span, and send material cases to a named human. Non-functionally, fail closed on access or broken citations, and measure cost per true issue, not just per document.”

This diagram condenses the source architecture, §4 into the ingest and review lanes. It names where the cheap model and strong LLM (large language model) run and where they lose authority.

Step-by-step architecture

• Step 1. Ingest case records and governing rules with version, effective date, ACL, and page/span metadata.
• Step 2. Parse text and tables. For a high-risk policy, low table or page coverage fails ingestion and triggers OCR/table reprocessing; missing content cannot be retrieved later.
• Step 3. Authenticate the reviewer and apply role plus patient, matter, or tenant scope before retrieving both the rule in force and the case evidence.
• Step 4. A small model extracts typed fields and exact evidence spans. Low-confidence fields become gaps, never invented facts.
• Step 5. Compare fields with the governing criteria, detect missing or conflicting evidence, and assign a risk tier.
• Step 6. Use a structured routine draft or send flagged risk to a strong LLM for an evidence-backed draft. Missing or conflicting support takes the refusal/escalation branch.
• Step 7. Cite the span actually used, then verify both claim-to-span support and permission. A wrong citation blocks the output even if the answer sounds correct.
• Step 8. The human reviewer edits, approves, or rejects. Record evidence, rule version, model/prompt version, edits, and final decision; nothing external is submitted in version one.

Model and agent role: this is a bounded document-review copilot, not an autonomous clinical, legal, or compliance decider. A small extraction model processes the broad page set, and the strong LLM drafts only flagged work. Deterministic rules, the citation verifier, and the named professional control what can be relied on and what leaves the workflow.

Bulk surveillance is a screening funnel: rules and small classifiers process all documents; the strong model sees only borderline or flagged risk. Dedupe templated packets by content hash, batch non-urgent work off-peak, cap context and top-k, and watch the flagged fraction because it drives strong-model spend. Keep the citation verifier on the path that a material draft takes.

Use separate datasets and gates for extraction fields, answer groundedness, citation span accuracy, high-risk escalation, and PHI/permission safety. The source examples offer ≥90% grounded claims, ≥95% correct span citations, ≥95% high-risk escalation, ≥80% task completion, and zero permission/PHI violations; agree final thresholds per workflow. Track reviewer overrides, false-positive burden, preparation time, first-pass approval/denial, p95, and cost per true issue found.

The two source incidents show why gates are separate. In the legal citation incident, the answer was supported but cited the wrong clause; grounding scored 0.91 while citation accuracy was 0.38. In the parser incident, 17 tables were detected but only 9 extracted (52.9% coverage), leaving pages 31–33 and the approval matrix out of the index. Retrieval tuning cannot repair either missing input or a verifier that ignores span mismatch.

Roll out one document type and rule source → offline golden cases → red-team and shadow mode → read-only reviewer pilot → one low-risk category with approved drafting. Expand only while extraction, citation, escalation, and privacy gates hold. Reviewer corrections feed a governed evaluation set, not silent training.

“I would first clarify which decision the reviewer makes and what the assistant must never decide. I would ingest the right rule version and case evidence under patient or matter permissions, with a parser gate for critical tables. A small model extracts typed fields and spans, deterministic checks compare them to criteria and flag gaps, and a strong LLM drafts only flagged cases. Every claim cites the exact span used; a separate verifier blocks a mismatched or forbidden citation. A named human approves material cases, and versioned evidence and edits go into the audit. I would launch on one document type, shadow real reviewers, and gate on extraction, citation, escalation, and zero privacy violations. Cost is controlled by cheap screening and strong-model routing, not by skipping review.”

Use the Deep Dive for incident and schema details and the Cheat Sheet for recall.

The Main guide is the spoken design; the unchanged source study holds the full cases. This file keeps the mechanisms and the two incidents that are easiest to miss in a short answer.

The claims-processing variant adds OCR (optical character recognition), anomaly signals, and claim-type rollout, but extraction and missing-information detection precede any adjudication. Do not infer a clinical fact from a “rule out” note. Do not invent step therapy when the EHR (electronic health record) has no record. Low-confidence extraction is marked missing because a wrong field corrupts comparison, risk, and citation downstream.

Map EHR/FHIR, payer PDFs, regulatory library, advisor communications, CLM/playbook, and case systems by owner, freshness, ACL (access control list), and risk. EHR is patient/role scoped, contracts are matter/privilege scoped, compliance obligations vary by jurisdiction/product. A source without usable ACL metadata stays out of production retrieval. Rule versions are selected by effective/expiry date and case attributes, so a request can be judged against the rule in force on its date of service.

PDF layout is a correctness boundary. Detect tables, extract them in table mode, keep row or row-group chunks together with page anchors, and measure table_count_detected, table_count_extracted, extraction_coverage, and layout_confidence. Validate reading order; apply OCR/table fallback where needed. A high-risk policy below the agreed coverage threshold fails ingestion rather than warning. Store page-level coverage in retrieval metadata and rerun offline evaluation on parser or chunker changes.

Minimise PHI (protected health information) and privileged material before prompts, redact logs, retain evidence IDs and policy versions, and audit break-the-glass access where allowed. The retrieval filter checks role plus patient/matter/tenant scope; the output verifier checks it again for each cited span. A retrieved email, clause, or payer PDF is untrusted evidence; keeping it apart from the task lowers injection risk but does not stop it, so retrieved text can never trigger an action or widen access.

Run a small extraction/classification model across the page set. Compare typed evidence with criteria and risk rules; use the strong LLM (large language model) only on flagged or borderline material for drafting and explanation. This is a bounded copilot workflow. The model does not make the clinical necessity, legal position, or compliance ruling. Draft text stays editable. A named professional decides, with material approval bound to the case, evidence, and rule version.

Citation construction and verification are separate. The citation points to the exact source span that supported the drafted claim, not a nearby concise or high-ranked chunk. The verifier checks span-to-claim support and that the user may see the span. Groundedness, citation accuracy, extraction accuracy, escalation, and permission/PHI leakage need separate evaluations. An overall groundedness score cannot compensate for a bad citation. Unsupported or missing claims are blocked or flagged; external submission is disabled in the first release.

In source §14, the assistant answered a termination question correctly but cited a data-processing clause (12.4) instead of the actual termination clause (8.2). The true chunk ranked seventh. answer_supported=true and groundedness 0.91 masked citation_span_match=false and citation accuracy 0.38. Citation generator cite_v5 preferred concise retrieved text; the verifier over-weighted answer groundedness.

Contain by rolling back the citation generator, requiring span match for legal workflows, and sending low-accuracy drafts to human review. Prevent with separate answer/citation/quote-span gates, clause-type features in reranking, and highlighting the actual supporting clause. The interviewer should hear: a correct answer with a false citation launders bad evidence into a professional decision.

The other source §14 incident asks about a restricted-list trade exception above €5M. The answer missed the Head of Compliance plus Legal sign-off row. Parser pdf_extract_v3.2 detected 17 tables but extracted 9, or 52.9%; pages 31–33 contained no text and the threshold matrix was absent. Ingestion still said success because the policy was classified as plain text. The regression affected rotated landscape tables.

Reprocess with OCR/table fallback, manually check the missing pages, and block autonomous answers on affected threshold questions until the corrected index is live. Prevent through table-heavy document classification, a hard coverage gate, parser visual/table regression cases, and retrieval metadata marking incomplete pages. Raising top-k cannot retrieve content that never entered the index.

Use historical cases with SME-labelled fields and human decisions. Source example goals are ≥90% supported claims, ≥95% span citations, ≥95% high-risk escalations, ≥80% task completion, and zero permission or PHI violations; set field-specific extraction bars and false-positive budgets with the customer. Red-team outdated rules, wrong patient, prompt injection in PDFs/messages, missing diagnosis or step therapy, “rule out” misread as confirmed, and “just submit it anyway.” In compliance, high recall on severe violations must be balanced with the reviewer’s false-positive workload.

Roll out one document type and rule → offline golden set → shadow reviewer decisions → read-only pilot → draft-only low-risk category behind approval. Measure reviewer edits/overrides, packet preparation and review time, first-pass approval or denial, citation accuracy, p95, and cost per true issue. Bulk surveillance runs overnight: deterministic rules and small classifiers screen everything, the strong model handles the flagged fraction, and high-risk cases receive priority. Dedupe repeated templates by hash and batch non-urgent work; never save money by silently dropping citation or approval controls.

Use the Main guide for the spoken answer, the Deep Dive for mechanisms, and the unchanged source for the full case.

The copilot extracts and drafts; exact evidence and a named human decide what can be trusted or sent.

Which document and decision? Who approves? Which current rule version? What fields/tables are critical? Patient/matter/tenant scope? What must be refused? False-positive budget and cost per true issue?

Document/rule ingest → OCR (optical character recognition) and table-coverage gate → authorised rule/evidence retrieval → small-model typed extraction → compare/gaps/risk → strong LLM (large language model) only on flagged cases → exact-span citation → independent verifier → named human approval → audit.

Agent role: bounded review copilot. Small model extracts; strong LLM drafts flagged work; neither makes a medical, legal, or compliance ruling.

1. One spine covers prior authorisation, compliance, and contracts; swap the schema, governing rule, and approver.
2. Missing/low-confidence evidence is a gap, not text to invent.
3. High-risk PDF with missing tables fails ingestion; retrieval cannot recover absent content.
4. A correct answer with a wrong citation is high severity; verify the exact span used.
5. Patient, matter, tenant, PHI (protected health information), and privilege boundaries apply before prompts and at output.
6. Material drafts and anything external require the named human; version one has no autonomous submission.

Interactive 3–8 s; packet 10–20 s; bulk surveillance overnight. Example gates: ≥90% grounded claims, ≥95% span citations, ≥95% high-risk escalation, ≥80% task completion, zero permission/PHI violations. Agree final bars per customer.

Rules and small models screen every page; strong LLM only on flagged risk. Measure tokens/document, flagged fraction, extraction quality, and cost per true issue found. One type + rule → golden historical cases → shadow → read-only pilot → approved draft-only rollout.

“Extraction is cheap; judgement is expensive. The citation and human decision remain release gates.”

Extraction is cheap and judgement is expensive, and the copilot must never let the cheap step stand in for the expensive one.

The three personas in this group look like three products: a prior-authorisation assistant, a compliance reviewer, a contract copilot. They are one system with one spine. A document comes in. Structure is extracted and compared against a policy, a playbook or clinical criteria. Gaps are flagged, risk is classified, and material cases go to a human who decides. This page consolidates the group's prompts into one read for the day before. The answer keys are templated. So the argument here is built from their workflows, records, risks and rollout, plus the vendor's prior-authorisation case. Anything constructed for this page rather than taken from a source is labelled as such.

Open by saying what the system is not. It does not make the clinical decision, the compliance ruling or the legal call. It gathers evidence, maps it to the governing rule, writes the draft and explains the risk. A named human approves. That is the whole design. Every component below exists to keep the cheap step from being mistaken for the expensive one.

"I would design this as a permission-aware prior-authorisation drafting system, not a clinical decision-maker. Every generated claim must point back to a source note, code, lab result, or payer rule. If evidence is missing, stale, or conflicting, the assistant should not invent language. It should flag the gap and route the case to a clinician."

The three answer tiers from the answer keys show what that framing buys. The weak answer connects the documents to a vector database and lets an LLM (large language model) answer, maybe behind a chatbot UI. It ignores the workflow, the permission boundary, source freshness, evaluation, human approval and monitoring. The average answer builds RAG (retrieval-augmented generation) with citations and tests it on sample questions. It never separates low-risk assistance from high-risk actions. It never defines the data model or the rollout gates. The strong answer maps the workflow and the risk boundary first. It ingests approved sources with metadata, freshness and ACLs (access control lists). It filters by permission before generation. It cites every claim, exposes uncertainty and missing evidence, and routes risky actions to approval. It proves all of that with golden cases, permission red-team tests, groundedness and citation metrics, and latency and cost budgets. The rollout is staged: offline prototype, shadow mode, read-only pilot, approved write-back.

Ask the discovery questions the answer keys share. Add the vendor case's domain ones. Each answer decides a component.

Map the people. Each notices a different failure first. The vendor case names four for prior authorisation. The other two personas have their equivalents.

Scope out loud before drawing. One document type, one policy source, draft-only. Internal drafts, not external submission. Human approval on every material case. Two assumptions are worth stating if withheld. The assistant is administrative drafting, and the professional judgement stays with the reviewer. A source that lacks ACL metadata is excluded from production retrieval until it is mapped.

A requirement the reviewer cannot test is a preference. Say so. "Catch the risky ones" is a preference. "Recall above the agreed bar on severe violations, with the false-positive burden below the analyst's budget" is a constraint. Only the second one changes the architecture. Split the functional list with MoSCoW. The launch gate is then the smallest set that keeps the promise.

The must-haves are five. Each is a test. Retrieve the governing rule for the case. That is the payer-policy version by payer, plan, procedure, diagnosis and date of service, the regulatory obligation by jurisdiction and product, or the playbook position by clause type. Extract the relevant evidence or clauses from the document and the records, mapped to that rule. Generate a draft with a citation to the exact source span for every claim, and never invent language where evidence is missing. Identify missing evidence, outdated or conflicting documentation and rule gaps, and route ambiguous or high-risk cases to the named human. Keep a complete audit trail of retrieved rules, generated text, reviewer actions and final state.

The should-haves belong in the first usable version. Produce a denial-risk, deviation-risk or violation-risk explanation before the case leaves the system. Let the reviewer edit, approve, reject or override the draft in place. Show confidence, missing evidence and the escalation reason whenever the system is uncertain. Collect reviewer corrections for evaluation, never for model training without governance. Provide admin controls for source inclusion, document freshness, policy rules, blocked actions and audit export. Version every rule source so a decision is judged against the rule in force at the time of the request. Autonomous submission, fine-tuning, cross-tenant review and external communication can wait, and most of them should never arrive.

Declare the non-goals:

• no clinical, regulatory or legal decision by the model
• no submission to a payer, regulator or counterparty without an approver
• no generated text sent externally in version one
• no PHI or privileged content in prompts or logs beyond what the task needs
• no learning from reviewer edits without a governance step

The non-functional requirements are the operating constraints, each stated so a test can fail it.

Every must-have then needs an owner in the architecture. This table is the proof.

The spine is the same in every persona. The interviewer will pick one. Learn the spine once and swap the nouns.

The claims-processing prompt from the OpenAI-style practice set is the same spine again. Its discussion list is classification, OCR (optical character recognition) and structured extraction, policy lookup, missing-information detection, fraud or anomaly signals, human review thresholds, explainability, regulatory constraints, PII (personally identifiable information) protection, evaluation against human decisions, and gradual rollout by claim type. Its one instruction is to avoid promising full automation. Start with extraction, triage, summarisation and missing-information detection before any adjudication.

Map each source with its permission model before choosing an embedding model. The connector's job is translating that model into one internal representation.

One diagram carries the whole design. It is this page's construction from the three answer keys and the vendor case. The organising split is control plane against data plane. Rule versions, schemas, risk thresholds, credentials and evaluation gates live in the control plane. Every document, extraction, comparison, draft and approval lives in the data plane. A control-plane change is a release. A data-plane call is a case.

 ╔══════════════════════════════ CONTROL PLANE (changes are releases) ═══════════════════════════════╗
 ║  rule versions (payer policy · regulation · playbook) · extraction schema per document type          ║
 ║  risk tiers + approval routing rules · ACL mapping · credentials · prompt/model/parser versions       ║
 ║  eval gates (extraction audit · citation span · escalation · PHI leak = 0) · budgets                  ║
 ╚═════════════════════════════════════════╤═════════════════════════════════════════════════════════╝
                                           │ configures every box below
 ╔══════════════════════════════ DATA PLANE (calls are cases) ═══════════════════════════════════════╗
 ║                                                                                                     ║
 ║  INTAKE + RULE INGESTION — asynchronous                                                             ║
 ║   documents (EHR/FHIR · uploads · email · CLM)     rules (payer PDFs · reg library · playbook)       ║
 ║        │ immutable copy + content hash                    │ versioned by effective date             ║
 ║        v                                                  v                                         ║
 ║   PARSE / OCR ─> TABLE DETECT + EXTRACT ─> layout QA gate ─> chunk (rows for tables) ─> index      ║
 ║        │ table coverage < threshold on a high-risk doc = FAIL ingestion, not warn                    ║
 ║                                                                                                     ║
 ║  REVIEW — synchronous per case, ordered by risk                                                     ║
 ║   user ─> gateway (authN, role, patient/matter scope) ─> RULE RETRIEVE (right version, permitted)   ║
 ║        ─> STRUCTURED EXTRACT (schema, cheap model) ─> COMPARE (evidence ↔ criteria, gap detect)     ║
 ║        ─> RISK CLASSIFY (tier) ─> DRAFT (strong model only on flagged tiers) ─> CITE (exact span)   ║
 ║        ─> VERIFY (span match + permission) ─> ROUTE ─┬─ low tier: reviewer accepts/edits            ║
 ║                                                      └─ material: named approver decides            ║
 ║                                              refuse + escalate when evidence missing/conflicting    ║
 ║                                                                                                     ║
 ║  AUDIT + OBSERVABILITY — every stage writes                                                         ║
 ║   audit log (rule IDs + versions · evidence IDs · draft · edits · approval)                          ║
 ║   trace store ─> eval service (golden cases · red-team · shadow) ─> dashboards (operational | safety)║
 ╚═════════════════════════════════════════════════════════════════════════════════════════════════════╝

The same flow as a rendered diagram, for viewers that draw Mermaid:

Read the components in dependency order. That is the order they exist and the order they fail.

Three boundaries are worth pointing at with the diagram up. Each one is a line on the board. The cheap-to-expensive boundary sits between EXTRACT and DRAFT. Everything to the left runs a small model on every page. Everything to the right runs the strong model only on what was flagged. The trust boundary sits at VERIFY: no citation leaves that was not the span used and permitted. And the decision boundary sits at ROUTE: to its left the system drafts and explains, to its right a named human decides.

The schema is where the copilot stops being a chatbot. It becomes a workflow. Structured extraction with a typed schema per document type is the one component every downstream step depends on. Comparison, risk tiering and citation all key on fields, not on prose. A model cannot compare evidence to criteria it has not first named.

For prior authorisation the fields are payer, plan, procedure, diagnosis codes, date of service, prior therapies, labs, imaging and the clinical facts that map to each payer requirement. For compliance they are document type, jurisdiction, business unit, the obligation that applies, and the evidence span that meets or breaks it. For contracts they are clause type, the normalised clause text, the counterparty, the jurisdiction and the confidentiality level. Every record in the answer keys is a row in one of those schemas. The EvidenceLink, Finding and RiskFinding records are the join between an extracted field and the rule it was compared against, with a confidence.

Two rules follow. In this design, extraction runs on the cheap model and on every page, because it is a narrow, schema-bound task whose output can be checked. A low-confidence field is flagged as missing rather than filled. A confident wrong field poisons every step after it. The vendor case's step-therapy test names the failure. A payer rule requires step therapy and the EHR has no record of it. The right output is a gap, not an invented history.

A model cannot reason over data that never reached the index. That is the whole section. Table-heavy policy documents are where that happens silently. A plain text-extraction PDF loader usually flattens a table into disconnected fragments. The header row and the data rows land in different chunks with no schema linking them. The parser incident in section 14 is exactly this. 17 tables detected, 9 extracted, pages 31 to 33 missing. An escalation-threshold matrix that was never indexed.

The chunking reference gives the rule. For PDFs, validate extraction order before trusting chunk boundaries. Keep page-number metadata as the only anchor back to the source. Extract tables separately with a table-mode parser rather than as prose. For tables, chunk by row or row group and never split a row. Use zero overlap, because rows are independent records. Then gate ingestion on what the parser reports: extraction_coverage, table_count_detected, table_count_extracted and layout_confidence. Fail a high-risk policy document when table coverage falls below the agreed threshold instead of logging a warning. Classify documents as policy-with-tables when they are. The incident's gate only warned because the document had been labelled plain policy text.

Store page-level extraction coverage in the retrieval metadata. A query that lands on a section with ingestion warnings then carries a warning to the reviewer. Re-run the offline evaluation after any parser or chunking change. Chunking is a retrieval-model change, not a set-once parameter.

Answer correctness and citation correctness are separate dimensions. In a regulated workflow the second one is what the reviewer relies on. A correct answer with a wrong citation is more dangerous than a refusal. The reviewer trusts the citation and repeats it in a negotiation or a submission.

So the citation builder quotes the exact span the drafting step used. Not the most concise chunk, not the highest-ranked chunk nearby. The verifier then checks two things before anything leaves. The cited span actually contains the claim. The user is permitted to see it. A span mismatch fails closed. The citation incident in section 14 shows the alternative. A citation generator was changed to prefer concise citations. A verifier over-weighted answer groundedness. A termination clause was cited to the data-processing addendum.

Playbook comparison in the contract persona is the same mechanism pointed at a rule. Each extracted clause is compared against the playbook's preferred position for its type. A deviation is flagged with a rationale. The copilot proposes the playbook's fallback language rather than inventing its own. Material deviations route to counsel. A fallback is never accepted without a lawyer's approval. In the compliance persona the rule citation is the obligation text by jurisdiction and product. The finding carries the evidence span and a severity.

PHI, privileged material and financial data change where content may travel. Not only who may read it. Minimise what enters the prompt to the fields the task needs. Redact unnecessary PHI from logs. Store policy and evidence IDs rather than raw content with every generated claim. Keep patient-level and matter-level access restrictions in retrieval, never in the prompt. Break-the-glass access, where it exists, is audited with a reason. No training on customer data unless the contract allows it. No vendor system sees raw content that the data classification forbids.

The regulated-industry prompt from the OpenAI-style practice set is a checklist for this section. Each item already has a home in the design:

• data classification and sensitive-data handling
• encryption
• access control
• tenant isolation
• retention and deletion
• audit logging
• data residency
• human oversight
• explainability
• model and prompt versioning
• incident response
• vendor and third-party risk
• governance approvals

When the business wants to skip the governance review for the MVP (minimum viable product), narrow the scope. Use synthetic or masked data where possible. Start read-only or assistive. Define explicit approval gates, and obtain security and legal review before any production exposure.

Risk tiers decide what reaches a human. The vendor case's list is the model. Refuse unsupported claims. Flag missing evidence. Escalate to the clinician. Cite the source. Prevent unauthorised access. Never submit without approval. Anything that would read as a clinical decision, a regulatory ruling or legal advice is a tier the model does not write.

Design the failure path with the happy path. One rule organises the table. The human decision, the permission check and the citation fail closed. Everything else degrades visibly, with the gap labelled.

Say what breaks first at scale. Bulk surveillance is a batch problem, so tier the screening. Rules and small classifiers run on everything. The LLM runs only on borderline cases. Low-risk traffic is sampled. Interactive drill-down keeps its own p95 under 10 s by precomputing embeddings and policy matches and prioritising high-risk queues. Strong-model cost stays bounded only if the flagged tier stays a small fraction of pages, which is an assumption to check. That fraction is the metric to watch.

The answer keys share one evaluation table. The vendor case adds workflow and safety metrics on top. The gate is not a groundedness score alone. The citation incident passed a groundedness gate. It is four separate dimensions, each with its own dataset.

The vendor case's workflow metrics belong on the operational dashboard, kept apart from safety. They are first-pass approval rate, time to prepare a packet, coordinator rework, clinician review time, and denial rate by payer and category. For compliance the equivalents are recall on high-risk violations, false-positive review burden, escalation precision, prompt-injection resilience, audit completeness and model drift by product line.

Red-team the boundary with the vendor case's ten tests. They are concrete, and each maps to one control:

• missing diagnosis code
• outdated payer rule
• conflicting clinical notes
• patient record from the wrong patient
• payer rule requiring step therapy not found in the EHR (electronic health record)
• prompt injection hidden inside an uploaded payer PDF
• a clinician note that says "rule out" treated as confirmed
• medical necessity not documented
• a user who asks to "just submit it anyway"
• a coordinator who tries to open a restricted patient record

The claims-processing follow-up asks how to prove the system is safe enough for production. This section is the answer. Evaluate against human decisions on historical cases. Block the release on any red-team violation. Roll out by claim type, starting with extraction and triage.

Week one is not the whole diagram. It is one document type and one policy source. It is the extraction schema agreed with the SMEs. It is a golden set built from historical cases with their human decisions. The answer keys and the vendor case describe the same staircase.

The OpenAI-style claims practice prompt gives the ordering rule for what to automate. Extraction, triage, summarisation and missing-information detection come first. Adjudication comes only after evaluation and governance. That is the risk tiers, applied over time.

The financial-compliance mock is the adversarial version of this design. It is the round to rehearse. Its customer wants "the AI to review all advisor messages and tell us which ones are risky". The demo caught banned phrases, missed subtle promissory language and followed instructions embedded inside emails. The strong framing is regulated decision support, not autonomous compliance judgement. Extract evidence, map it to policy clauses, propose risk labels with confidence, and preserve an audit trail. Launch requires high recall on severe violations, controlled false positives and explicit resistance to adversarial instructions.

The interviewer pushes on five objections in order. Each has a prepared shape.

The mock's five follow-ups are the ones to have ready. A retrieved document with malicious instructions is untrusted data: it can still sway the model, so it can never trigger an action or widen access. The red-team suite plants such instructions in documents, tickets, emails and logs. A correct answer that cites the wrong source is the incident in section 14. One customer seeing another's data is a tenant predicate on every retrieval, which G07 owns. Cost after launch is the tiered screening above. What is human-approved versus automated is the risk tiers in section 8.

The scoring rubric rewards the same things in every dimension. Quantified business value and failure cost. An explained data flow with auth, eval gates and rollback. Threat-modelled misuse, adversarial tests and launch gates. A phased launch with incident response, and executive-ready honesty about limits. The debrief line is to give concrete numbers: the p95 target, the acceptable false-positive rate, the minimum citation accuracy, the launch-blocking security threshold, and the expected ROI (return on investment).

The interviewer's pivot after a good design is "the copilot costs too much per document". The self-drill on the index answers it with the same architecture. The cheap-to-expensive boundary in section 4 was drawn for exactly this.

The sixty-second line: extraction is cheap, judgement is expensive. Route pages to extraction. Route only the flagged risks to the strong model. Prove quality by sampling. Every strong cost answer is generated by four verbs in order. Measure, by tracing and attributing first. Route, matching model and path to risk. Bound, with limits on tokens, top-k, timeouts and budgets. Cache safely, with tenant, permission and version in the key.

Both scenario incidents are questions about where the design would have caught the failure. Answer each as detect, contain, root cause, prevent. Read the telemetry aloud. The numbers are the argument.

Correct answer, misleading citation

An account executive asked whether a customer could terminate for convenience with 30 days' notice. The assistant answered correctly. It cited Section 12.4, the Data Processing Addendum, instead of Section 8.2, Termination for Convenience.

2026-07-08T11:28:03.331Z level=warn service=citation-verifier
  trace_id=trc_cite_2088 request_id=req_legal_33102 tenant_id=acme
  query="Can Contoso terminate for convenience with 30 days notice?"
  answer_supported=true answer_clause_id=clause_8_2
  citation_doc_id=doc_contoso_msa_2026 citation_clause_id=clause_12_4
  citation_span_match=false citation_generation_version=cite_v5

2026-07-08T11:28:03.401Z level=info service=reranker
  trace_id=trc_cite_2088 retrieved_chunk_rank_for_true_clause=7
  top_ranked_chunk_id=chk_12_4_dpa top_ranked_score=0.74
  true_supporting_chunk_id=chk_8_2_term score=0.68
  reranker_features=[semantic_similarity,heading_match] missing_feature=clause_type_boost

2026-07-08T11:28:04.010Z level=error service=response-evaluator
  trace_id=trc_cite_2088 eval=bad_citation severity=high
  grounding_score=0.91 citation_accuracy_score=0.38
  policy_action=allowed reason="answer_supported=true AND grounding_score>0.85"

Detect: answer_supported=true with citation_span_match=false, and the true clause ranked seventh. Contain: roll back to cite_v4. Require citation span match for legal workflows. Route low citation-accuracy answers to human review. Root cause: cite_v5 changed citation selection from "quote the exact supporting span" to "select the most concise citation from retrieved context". The verifier allowed it because it over-weighted groundedness and under-weighted span matching. The release gate never evaluated citation accuracy separately. Prevent: split evaluation into answer correctness, citation correctness and quote-span faithfulness. Add clause-type features to the reranker. Require the citation to come from the actual supporting span, and highlight the exact clause in the UI.

"I would treat this as a high-severity citation faithfulness issue. The telemetry shows answer_supported=true but citation_span_match=false, with the real clause ranked 7th. I would roll back the citation generator, require span-level verification, and add separate citation accuracy evals. In legal workflows, a correct answer with a wrong citation can be more dangerous than an answer that clearly refuses."

PDF parser drops critical tables

A compliance reviewer asked what approval is needed for a restricted-list trade exception above €5M. The assistant gave the general escalation process. It missed the table row requiring Head of Compliance plus Legal sign-off.

2026-07-08T07:44:12.002Z level=warn service=pdf-parser
  document_id=doc_trade_policy_2026_v4 source_file=TradingPolicy_2026_v4.pdf
  parser_version=pdf_extract_v3.2 page_count=48
  table_count_detected=17 table_count_extracted=9 table_extraction_coverage=52.9%
  layout_confidence=0.61 ocr_fallback_triggered=false ingestion_status=success

2026-07-08T08:02:31.515Z level=info service=chunker
  document_id=doc_trade_policy_2026_v4 chunks_created=184
  pages_with_no_text=[31,32,33] missing_sections=[restricted_list_threshold_matrix]
  chunking_strategy=heading_aware_v2 embedding_model=text_embed_v5

2026-07-08T10:19:07.724Z level=error service=response-evaluator
  trace_id=trc_parse_8827 request_id=req_fin_55019
  query="approval needed for restricted-list trade exception above €5M"
  expected_source_page=32 retrieved_pages=[12,13,14,21]
  table_required=true table_chunk_present=false answer_completeness_score=0.46

Detect: 17 tables detected and 9 extracted, 52.9% coverage, pages 31 to 33 with no text, ingestion_status=success anyway. Contain: reprocess with OCR and table-extraction fallback. Validate pages 31 to 33 by hand. Disable autonomous answers on restricted-list threshold questions until the corrected index is live. Root cause: the parser upgrade from pdf_extract_v2.9 to pdf_extract_v3.2 regressed on rotated landscape tables. The ingestion gate treated low coverage as a warning, because the document was classified as plain policy text rather than policy with tables. Prevent: classify table-heavy policies correctly and fail ingestion below the coverage threshold. Add visual and table regression tests to parser releases. Store page-level coverage in retrieval metadata, and warn at retrieval time when a query targets a section with ingestion warnings. Retrieval tuning cannot recover content that was dropped before indexing.

"The telemetry shows the parser detected 17 tables but extracted only 9, with pages 31–33 missing. The relevant threshold matrix was never indexed. I would reprocess with OCR/table fallback, block affected answers, and add ingestion gates so high-risk PDFs fail if table coverage is low."

Spend minutes in proportion to risk. The decision boundary, the citation verifier and the ingestion gate deserve more of the hour than the embedding model. Write the budget in the corner of the board. Say it once.

The three-sentence close is this page's construction. A permission-aware drafting system extracts structure on every page with a cheap model and spends the strong model only on flagged risk. Every claim cites the exact span that supported it, and a verifier fails closed on any mismatch. A named human decides every material case. The release is gated on extraction accuracy, citation accuracy, escalation quality and zero sensitive-data violations.

The two-minute spoken answer, from the anchor's key:

I would not start with the model. I would start by clarifying the broken healthcare prior authorisation workflow, who uses the system, what decision they need to make, and what risk we cannot automate. For Healthcare Prior Authorisation Assistant, I would design a permission-aware assistant around the workflow: staff prepares prior-auth package by checking payer policy, patient record facts, clinical criteria, missing documents, and drafts a submission for clinician approval. The architecture would ingest approved sources from systems like EHR (electronic health record) via FHIR, payer policy portals, document store, claims/prior-auth system, identity provider, audit/compliance logs, preserve metadata, freshness, and ACLs (access control lists), then use hybrid retrieval with permission filtering before generation. The LLM (large language model) would produce cited answers, show uncertainty, and escalate when evidence is missing or risk is high. Tool use would be allowlisted: read-only tools can run automatically, but any write-back or externally visible action needs preview and human approval. I would evaluate with SME-approved golden cases, citation accuracy, groundedness, permission red-team tests, task completion, latency, and cost. Rollout would be staged: offline prototype, shadow mode, read-only pilot, then limited approved actions with monitoring and rollback. The production goal is not a flashy demo; it is a trusted workflow assistant that is secure, auditable, and measurably improves the business process.

The lines that carry the round, drawn from the sources and this page:

1. "The AI produces an evidence-backed draft, not a final medical or payer decision."
2. "Extraction is cheap, judgement is expensive. Route pages to extraction, route only the flagged risks to the strong model."
3. "Answer correctness and citation correctness are separate evaluation dimensions."
4. "A correct answer with a wrong citation can be more dangerous than an answer that clearly refuses."
5. "Retrieval tuning cannot recover content that was dropped before indexing."
6. "If evidence is missing, stale, or conflicting, the assistant should not invent language. It should flag the gap."
7. "Constrain what the model can see and do before the answer is generated; do not apologise after the mistake."
8. "Never submit without approval. Autonomous submission is disabled by design."

The follow-ups arrive in a predictable order.

Repair the common weak answers on the spot. "The answer is correct, so the citation is cosmetic" becomes citation faithfulness as a high-severity dimension. "Increase top-k or tell the model to read tables" becomes an ingestion gate. Absent content cannot be retrieved. "Put the user's role in the prompt" becomes permissions enforced in retrieval and tool execution. "Use a faster model if it is slow" becomes a latency budget by stage and tiered routing. "Test a few examples and ask users" becomes a golden set with human decisions, adversarial cases and launch gates.

On this design the levels separate on which checks an answer treats as release gates rather than quality knobs, not on how fluent the draft reads.

• The copilot drafts and explains; a named human decides, and every component exists to keep that boundary.
• Requirements are stated so a reviewer can fail them: five must-haves as the launch gate, latency by workflow, cost per true issue found.
• Three personas are one spine: document in, structured extraction, comparison against a rule, gaps and risk, human approval.
• One diagram splits control plane from data plane, and the cheap-to-expensive boundary sits between extraction and drafting.
• The extraction schema is the seam; a low-confidence field is flagged, never filled.
• A high-risk document with low table coverage fails ingestion, because a model cannot reason over what never reached the index.
• The citation is the span that supported the claim, and a verifier fails closed on any mismatch.
• Sensitive data is a boundary on where content travels, and the regulated-industry checklist already has a home in the design.
• The decision, the permission check and the citation fail closed; everything else degrades with the gap labelled.
• The release gate has four dimensions with four datasets, and groundedness alone is not one of them.
• Rollout starts in shadow mode and expands one category at a time, extraction and triage before any adjudication.
• The red-team round is won by narrowing scope, refusing disclaimers as controls, and pricing cost per true issue found.
• The cost pivot is answered by the same architecture: cheap extraction everywhere, the strong model only on flagged risk.
• Incidents on this design are answered by reading the telemetry aloud and naming the gate that was missing.

1. Why is a correct answer with a wrong citation treated as high severity? The reviewer relies on the citation and repeats it in a negotiation or submission, so a wrong citation launders an unsupported claim; answer correctness and citation correctness are separate dimensions.
2. What did the citation incident's release gate check, and what did it miss? It checked answer groundedness, which scored 0.91, and never evaluated citation span accuracy separately, which scored 0.38.
3. Why does increasing top-k not fix the parser incident? The threshold matrix on pages 31 to 33 was never chunked or indexed; retrieval cannot return content that is absent from the index.
4. What should the ingestion gate have done with 52.9% table coverage on a trading policy? Failed the job, because the document is a high-risk policy with tables; it only warned because the document was classified as plain policy text.
5. Where does the strong model run, and why? Only on the flagged risk tier after cheap extraction on every page, because extraction is a narrow, schema-bound task whose output can be checked, and judgement is the expensive step.
6. A payer rule requires step therapy and the EHR shows none. What is the right output? A flagged gap routed to the clinician, never an invented treatment history.
7. How is "which version of the rule" decided? By payer, plan, procedure, diagnosis and date of service against a versioned rule store with effective and expiry dates; a mismatch is counted and shown.
8. What is the answer to "add a disclaimer and let users decide"? A disclaimer does not prevent leakage, stale data, wrong citations, unsafe actions or over-trust; the system constrains what the model sees and does before generation.
9. What is the sixty-second cost answer? Extraction is cheap and judgement is expensive: route pages to extraction, route only flagged risks to the strong model, dedupe by hash, batch the non-urgent queue, and prove quality by sampling.

Fact-check sources (checked 27 Sep 2026)

• FDE (Forward Deployed Engineer) problem-decomposition question bank — claims and regulated-industry practice cases
• HL7 FHIR R5 Subscriptions framework — event-driven FHIR needs Subscriptions; otherwise you poll
• FINRA Books and Records (SEA 17a-4, FINRA 4511) — retention of advisor communications; WORM or audit-trail storage
• FINRA Rule 3110: Supervision — supervisory review of correspondence
• Unstructured partitioning (infer_table_structure) — layout-aware parsers keep table structure that plain loaders flatten
• OWASP GenAI LLM01:2025 Prompt Injection — no fool-proof prevention; limit what the model can do
• MSRC: How Microsoft defends against indirect prompt injection attacks (Jul 2025) — separating task from content lowers but does not remove the risk