Build When the Customer’s Data Is Poor

The customer wants value this quarter, and the data can only prove what it actually contains.

Poor data is not a reason to refuse the work. It is a reason to change what gets built first. The page answers case #62 from the OpenAI Applied problem-decomposition bank. It follows the scoping discipline of G12 and the gating discipline of G13 without repeating them.

The source gives this case one prompt, eleven talking points and one closing line. It has no "likely follow-up" block, unlike most questions in the bank. Everything past those lines is assembled from other repo material or marked (own construction). The worked example in sections 6 to 10 is entirely own construction, and every number in it is an assumption.

The prompt, verbatim from the question bank:

"The customer wants an AI solution, but its data is incomplete, inconsistent, and poorly documented."

The source lists what a strong answer discusses: data-quality profiling, ownership, missingness, duplicates, and schema inconsistencies. It adds label quality, freshness and data contracts. It ends with human-in-the-loop correction, narrowing the initial use case, and whether AI is actually the right solution.

It also gives the conclusion to land on:

"I would not begin with model selection. I would first determine whether the available data can support a measurable workflow improvement."

Say that sentence early. A model cannot be more reliable than the evidence it is given. A better model mostly produces more fluent answers from the same broken records. The interviewer is testing whether that instinct arrives before the architecture does.

The bank's list of mistakes to avoid applies with extra force here. "Starting with 'I would use RAG (retrieval-augmented generation) and agents'" is mistake one. "Discussing model accuracy without defining evaluation" is mistake five. Poor data makes both worse, because the labels that would define accuracy are themselves suspect.

Not every data problem blocks every use case. The clarifying questions find out which defects touch the decision the customer actually wants improved.

The last question is on the source's list for a reason. Some "AI projects" are a missing dropdown on an entry form. Fixing the form beats building a model to guess what the dropdown would have said. Saying so is a signal of judgement, not a failure to sell.

The bank's reusable opening fits this case well. Clarify the outcome, user, workflow, constraints and success measure, then state assumptions. The difference here is one extra promise: "and I will measure the data before I trust it."

"The data is bad" is not actionable. Eight distinct failure types hide inside it, and each has a different week-one measurement and a different remedy (own construction, built on the source's talking points).

Two rows deserve emphasis, because in my experience they fool teams most often.

Inconsistent labels are worse than missing labels. A missing label is visibly missing. An inconsistent one looks like ground truth, gets trained on, and then scores the model's copied errors as correct. The Cracking book's routing case is the canonical example. A fine-tune reported 94% accuracy against an 81% prompted baseline, then reached only ~78% in production. One of three compounding causes was that labels came from a previous version of the same system's routing decisions. The model learned the old system's errors, and evaluation marked them right. After deduplication, a split by time and a human-adjudicated relabel, the honest number came out near 83%. Production matched it within two points.

Access restrictions are a data-quality problem in disguise. The delivery framework in G12 treats data access as its own risk, with a mechanism attached: "Start day 1, escalate day 3". Data nobody can read has quality zero, whatever it contains.

A readiness assessment turns "poor" into a number the customer can argue with. It also produces the decision memo that justifies what gets built. One week is enough if the scope is one decision, not the whole estate (own construction).

Day 3 is the most important day. Relabelling by two SMEs measures label consistency directly, instead of asking someone whether the labels are good. Use Cohen's kappa rather than raw agreement: agreement corrected for what two random labellers would achieve by chance. The Cracking book explains why. Raw agreement overstates quality when one label dominates, because always guessing the majority class looks accurate. Its floor for a judge is κ ≥ 0.60, and I'd use the same floor for a pair of humans.

The scorecard scores only the dimensions the target decision depends on (own construction):

The thresholds are assumptions to adjust per domain. The decision rule is not. A red on a dimension the chosen design depends on means redesign or wait. A red on a dimension the design can route around means narrow the scope and say so in writing.

The delivery framework in G12 already has the slot for this week. It allocates days 3–4 to data readiness, and its gate is data_access_granted, signed by the customer SME. The scorecard is what raises the evidence bar behind that signature. Access being live is necessary. Access to data that scores red is not readiness.

Poor data rules out some designs and barely affects others. The move is to pick a design whose weakest input is one the data can supply (own construction).

Weak supervision needs a gloss. Instead of hand-labelling every record, write several cheap labelling rules: a keyword match, a lookup table, a model's guess. Each rule is individually unreliable. Their combined votes, weighted by each rule's estimated accuracy, give a usable label at scale. Tools like Snorkel estimate those weights from where the rules agree and disagree; a small trusted sample checks the result.

Synthetic data carries the sharpest caveat. The RAG study guide's advice on eval sets applies directly: "a small, real eval beats a synthetic one for customer-specific evaluation." Synthetic examples fill gaps in coverage. They cannot tell anyone how the system performs on this customer's records.

Say "not yet" when three things line up. The target decision depends on a field that scores red. No design can route around it. And no human process exists that could generate the missing truth. The honest answer then is a data project with a date, not an AI project with a hope. The delivery framework's intake rule says the same thing about metrics: "Refuse to start. A two-week clock against an unmeasurable goal is worse than no clock."

A worked example makes the method concrete. The scenario is invented for this page, and every number in sections 6 to 10 is an assumption (own construction).

The customer. A manufacturer of industrial pumps processes about 12,000 warranty claims a month. Adjusters decide approve, reject, or request more information, and route suspected design defects to engineering. Each claim takes an adjuster about 11 minutes. The customer wants AI to cut that time and catch defect patterns earlier.

The data, as found in week 1.

That last row decides the design. Adjusters agree on outcomes, so outcomes are a usable label. They disagree on failure codes, so failure codes are not ground truth. Any design that trains on historical failure codes would learn the disagreement.

Functional requirements.

Non-functional requirements.

Success metrics. Adjuster minutes per claim fall from 11 to 6. Share of "OTHER" codes on new claims falls from 38% to under 10%. Defect clusters surface in engineering's queue within two weeks of the first claim, against a current lag of about a quarter.

The architecture is a data-repair pipeline with a small decision layer on top. Draw it that way round. Most of the value in phase 1 comes from the lower half.

  ERP-A ──┐                                        ┌──> quarantine + reason (dead letter)
  ERP-B ──┼──> INGEST ──> VALIDATE / QUALITY GATES ─┤
  scans ──┘    (hash,      (schema, required fields,└──> clean + flagged records
               source,      placeholder detection,              │
               arrival)     freshness)                          v
                                                  EXTRACT (scans, free text -> schema, confidence)
                                                                │
                                                                v
                                                  NORMALISE (code list, units, entity match)
                                                                │
                                                                v
                                         confidence >= threshold ?
                                          │ yes                  │ no
                                          v                      v
                                   DRAFT RECOMMENDATION     REVIEW QUEUE (adjuster)
                                   (fields + evidence)            │
                                          │                       │ correction
                                          v                       v
                                   ADJUSTER DECIDES ──────> LABEL STORE (adjudicated)
                                                                   │
                                          ┌────────────────────────┼─────────────────┐
                                          v                        v                 v
                                   GOLD SET GROWTH         THRESHOLD TUNING     DEFECT CLUSTERS
                                   (stratified, by time)   (per field)          -> engineering

Read the components in dependency order (own construction):

A defect caught at ingestion costs one rejected record. The same defect caught after the model costs a wrong recommendation, a reviewer's time, and some trust. So the gates sit before extraction, not in the output guardrail.

The enterprise RAG (retrieval-augmented generation) design's ingestion pipeline states the rule in one line: "An unmappable document is quarantined, not defaulted to internal." Defaulting is how a latent defect gets indexed. The same rule applies to a warranty claim. A claim with a placeholder serial number is flagged as missing, never assigned a guessed value. The pipeline also records per-source last_synced_at and every rejected document with its reason. It keeps those queryable after the process exits, not just printed. That is the quarantine box in section 7.

Content hashing belongs here too. The same pipeline hashes each document's text and skips re-embedding anything unchanged. In the claims design, that hash stops re-extraction of scanned invoices that have not changed. Extraction is the most expensive step, so this matters for cost as well as correctness.

Repairing at the pipeline treats the symptom. Repairing at source treats the cause. The FDE (Forward Deployed Engineer) cross-team case resolved exactly this conflict with data contracts. Software teams wanted speed, while data and AI teams needed stable, high-quality data. "Data contracts turned an ownership argument into an interface agreement." In this design, the contract makes serial number required at entry and fixes one failure-code list across both ERPs. The pipeline's quarantine report becomes the evidence that the contract is needed.

Poor labels are a starting condition, not a permanent one. The review queue is the machine that turns adjuster time into trusted labels. Design it that way from day one (own construction).

Three rules make corrections usable as labels. First, the correction happens in the tool, on the field, with the original value kept. A reviewer who fixes a claim in the ERP directly leaves no trace the system can learn from. Second, each correction records who made it and when. Labels from a reviewer with low agreement can then be weighted down. Third, disagreements go to adjudication, not to whoever clicked last.

The Cracking book's dataset hygiene applies unchanged. Deduplicate first, because "100,000 raw traces might be a few thousand distinct problems". Split held-out cases by time and user, never at random. Route low-agreement, high-impact cases to human adjudication. It also names the trap in automatic labels. An automatic outcome label such as "user didn't complain" is not the same as "correct." For claims, "approved and not disputed" is a weak label, not a gold one.

Weak supervision earns its place on the failure-code field, where κ was 0.52. Three cheap rules vote: a keyword map from description to code, the part's historical most-common failure, and the extraction model's own guess. Each vote is scored against the adjudicated sample. Where the rules agree with high confidence, the label is used for training. Where they disagree, the claim goes to the review queue. The disagreement itself is the signal that tells the system where people are needed.

Synthetic claims fill one gap only: rare failure modes the history lacks, such as a new pump model with no claims yet. They are tagged as synthetic in the label store and excluded from any headline metric.

An evaluation set cannot wait for clean labels, because clean labels are what the project produces. Build a small, trusted set first and grow it from the label store.

The source material gives three ways to seed it. The RAG study guide suggests mining support tickets or an existing FAQ for real questions with known answers. Its alternative is to "ask 3-4 of their power users for 10 questions each". The question bank's evaluation case lists sampling production traffic, expert labelling, weak supervision and synthetic examples used carefully. It adds measuring grader agreement and human review of difficult cases. For claims, the seed is the 100 claims already relabelled in week 1, adjudicated where the two adjusters disagreed.

Size the set honestly. The Cracking book's arithmetic is n ≈ 15.7 × p(1−p) / δ² per arm, to detect a difference δ near a success rate p. At p = 0.8, a 5-point gain needs ~1,000 cases per arm. A 200-case suite cannot reliably detect anything smaller than ~11 points. A 100-claim seed set therefore gates only large changes. Say so, and detect smaller changes through paired evaluation on the same cases, or through production metrics.

Stratify the set and report per stratum. The book's list is intent, tenant size, data recency and difficulty. For claims it becomes ERP source, product line, attachment type and period. Aggregate accuracy hides a collapse on scanned claims from the acquired ERP, which is exactly where the data is worst.

Include the "insufficient data" case at a realistic rate. A system never shown a claim that should be returned for more information learns to always decide. The book makes the same point about refusal and escalation cases.

Watch the labels in the eval set as closely as the system under test. The enterprise RAG golden set's first war story is a label defect, not a system defect. A document was reported as leaked to an account manager who was permitted to read it. It was simply irrelevant. The fix split the label into two fields, one that gates and one that is only tracked, plus a test that stops the labels drifting again. Its lesson travels: a false alarm from a bad label costs as much trust as a real failure.

Ship the set with a datasheet: collection window, filters, dedup method, split strategy, label source and known biases. The Cracking book's estimate is that an hour of documentation saves weeks of debugging six months later.

The hardest part of this case is a conversation, not a component. Nobody enjoys hearing that their data is poor, and the person told is often the person responsible for it (own construction).

Lead with measurements, not adjectives. "Your data is messy" invites a defence. "38% of claims are coded OTHER, and two of your adjusters agree on the code about half the time beyond chance" invites a plan. The scorecard exists partly to make this conversation factual.

Frame phase 0 as value, not as delay. The readiness week already produces things the customer can use. It shows where the claim backlog is duplicated across the two ERPs. It gives a measured estimate of how much adjuster time goes to hunting for facts in scans. And it produces an agreed code list the business needed anyway. A data contract that stops "OTHER" at entry improves every report the customer runs, with or without AI.

Offer a narrow first win alongside the repair. Extraction of serial numbers and invoice amounts from scans is useful on day one and needs no historical labels. It shortens every claim while the failure-code work matures. The customer then sees progress on the same dashboard where the data problems are tracked.

Name what will not be promised. In the worked example, the first release makes no automatic decisions and gives no defect prediction on product lines with under two quarters of history. Written non-goals protect the relationship when the pressure to overpromise arrives.

The phases follow the readiness scores, not a generic timeline. Each phase exits on a measured condition, in the spirit of G12's gates (own construction).

Shadow mode in phase 3 has the same meaning as in the delivery framework. The system runs against real traffic, decides what it would do, and takes no action, while humans compare. Its first real action is never also the first time nobody is watching.

The data contract runs in parallel with every phase. Its effect shows up as a falling quarantine rate and a falling "OTHER" rate on new claims. Both are tracked on the same dashboard as model quality.

Most failures in this case come from trusting data or labels more than the week-1 numbers justify (own construction, except where attributed).

The round rewards spending the first third on diagnosis. A candidate who draws the pipeline at minute three has skipped the part being tested (own construction).

The two-minute spoken answer.

"I wouldn't start with a model. I'd first find out whether the data can support a measurable improvement in one specific decision. So I'd pick that decision with the customer and spend a week measuring the data it depends on. That means null and placeholder rates, a join test across systems, freshness, and — most important — two experts relabelling the same hundred records so I know whether the historical labels are ground truth or noise. That gives me a scorecard. If the field the decision depends on is red and nothing can route around it, the honest answer is a data project first. Usually something can: extraction from documents doesn't need historical labels, and a review queue turns every correction into a trusted label. So phase one ships extraction with confidence scores and human review. Quality gates at ingestion quarantine bad records rather than defaulting them, and a data contract fixes the worst defects at source. Evaluation starts from a small adjudicated set, reported per segment and split by time. I'd say out loud that a set that small only catches large regressions. Every phase exits on a measured condition, and the system doesn't decide anything on its own until it has matched the experts in shadow mode."

Likely follow-ups. The source gives none for this question, so these are (own construction), each with a one-line answer.

• "The customer says there's no time for a readiness week." Then scope it to two days on one decision; skipping it means discovering the same facts in production, more expensively.
• "What if the experts themselves disagree?" Their disagreement is a finding. Write a label guide, adjudicate, and measure κ again. A task humans cannot agree on cannot have an accuracy target.
• "Can't a large model just handle messy data?" It can read messy text well. It cannot recover a fact that is missing or wrong. And it cannot be evaluated against labels nobody trusts.
• "How do you know the data is getting better?" Quarantine rate, placeholder rate on new records and κ against adjudicated labels, tracked on the same dashboard as model quality.

The pivot is "what does all this review and relabelling cost, and how do you stop it growing?" There is no playbook drill for this case, so the card is a self-drill (own construction).

The four verbs from the cost playbook apply in order. Measure cost per correctly processed claim, including reviewer minutes. Route easy fields to a cheap extractor and uncertain ones to people. Bound the review queue with per-field thresholds tuned from measured precision. Cache safely by content hash, so a re-sync never re-extracts an unchanged scan.

I'd expect review cost to fall on its own as labels accumulate. Thresholds can then be set from measured precision rather than caution. That is the economic argument for building the label store in phase 1 rather than later.

The levels split on how far past the model the answer reaches: into the data, into the label loop, or into the customer's own source systems.

• Start by testing whether the data can support a measurable improvement, not by picking a model.
• Clarifying questions find which data defects touch the decision being improved.
• "Poor" splits into eight failure types, each with its own week-one measurement and remedy; inconsistent labels are the most dangerous.
• A one-week readiness assessment produces a scorecard and a build, narrow or not-yet decision.
• Some designs tolerate poor data; say "not yet" only when a red field blocks every route.
• The worked warranty example turns measured defects into testable requirements.
• The architecture is a data-repair pipeline with a small decision layer on top.
• Quality gates at ingestion quarantine bad records instead of defaulting them, and data contracts fix defects at source.
• The review queue is a label factory, with deduplication, time splits and adjudication.
• The eval set starts small and trusted, is sized honestly, and is reported per stratum.
• Tell the customer with measurements, and frame phase 0 as value.
• Phases exit on measured conditions, with shadow mode before any live decision.
• Most failure modes come from trusting data or labels more than the numbers justify.
• The forty-five minutes front-load diagnosis; the spoken answer ends on shadow mode.
• The dominant cost is expert time, so spend it only where confidence is low and keep every correction.

1. What sentence should open the answer? "I would not begin with model selection. I would first determine whether the available data can support a measurable workflow improvement."
2. Why ask whether a rules or form fix is enough? Because some data problems are an entry-form defect, and fixing the form beats building a model to guess the missing value.
3. Why are inconsistent labels worse than missing ones? Missing labels look missing; inconsistent labels look like ground truth, get trained on, and make evaluation score copied errors as correct.
4. What happens on day 3 of the readiness week, and why use kappa? Two SMEs relabel the same sample; kappa corrects agreement for chance, so a dominant class cannot flatter it. The floor is 0.60.
5. When is "not yet" the right answer? When the decision depends on a red field, no design routes around it, and no human process can generate the missing truth.
6. In the warranty example, why are outcomes usable as labels but failure codes are not? Adjusters agree on outcomes at κ = 0.81 and on codes at only κ = 0.52.
7. Where do quality gates sit, and what do they do with a bad record? At ingestion, before extraction; they quarantine it with a reason and never default a value.
8. What makes a review correction usable as a label? It is made in the tool, on the field, with the original kept, the reviewer and time recorded, and disagreements adjudicated.
9. What can a 200-case eval set detect at p = 0.8? Nothing reliably smaller than about 11 points; a 5-point gain needs about 1,000 cases per arm.
10. Why split the eval set by time rather than at random? Random splits put near-identical records on both sides and inflate every number.
11. How should the customer hear that their data is poor? As measurements with a plan, with phase 0 framed as value they can use without AI.
12. What must be true before the system shows live drafts? Adjuster agreement in shadow mode clears the phase-3 bar, and minutes per claim are measurably down.
13. What is the sixty-second cost answer? Expert time is the expensive resource: spend it only on low-confidence fields, turn every minute into a label, and never re-extract an unchanged page.

Related packs, for cross-reference rather than repetition: G12 Scoping to Deployed Agent for gates and intake refusal; G13 Evaluation and Release Gating for the release gate; G01 Enterprise Knowledge Assistant for messy enterprise documents and permission normalisation; G06 NL Over Governed Data for governed metric definitions; Decomposition Classics for #68, the bank-fraud case built on inconsistent labels.

Fact-check sources (checked 27 Sep 2026)

• Snorkel: Rapid Training Data Creation with Weak Supervision (Ratner et al., 2017) — labelling functions vote, and their accuracies are estimated without ground truth
• Cohen (1960), A Coefficient of Agreement for Nominal Scales — kappa as agreement corrected for chance