Ownership, Standards, and Failure

CEO-ready answer.

"Two failures, and I own both. First, my opening design at a large insurer: one agent doing everything. It passed demos and failed real testing. The lesson was the mechanism. Too much responsibility in one context degrades tool selection. That let me prevent the same failure later at a lender. Second, at that lender I shipped a working system but cannot prove its impact. I didn't measure resolution time by ticket type from day one. My estimate is that roughly half the volume fit automation well. I can't prove it. Now I agree the success metric and instrument it before the first line of code."

Technical-interviewer answer.

"Two failures, two different root causes. At the insurer, version one put twenty-plus tools in one agent's context. Tool selection degraded under that load. The lesson was the mechanism, not the pattern. Undifferentiated responsibility in one context window degrades selection. At the lender I applied it up front. A triage layer narrows the toolset before the agent ever runs. The second failure was measurement. I shipped a working triage-to-root-cause system without resolution-time instrumentation per ticket type. No baseline means no defensible before-and-after comparison. My estimate is that roughly half of ticket volume was a strong automation fit. I don't quote it as a result. Now the success metric and its instrumentation are defined before the first commit."

GROUNDED — insurer v1 + lender metrics honesty

"Two, at different scales. The clean failure was version one at the insurer — the single do-everything agent. It worked in demos and failed in real testing. What I learned wasn't 'use a Supervisor'; it was the mechanism — undifferentiated responsibility in one context window degrades a model's tool selection — and that mechanism let me predict the same failure one level up the stack later, and pre-empt it at a large Indian consumer lender with a triage layer before the agent ever runs.

The quieter shortfall was the lender itself. I shipped a working triage-to-root-cause analysis system, but I don't have a client-confirmed before/after resolution-time number, because I didn't instrument per-ticket-type resolution time from day one. I can defend an estimate — roughly half of ticket volume was a strong automation fit — but I can't prove it, and I say that plainly. The lesson I've carried since: agree the success metric and instrument it before the first line of code, not as a retrofit. That's now the first thing I do on any customer project."

CEO-ready answer.

"One question decides it: is this a tuning problem or a structural problem? At the insurer, my single-agent design was failing mid-customer project. Pushing through meant more prompt tuning. Changing course meant re-architecting with a shrinking window. Tracing showed the flaw was structural. Too many tools in one context degrade tool choice, and no prompt fixes that. So I changed course, but kept it small: same tools, same data, new orchestration on top. When the same pattern reappeared later, I changed course earlier, before it broke. Push through when the structure is sound. Change course when the structure is the problem."

Technical-interviewer answer.

"The tracing answered it. Prompt changes moved individual cases but did not fix the pattern. The errors were wrong tool selections, and they grew with context size. That is a structural signature. Twenty-plus tool schemas in one context degrade selection regardless of prompt wording. So I made the smallest structural change available. Tools and data stayed, and only the orchestration changed, to a Supervisor routing across specialists. A held-out evaluation set confirmed the change. Prompts lived in a approved and controlled table with a five-minute cache, so tuning inside the new structure needed no redeploy. Later, as domains grew, the Supervisor's context started to bloat. I recognised the signature and moved to a Deep Agent before accuracy dropped."

GROUNDED — insurer pivot logic

"At the insurer, when the single agent broke mid-customer project, 'push through' meant prompt-tuning a design with a structural flaw; 'change course' meant re-architecting with a shrinking window. The deciding question was: is the failure a tuning problem or a mechanism problem? Tracing showed it was mechanism — tool selection degrades with twenty-plus schemas in context — and no amount of prompt work fixes that. So I changed course, but bounded it: same tools, same data, new orchestration on top.

The second time — when the Supervisor itself started re-approaching the same bloat — I changed course earlier, before it broke, because I recognised the mechanism.

The rule: push through when the problem is tuning and the structure is sound; change course when the structure is the problem — and make the change as small as the mechanism allows."

CEO-ready answer.

"The failure you volunteer builds more trust than a clean record. On my own approved and controlled retrieval build, the evaluation harness flagged a leak. A claims manager saw a case outside their assignment. I stopped everything and treated it as real. It was a false alarm. The manager had just been reassigned, and my test data was stale. That was my mistake. I had labelled data carelessly. I wrote up the whole thing, false alarm included, and added a check so it can't recur. The team started reporting their own misses earlier. A false security alarm is dangerous, because people learn to ignore it."

Technical-interviewer answer.

"The harness flagged a leak on my own reference build, Meridian Assist, a 22-document approved and controlled retrieval system. I treated it as a real incident before diagnosing it. The trace showed a claims manager receiving a case outside their assignment. The root cause was my test data. The manager had been reassigned that case, so access was correct and the label was stale. Underneath, I had conflated two labels: 'not relevant to this question' and 'not allowed to see it.' I separated them in the test data and added a check so they cannot be conflated again. The write-up covered the alarm, the false alarm and my labelling error. A noisy security gate trains people to ignore it."

GROUNDED — the false-alarm leak + the failed first system plan

"Two, and I'll take the one that's more uncomfortable. On my own reference build — Meridian Assist, a 22-document approved and controlled RAG (retrieval-augmented generation) system — my evaluation harness flagged a leak: a claims manager shown a case outside their assignment. I stopped everything and treated it as real before I'd fully diagnosed it. It turned out the manager had actually just been reassigned that case: my test data was stale, not the system.

I wrote up the whole thing — the alarm, the false alarm, and the fact that I'd mislabelled my own test data by confusing 'not relevant to this question' with 'not allowed to see it.' Then I added a check so those two ideas can never be conflated again, because a false security alarm is worse than no alarm — people learn to ignore it.

The response Your detail from the team — e.g., how people reacted when you owned the false alarm was more trust, not less. And on a customer customer project I'd already told the sponsor mid-build that my first system plan didn't work and why. What I've found is that the failure you volunteer builds more trust than the clean track record you present."

CEO-ready answer.

"Yes, deliberately, and in writing. On an agent guardrail engine, the locks and retry keys lived in process memory. They don't survive a restart. I shipped it because the build had to prove three things: no destructive action without approval, no double-apply on retry, no re-run after a crash. Where the keys live doesn't change those. I documented the live system path in the coverage map. At a large lender, the triage classifier was simple pattern matching, brittle to new phrasing. Clarification tickets had no path yet. I flagged both as the next gaps. Imperfect is fine when it is chosen, bounded and written down."

Technical-interviewer answer.

"I separated the properties under test from the storage layer. The guardrail engine has no LLM (large language model) in it, by design. It proves three properties with 21 deterministic tests. No destructive action without approval, no double-apply on retry, no re-run after a crash. Entity locks and idempotency keys live in in-process Python dictionaries. They don't survive a restart or share across workers. The properties depend on key shape and check order, not on the store. So the live system path is the same key shape in Redis or Postgres. That is written in the coverage map. At the lender, the triage classifier is pattern-based. It is cheap and interpretable but brittle to phrasing drift. Both gaps are written down."

GROUNDED — Agent Platform in-process state + lender triage

"Deliberately, and in writing. On the agent guardrail engine, the entity locks and idempotency-key store are in-process Python dictionaries — correct in shape, but they don't survive a restart or share across workers. I shipped it that way because the point of the build was to prove three properties — no destructive action without approval, no double-apply on retry, no re-run after a crash — and those properties don't depend on where the keys are stored. I documented it directly in the coverage map with the live system path: same key shape, moved to Redis or Postgres.

At a large Indian consumer lender, the triage classifier is pattern-based — cheap, interpretable, and brittle to phrasing drift — and the roughly one-third of tickets that were clarification round-trips have no path at all yet. I flagged both as the next gaps rather than letting the demo imply they were solved.

Imperfect is fine when the imperfection is chosen, bounded, and written down. It's not fine when it's discovered."

CEO-ready answer.

"Done is something you can prove, not a demo that went well. For a approved and controlled AI system, I hold four tests. First, a release gate: one leak on the golden set fails the build. Second, security decisions are made by hard rules, never the model. Third, every step is traced, so a wrong answer leads back to the exact tool call. Fourth, a check fails if the security documentation drifts from the code. Then one honesty requirement: the limits are written down. A small test corpus. State that would not survive a restart. A system that claims no edges is not done. It is untested."

Technical-interviewer answer.

"Done means four mechanisms, each with a failure condition. One, the evaluation harness runs the golden set, and one leak fails the build. Two, access decisions are hard rules in the policy engine, never the model. Three, every node is traced in MLflow, so a wrong answer resolves to the exact tool call. Four, a drift check fails if the security reference stops matching the policy code. Then I document the limits. The corpus is 22 documents, so six retrieval strategies all scoring 22 of 22 proves the gate, not retrieval at scale. The guardrail engine's locks are in process and don't survive a restart. Each limit names what would close it."

GROUNDED — Meridian / insurer / Agent Platform

"Done is a property you can prove, not a demo that went well. For the approved and controlled RAG work, done meant four concrete things. A release gate: the evaluation harness runs the golden set and one leak fails the build. A separation of concerns: leak/no-leak is decided by hard rules, never the model — a security decision can't be probabilistic. Diagnosability: every node traced in MLflow, so a wrong answer can be walked back to the exact tool call that produced it, not just noticed. And documentation that can't drift: a standalone check that fails if the security reference no longer matches the running policy code, so what compliance was told stays true.

Plus one honesty requirement: the limits are written down. The 22-document corpus caveat, the in-process locks in the guardrail engine that wouldn't survive a restart — named in the docs, not discovered by the customer. A system that claims to have no edges isn't done; it's untested."

CEO-ready answer.

"A perfect score can answer the wrong question. My retrieval platform scored 22 of 22 on its golden set. Zero leaks, every strategy near-perfect. But the corpus was 22 documents. At that size retrieval is easy, and the differences between strategies are noise. So I split the claims. The security property holds, and the harness gates every release. That I could defend. Which strategy wins in live system, I could not. I wrote down what would make that claim true: rerun at real scale. Good numbers on the wrong question are dangerous, because nobody argues with them."

Technical-interviewer answer.

"The root cause was a test set too small to separate the strategies. With 22 documents, recall 1.0 and groundedness 1.0 are easy for everyone. The spread between six strategies sat inside the noise. So I split the result by what it actually tested. The zero-leak gate does not depend on corpus size. It checks whether a forbidden document reaches the model, and that property holds at any scale. Strategy ranking does depend on scale, because retrieval gets harder as similar documents multiply. I kept the harness as the release gate and marked the ranking as unproven. The write-up names the next experiment: rerun the same method at a representative size, such as 200,000 documents."

GROUNDED — Meridian 22/22

"The enterprise RAG platform scored 22 of 22 on the golden set — recall 1.0, groundedness 1.0, zero leaks, six retrieval strategies all near-perfect. It looks like a finished result. It isn't, and I said so in the write-up.

The corpus is 22 documents. At that size retrieval is easy, so every strategy scores well and the differences between them are noise. The table proves nothing about which strategy wins at 200,000 documents — which is the question a real customer would actually have. What it does prove, independent of scale, is the zero-leak gate and the testing discipline.

So I separated the claim I could make — 'the security property holds and the harness gates releases' — from the one I couldn't — 'this retrieval strategy is right for live system.' And I wrote down what would need to be true to make the second claim: re-run the benchmark at a representative corpus size. Good numbers on the wrong question are the most dangerous kind, because nobody argues with them."

CEO-ready answer.

"The insurer's compliance lead told me bluntly I'd brought them in too late. I ran discovery in two passes: users first, then compliance, legal and security. They said being second meant reacting to a design instead of shaping it. They were right. Some access rules I'd derived from user interviews had to be reworked. That cost a cycle. Now safety people involved join the first round of discovery, the same week as users. I agree the success bar with compliance before any build. At the consumer lender, that sign-off was settled in the first week. It's slower on day one and faster every day after."

Technical-interviewer answer.

"The feedback exposed a sequencing flaw in my discovery. I derived access rules from user interviews and treated compliance as a review pass. Several rules in the role by document type by market matrix had to be reworked. That rework reached the policy layer and the leak-test cases. The change is structural. safety people involved join the first discovery round. The access matrix and the success bar are signed off before the build. They become inputs to the policy code and the evaluation set, not corrections to them. At the consumer lender, that sign-off was settled in the first week. It sits on the plan as a dated dependency, not an assumption inside my timeline."

PARTLY GROUNDED — "bring compliance into the first round of discovery" is listed in your insurer material as a lesson; frame it as feedback only if it actually came from a person involved. Your detail who said it, and how

"At the insurer I ran discovery as two separate conversations — users first, then compliance, legal, and security as a follow-up pass. Your detail who gave the feedback — e.g., the compliance lead told me, fairly bluntly, that being brought in second meant they were reacting to a design rather than shaping it, and that some of the access rules I'd derived from user interviews had to be reworked once they weighed in.

They were right. The access matrix was correct in the end, but it cost a cycle I didn't need to spend. The change I made is simple and permanent: safety people involved are in the first round of discovery, in the same week as users, not a separate pass afterward. It's slower on day one and faster on every day after — and it's the reason I now treat 'agree the success bar with compliance before the build' as a non-negotiable rather than a nice-to-have.

Your detail one line on a later customer project where you applied it."

CEO-ready answer.

"I changed two beliefs, both because my own systems failed in testing. First, I thought a well-prompted model could enforce security. Tell it what it may reveal and trust it to refuse. Then a security test on my retrieval work passed or failed almost at random. The model's judgement was inside a security decision. Now access is structural. The model never sees what it isn't allowed to see. Second, I assumed more tools make a more capable agent. At the insurer, twenty-plus tools made it worse at choosing any of them. Now I design for specialists with small context from the start."

Technical-interviewer answer.

"First change: access moved out of the prompt. My retrieval work had a security test that passed or failed almost at random. The model's judgement was inside the security decision. Now access control is attribute-based and enforced before generation. The model only receives documents the user may see. Leak or no leak is a deterministic check, and tone is evaluated separately as a soft check. Second change: tool count. At the insurer, twenty-plus tool schemas in one context made selection worse. Now each agent gets a small, specialised toolset. At the lender, a triage layer narrows the toolset before the agent runs. The model decides tone and phrasing, never access or amount of work."

GROUNDED — prompt-based security → hard rules; one big agent → specialists

"Two. I used to think you could make an AI system safe by prompting it well — tell the model what it's allowed to reveal and trust it to refuse. I don't believe that anymore. On the enterprise RAG work I found a test that passed or failed almost at random because the model's judgement was inside a security decision. Now access control is structural: the model never sees a document it isn't allowed to see, and leak/no-leak is a hard rule. The model gets to decide tone, never access.

The second: I assumed a capable model with more tools is a more capable agent. The insurer customer project proved the opposite — twenty-plus tools in one context made the agent worse at choosing any of them. I now design for specialisation from the start, small context per agent, and I pre-narrow the toolset before the agent runs, as at the lender.

Both changes came from my own systems failing in testing. That's the only kind of evidence that actually changes my mind."

CEO-ready answer.

"If I'm loose with small claims, a customer should assume I'm loose with big ones. At the insurer, adoption grew after launch. Nobody would have challenged me calling that my result. I report it as a correlated signal, not a controlled experiment. At the lender, I could quote an estimated resolution-time gain. I say I don't have a client-confirmed number. On the RAG platform, I added a check that fails the build if the security documentation drifts from the running policy code. Nobody asked for it. It keeps what compliance was told true after I've left."

Technical-interviewer answer.

"I make honesty mechanical where I can. On the RAG platform, a standalone check compares the security documentation with the running policy code. If they diverge, the build fails. On the guardrail engine, entity locks and idempotency keys live in in-process dictionaries. They don't survive a restart or share across workers. I wrote that into the coverage map, with the live system path: same key shape, moved to Redis or Postgres. On metrics, I keep the evidence class explicit. The insurer's adoption growth is correlated only, because I didn't instrument the outcome from day one. The lender's resolution-time gain is an estimate, not a client-confirmed number. A reviewer should find every limitation without having to ask me."

GROUNDED — attribution caveats, doc-code sync, in-process state disclosure

"Small things, mostly, which is the point. At the insurer, adoption grew after launch; nobody would have questioned me calling that a result. I report it every time as a correlated signal, not a controlled experiment. At the lender I could quote an estimated resolution-time improvement and nobody would check; I say I don't have a client-confirmed number instead.

On the RAG platform I wrote a standalone check that fails the build if the security documentation drifts from the running policy code — nobody asked for it, and its only purpose is to make sure what compliance was told stays true after I've moved on. On the guardrail engine I documented that the locks are in-process and wouldn't survive a restart, in the coverage map, where a reviewer would find it.

None of these are dramatic. But the customer's security team is trusting a claim they can't verify themselves. If I'm loose with the small claims, they should assume I'm loose with the big ones."

CEO-ready answer.

"Multi-agent orchestration. My background is data platforms and large-scale pipelines, including high-volume streaming at a global bank. Two years ago, the agent patterns I use daily weren't live system-ready. I learned by building things that could fail. At a large insurer, a single agent with twenty-plus tools collapsed. I learned why: too much context degrades tool choice. That let me predict the same failure at a consumer lender and design around it from day one. I also rebuilt a approved and controlled retrieval system on a second platform. It showed me where its security guarantees didn't carry over. I couldn't lead a security review without that."

Technical-interviewer answer.

"I learned orchestration by shipping it and watching it fail. The single agent at the insurer taught me the mechanism: tool selection degrades as tool schemas and history fill the context. I moved to a Supervisor, an 8-node LangGraph state machine. Later I moved to a Deep Agent with self-contained subagents and a memory-manager subagent for long-term memory. At the lender, I applied that up front: a triage layer narrows the toolset before the agent runs. On safety, I rebuilt the same retrieval system on a second platform. Its Vector Search index is a derived copy that does not inherit Unity Catalog row filters. That finding produced the live grant re-check before generation. Next on my list is an AST-based code graph for root-cause analysis."

GROUNDED — multi-agent orchestration, and rebuilding on a second platform to learn its gaps

"Multi-agent orchestration — Supervisor and Deep Agent patterns, LangGraph state machines, memory management across conversations. My background is data platforms and large-scale pipelines Your detail e.g., a global tier-1 bank, hundreds of millions of events a day; two years ago none of the agentic patterns I now use daily existed in a form I'd trust in live system.

I learned it the way I learn anything: by building something that could fail. At a large Asian life insurer that meant learning the hard way that a single agent with 20-plus tools collapses, and then learning why — context bloat degrading tool selection — well enough to predict the same failure one level up and pre-empt it at a large Indian consumer lender with a triage layer before the agent ever runs.

The second piece I deliberately learned was platform-specific safety. I rebuilt the same enterprise RAG system on a second platform specifically to find out where its guarantees didn't travel — and found that a Vector Search index — it's called AI Search now — is a derived copy that doesn't inherit Unity Catalog's row filters. I couldn't have led a customer's security review without having found that myself. Next on the list: AST-based code graphs for root-cause analysis, which is where my code-lookup agent needs to go."

CEO-ready answer.

"Two areas, stated plainly. First, I've led as a senior engineer guiding customer project teams, not as a line manager. Hiring, retention and performance conversations are where I have principles but no track record. I'm closing that gap by mentoring engineers on my customer projects and running their design reviews. I won't assume engineering judgement transfers on its own. Second, when the window is tight, I default to solving the problem myself. Sometimes that's right. It's also the fastest way to become the bottleneck. The counterweight is discipline: unblock others first, review designs before code. I have to keep choosing it."

Technical-interviewer answer.

"My growth area is scaling my judgement through other engineers' designs, not my own code. I review designs before code exists, with three questions. What fails silently? What's the blast radius if it's wrong? Which test or trace will tell us? I'm learning to assess engineering judgement from those answers, not from code volume. That's the part of line management I can practise now. The harder habit is restraint. On the insurer's data foundation, building it myself was right, because the schema was the ramp. The cost is that I become the reviewer of everything. So I push decisions into written design reviews and test contracts that others own. The signal I watch for is a design passing review without me in the room."

PARTLY GROUNDED — your material implies the honest answer; validate it.

"Two things I'd say plainly. First, I've led as a senior IC (individual contributor) guiding customer project teams, not as a line manager — hiring, retention, and performance conversations are areas where I have principles but not a track record, and I'd want to be deliberate about learning them rather than assuming engineering judgement transfers. Your detail anything you've done toward this.

Second, I default to solving the problem myself when the window is tight — the insurer's data foundation is an example where that was the right call, but the habit has a cost: it's the fastest way to become the bottleneck. The day-to-day discipline I hold — unblock others first, review designs before code, protect deep work only for the highest-blast-radius decisions — is the counterweight, and it's something I have to keep choosing, not something I've finished learning."