Interview prompt
Your brief as the FDE (Forward Deployed Engineer) is to plan and put in place proper Observability of a Customer-Facing AI Application. According to the customer, the AI is slow, it is sometimes wrong, and no one can say why. Before choosing what to capture, settle what you will refuse to capture.
Clarify the customer problem
- When people say slow, which percentile do they mean, and on which path?
- When they say wrong, is it an incorrect answer, an empty one, a hallucination, or a failed tool?
- Which tenants and which workflows feel it most?
- What may we record by default, without extra sign-off?
- Who has to find the cause of a failure without handing it up the chain?
Users and workflows
| User | Workflow | Current pain | AI assist opportunity | Human approval needed? |
|---|---|---|---|---|
| Support agent | ||||
| Engineer on call | ||||
| Security reviewer | ||||
| Executive sponsor |
Requirements
Functional
Non-functional
- Latency target
- Availability target
- Cost budget
- Security/privacy constraints
- Audit/compliance requirement
Data and integration map
| Data source | Format | Owner | Freshness | Permission model | Risk |
|---|---|---|---|---|---|
Proposed architecture
Start from a rendered diagram and shape it to fit this case. Cover each part below:
- Correlation and spans
- Sensitivity classifier
- Trace store
- Sampling policy
- Support access
- Dashboards
Evaluation plan
| Metric | Good threshold | Bad threshold | Test dataset | Owner |
|---|---|---|---|---|
| Redaction pass rate | 0 raw prompts | A single leak | Scanners for PII and secrets | Security |
| Trace coverage | Every complaint has a trace | Failures with no trace | Complaint count against trace count | Platform |
| Mean time to diagnose | Going down | Going up | Timelines of past incidents | Support eng |
| Telemetry overhead | Inside its budget | Skews production behaviour | Latency measured with and without export | Observability owner |
Failure modes
- The sampler throws away the one trace that shows the bug
- Tagging metrics with tenant ID blows up cardinality
- A raw prompt ends up in the logs
- Trace context is lost when a request passes through a queue
- The quality score climbs while fewer people use the product
- Exporting telemetry is what causes the outage
Rollout plan
- Write down the SLOs (service level objectives) users actually feel before anything else.
- Turn "slow" and "wrong" into things you can measure.
- Add instrumentation to one critical path.
- Check every export for raw content.
- Give support a workflow that links straight to the trace.
- Then sample for quality and cost.
Weak vs strong answer
Weak: "I'd turn up the logging and put a dashboard on top."
Strong: "My first call is what we will not collect. Each request gets a stable correlation ID. Sensitive fields are classified, then salted and hashed, before they are stored. Sampling leans towards errors and slow outliers. I'd judge success by one test: can support explain a failure on their own?"
Candidate scorecard
Final 2-minute spoken answer
Rehearse this out loud before you look at it. Summarising the whole design in two minutes is its own skill, and it is usually the last thing you are asked for.