LLM Provider APIs & Model Selection: interview questions

Production questions of the kind asked in applied AI engineering interviews.

1. Larkspur moves its answer generator from gpt-4o-mini to o4-mini. The monthly bill triples, yet the answers in the transcripts are the same length as before. Where did the money go, and how do you show it?

What a strong answer covers:

• Reads usage.completion_tokens_details.reasoning_tokens per call: those tokens are inside completion_tokens, billed at the output rate, and never appear in the reply
• Recomputes cost from usage (prompt tokens times input price plus completion tokens times output price), never from the length of the reply text
• Splits spend by question type and finds how much of the traffic is lookups such as How long does a refund take?, where reasoning buys nothing
• Proposes routing (T7.3) or a lower effort for lookups, and logs reasoning tokens per request so the next change shows up in a dashboard instead of an invoice

2. The product team wants one model for every Larkspur question. You have a fast model and a reasoning model. Which questions go where, and how do you defend the split?

What a strong answer covers:

• A lookup with the answer in one retrieved chunk goes to the fast model; a multi-step question such as proration after 45 days is the reasoning model's case
• Benchmarks both models on the team's own hard case and own easy case, since a hard-only benchmark teaches "always use the strong one"
• Weighs what a wrong answer costs (a wrong refund amount becomes a support ticket) against latency and price per answer
• Names the router's costly error, a hard question sent to the cheap tier, and makes the router err toward the reasoning tier for billing questions

3. Your effort sweep on 20 golden questions shows high scoring one question more than medium at more than twice the cost. Do you ship high, and how do you know the sweep measured anything?

What a strong answer covers:

• Checks that mean reasoning tokens climb across the levels; if they do not, reasoning_effort never reached the model
• Converts the result to dollars per extra accuracy point at production volume and compares it with what a wrong answer costs
• Treats one question out of 20 as five points of noise: re-runs with more trials, and reads which question flipped and whether it flips back
• Audits the golden labels: lesson 4 of T2.2's notebook expects c for an LRU question whose correct answer is B, which caps a model that answers correctly at 5 of 6

4. A customer asks “How much was my last refund?” The assistant, at high effort, spends many reasoning tokens and still answers wrongly. An engineer proposes raising the effort further. What do you do instead?

What a strong answer covers:

• Recognises a missing fact: the amount lives in the account system, not in the handbook or the model, so no amount of thinking finds it
• Adds a tool call to the billing API (T7.1) or retrieval over account data, and an explicit "I don't have that" path when the data is absent
• Points to the signature in lesson 4 of T2.2: flat accuracy while reasoning tokens climb means the wrong lever
• Caps reasoning spend per request and alerts when reasoning tokens rise with no change in the quality metric

5. To cut cost, the team proposes moving the generator to an open model through the Hugging Face InferenceClient, or running it on a GPU in-house. What changes, and what do you check before switching?

What a strong answer covers:

• The call shape barely changes (client.chat.completions.create with the same messages), so the code diff is small and the risk is in behaviour
• With no provider set, the hosted call goes to auto routing; pins a provider so the served model, latency and price stay reproducible
• In-house means owning GPU memory, the model's chat template, generation settings (max_new_tokens, sampling) and uptime
• Re-runs the golden set and compares accuracy, latency and cost per answer against the current model's baseline row before moving any traffic