LLM Problems & Mitigations — Deep Dive

Frontier-lab interview prep. Pair with INTERVIEW_GRILL.md.

This deep dive covers the practical failure modes of deployed LLMs — long context, hallucination, prompt sensitivity, jailbreaks, agents, and tool use. These are different from training problems (covered elsewhere); they’re what users and engineers actually encounter and what frontier interviews probe to test product-engineering judgment.

1. Long-context challenges

Computational cost

Attention is $O(L^2)$ in sequence length. 128K context → ~16B attention scores per head per layer (and roughly $O(L^{2}d)$ FLOPs to compute attention). Slow even with FlashAttention.

Saying it out loud. Attention compares every token to every other token, so the work grows with the square of the length — double the context and you quadruple the attention cost. At 128K that’s on the order of sixteen billion attention scores per head per layer, which is why long prompts feel slow even on good hardware. FlashAttention helps a lot but it’s fixing the memory traffic, not the asymptotics — it’s still quadratic, just with a much better constant. The practical consequence: prefill cost is where long context hurts, and that’s the part the user waits through before seeing a single token.

Memory cost

In plain language. The KV cache is the model’s working memory for a conversation — every token it has already processed leaves behind a pair of vectors it needs to keep around. The arithmetic below just multiplies out how many of those vectors exist and how many bytes each one takes. KV cache scales linearly with context. 128K tokens of Llama 3 70B KV cache (with GQA-8): $\approx 2\cdot 80\cdot 8\cdot 128\cdot 128,000\cdot 2\approx 41$ GB per request (without GQA, full MHA would push this to ~328 GB).

Saying it out loud. Separate from compute, there’s a memory problem: every token you’ve processed leaves behind key and value vectors that have to stay in GPU memory for the whole request. That grows linearly, and linear is still brutal at scale — a single 128K-token request on a 70B model costs about 41 gigabytes of KV cache, which is more than a whole A100. Grouped-query attention is what makes it survivable; without it, sharing keys across eight query heads, the same request would be pushing 300-plus gigabytes. The number to carry: KV cache, not weights, is what limits how many concurrent long-context users you can serve.

Quality at long context

Lost in the middle (Liu et al. 2023): models recall information at the start and end of context but miss middle. Often >>50% recall at edges, <20% recall at middle.

Saying it out loud. Even when it fits and you can afford it, the model doesn’t actually use the middle of the context well. Recall is U-shaped — strong at the beginning, strong at the end, and it sags badly in between, which is the lost-in-the-middle result from Liu et al. in 2023. The gap is not subtle: you can see well over 50% recall at the edges and under 20% in the middle. That’s why “just use the bigger context window” is the wrong answer — the advertised length is a capacity limit, not a guarantee that anything in the middle gets read.

Mitigations

• Architecture: efficient attention (FlashAttention), sparse attention patterns, sliding window, hybrid SSMs.
• Position encoding: RoPE NTK / YaRN for extension; ALiBi for native extrapolation.
• Training: long-context fine-tuning on documents specifically requiring middle attention.
• Prompting: place critical content at start or end; structure with clear delimiters.
• External: RAG instead of stuffing context.

Saying it out loud. The fixes stack across four levels and I’d walk them in order of how cheap they are. Prompting is free: put the critical material at the top or the bottom, never buried, and use clear delimiters. Retrieval is next — pull in the ten relevant chunks instead of stuffing a thousand pages. Position encoding tricks like YaRN or NTK scaling extend the usable window. And architecture — sliding-window or sparse attention, hybrid state-space layers — is the deepest and slowest lever. The point to make: reordering your prompt costs nothing and often buys more recall than doubling the context window.

When long context wins vs RAG

• Long context: when retrieval is unreliable, when document is small enough, when in-context reasoning needs full text.
• RAG: when corpus is huge, when freshness matters, when sources need cited.

In practice: most production systems use both — RAG for retrieval, but use 32K+ context for the retrieved chunks.

Saying it out loud. It’s not either-or, and saying so is the answer. Long context wins when the document is small enough to fit and the reasoning genuinely needs the whole thing at once — you can’t retrieve your way to “summarize this contract’s internal inconsistencies.” RAG wins when the corpus is huge, when freshness matters, or when you need to cite where something came from. In production you almost always do both: retrieve the relevant chunks, then hand a generous 32K window of them to the model. The tradeoff to name is cost against recall — RAG is cheaper but can miss, long context is expensive but sees everything it’s given, badly.

2. Hallucination

LLMs confidently produce false information. The most-discussed LLM failure.

Types

• Factual: wrong facts about the world. (Most common.)
• Faithfulness: contradicts source documents (e.g., in summarization or RAG).
• Logical: internally inconsistent reasoning.
• Source: invents citations, URLs, papers.

Saying it out loud. Four buckets, and they need different fixes, which is the whole reason to bucket them. Factual means wrong about the world. Faithfulness means it contradicts the document you handed it — that’s the RAG and summarization failure. Logical means the reasoning contradicts itself. Source means it invented a citation. If someone asks “how do I fix hallucination,” my first move is asking which of these they’re seeing, because retrieval fixes factual, entailment checking fixes faithfulness, and neither one touches self-contradictory reasoning.

Causes

• Knowledge cutoff: model doesn’t know recent events.
• Coverage gaps: training data didn’t include the answer.
• Pattern matching: model produces plausible-sounding text without checking facts.
• Greedy decoding: forces the most-likely next token even when uncertain.
• Distribution shift: prompt different from training distribution.

Saying it out loud. The causes split into knowledge problems and behavior problems. Knowledge: the model’s cutoff was two years ago, or the fact was never in the training data at all, or it’s a long-tail entity it saw twice. Behavior: it’s a pattern-matcher, so it produces text shaped like an answer whether or not it has one, and decoding commits to the top token even when the top token is barely ahead. The framing that scores is that none of these are bugs — they’re what next-token prediction does by design, which is why you contain hallucination with system architecture rather than expecting a model release to end it.

Mitigations

• RAG: ground responses in retrieved sources.
• Self-consistency: sample multiple times; pick majority.
• Confidence calibration: produce uncertainty estimates; refuse when low.
• Tool use: outsource factual lookups to search / databases.
• Fine-tuning: train on curated factual + grounded data.
• System prompts: “Cite your sources” or “I don’t know if uncertain.”
• Verification: separate pass to check answer against sources.
• Reasoning models: extended chain-of-thought reasoning helps reduce errors on math/logic.

Saying it out loud. Ranked by how much they actually buy you: retrieval grounding first, because giving the model the document is worth more than any amount of prompting. Then tool use for anything computable — a calculator doesn’t hallucinate arithmetic. Then self-consistency, sample several times and take the majority, which costs you K times inference. Then calibration plus refusal so the model can decline instead of guessing, and a verification pass for high-stakes output. The tradeoff running through all of it is cost and latency against accuracy — every one of these makes the request slower or more expensive, so you apply them by stakes, not uniformly.

Detection

• Reference-free: SelfCheckGPT, NLI-based.
• Reference-based: compare to sources (in RAG, faithfulness metrics).
• Confidence signals: token logprobs, entropy.

Saying it out loud. Detection splits on whether you have something to check against. If you do — retrieved context, a gold answer — it’s an entailment problem: does the source support each claim. If you don’t, you make the model give itself away by sampling it several times and looking for disagreement, which is SelfCheckGPT and its descendants. And there’s a nearly-free tier underneath both: token log-probs and entropy, which cost nothing since you get them during generation. The caveat on that cheap tier is that RLHF broke its calibration, so it’s a feature in a classifier rather than a decision on its own.

3. Prompting

The interface that shapes LLM behavior.

Common techniques

• Zero-shot: direct instruction. “Translate to French: …”
• Few-shot: include examples. “Examples: … Now do: …”
• Chain-of-thought (CoT): ask for reasoning before answer. “Think step by step.” (Wei et al. 2022.)
• Self-consistency: sample multiple CoTs; majority vote.
• Tree of Thoughts: explore multiple reasoning paths; backtrack.
• ReAct: interleave reasoning + actions (tool calls).
• Self-refinement: generate, critique, revise.

Saying it out loud. The ladder goes: zero-shot for anything simple, few-shot when the format matters more than the reasoning, chain-of-thought when there are intermediate steps, self-consistency when you can afford to sample several chains and vote, and ReAct when the model needs to actually go get information. Each step up costs more tokens, and the honest framing is that they’re buying different things — few-shot buys format compliance, chain-of-thought buys multi-step accuracy, and confusing the two is why people add examples to a math problem and see nothing improve. Tree of Thoughts and self-refinement are the expensive end and rarely earn their cost in production.

Prompt sensitivity

• Small wording changes can change benchmark scores 5-10 points.
• Order of few-shot examples matters.
• Position of question in prompt matters.

Saying it out loud. The uncomfortable fact is that rewording a prompt without changing its meaning can move a benchmark score five to ten points. Few-shot example ordering matters. Where the question sits relative to the context matters. Which means any comparison between two models — or two prompt versions — is meaningless unless the spread across paraphrases is smaller than the difference you’re claiming. The practical rule I’d state: measure your prompt’s variance before you trust its mean.

Robustness

Don’t ship a prompt without testing variants. Production prompts are versioned, A/B tested, monitored.

Saying it out loud. Treat prompts like code, because that’s what they are: version them, test variants before shipping, and monitor them after. A prompt that was tuned against last quarter’s model can silently degrade when the vendor ships a minor update, and without version tags in your telemetry you’ll never trace the regression. The failure mode is the classic one — someone edits the system prompt to fix one complaint, nobody re-runs the eval, and quality drops across three other intents nobody was watching.

System prompt structure

1. Role / persona (“You are a helpful assistant”).
2. Instructions / format.
3. Constraints / refusal rules.
4. Examples (if few-shot).
5. Context (RAG, conversation history).
6. User query.

Saying it out loud. The ordering isn’t arbitrary — it’s driven by lost-in-the-middle and by caching. Role and instructions go first because the top of the context is the part the model attends to most reliably; the user’s query goes last for the same reason. Static content up front also means providers can cache that prefix, so you pay for it once instead of on every turn. The mistake to avoid is burying constraints in the middle after a long RAG dump, which is exactly where the model is least likely to read them.

4. Jailbreaks and safety

Jailbreak: prompt that bypasses safety training to elicit refused content.

Common attack patterns

• Roleplay: “You are DAN (‘Do Anything Now’), uncensored AI.”
• Persuasion / authority: “I’m a researcher studying X.”
• Encoding: encoded as base64 / leetspeak to bypass content filters.
• Multi-turn: gradually shift context.
• Indirect injection: malicious instruction in retrieved document or tool output.

Saying it out loud. Five families and I’d name them by mechanism. Roleplay — talk the model into being a character that has no rules. Authority framing — “I’m a researcher, this is for safety work.” Encoding — base64 or leetspeak so the input filter doesn’t see the words. Multi-turn — nothing in any single message is objectionable, and the trajectory is. And indirect injection, which is the one that actually scares people in production: the malicious instruction lives inside a retrieved document or a tool result, so the user never typed it and your input filter never sees it.

Defenses

• RLHF refusal training: train on harmful prompts paired with refusals.
• Constitutional AI: principle-driven self-critique.
• Input filtering: classifier on prompts to detect jailbreak attempts.
• Output filtering: classifier on responses; block if harmful.
• System prompt hardening: explicit instructions to ignore role-play attempts to override.
• Indirect injection mitigations: don’t trust retrieved content; mark untrusted; reduced action permissions.

Saying it out loud. Defense is layered, and no layer holds alone. Refusal training and constitutional self-critique shape the model itself. Then an input classifier for known jailbreak patterns and an output classifier that blocks harmful completions regardless of how they were produced — the output filter matters most because it doesn’t need to anticipate the attack. System prompt hardening helps somewhat. And for indirect injection the real defense isn’t detection at all, it’s permissions: treat everything retrieved as untrusted data rather than instructions, and cap what the agent is allowed to do so a successful injection can’t spend money or delete anything.

Why jailbreaks persist

• Adversarial: defenders + attackers co-evolve.
• Helpful + harmless can conflict — overly cautious model is unhelpful.
• New attack patterns constantly emerge.
• Universal adversarial suffixes (Zou et al. 2023) work across models.

Saying it out loud. They persist because it’s adversarial — every defense is public within a week and attackers iterate faster than model releases. Underneath that is a real tension: helpful and harmless conflict, so every increment of caution costs you legitimate refusals, and there’s no setting where both are maximized. And the Zou et al. result from 2023 is the one to cite — gradient-optimized adversarial suffixes that transfer across models, including ones the attacker never had weights for. The honest framing is that this is a risk-management problem with an equilibrium, not a bug with a fix.

5. Agents and tool use

LLM as orchestrator: decides which tools to call, processes results, plans next action.

Tool use mechanics

• LLM outputs tool call (function name + args, often as JSON).
• System executes tool; returns result.
• LLM continues with result in context.
• Repeat until task complete.

Saying it out loud. Mechanically it’s a loop, and it’s simpler than people expect: the model emits a structured tool call, your runtime — not the model — executes it, the result gets appended to the context, and the model continues. The important thing to say is that the model never runs anything; it produces JSON and your code decides whether to honor it. That’s where validation and permissions live. The failure mode this framing makes obvious: the model can request anything at all, including tools that don’t exist and arguments of the wrong type, so the runtime has to be the one enforcing the schema.

Common tools

• Search: fetch up-to-date info.
• Code interpreter: run code for math, data analysis.
• API calls: external services (weather, calendar, payment).
• Database: query structured data.
• File system: read/write files.

Saying it out loud. The standard kit is search for freshness, a code interpreter for math and data work, API calls for actions, a database for structured lookups, and file access. The way to think about which to add is: what does the model reliably get wrong on its own? Arithmetic and current events, mostly — so a calculator and a search tool cover the majority of hallucination in a typical product. The tradeoff is blast radius: read-only tools like search are nearly free to add, and anything that writes or spends money needs confirmation and permission scoping.

Architectures

• Single-step ReAct loop: think + act + observe + repeat.
• Multi-step plan: generate full plan upfront; execute.
• Hierarchical: planner produces subtasks; executor handles each.
• Multi-agent: specialist agents collaborate (e.g., researcher + writer + critic).

Saying it out loud. Four shapes, roughly in order of ambition. ReAct is the workhorse — think, act, observe, repeat — and it’s reactive, so it recovers well from surprises. Upfront planning is better when the task has known structure and worse when the world doesn’t cooperate, because a plan made before the first observation is often wrong by step three. Hierarchical splits planning from execution so each model has a smaller job. Multi-agent looks elegant in a diagram and mostly multiplies cost and failure surface. The tradeoff to name: planning buys coherence over long horizons and loses adaptability, and most production agents are still ReAct with a step limit for exactly that reason.

Common failure modes

• Tool selection error: model picks wrong tool.
• Argument formatting: malformed JSON, wrong types.
• Infinite loops: model can’t decide when to stop.
• Context bloat: tool outputs exceed context.
• Hallucinated tools: model calls a tool that doesn’t exist.
• Cascading errors: bad early step propagates.

Saying it out loud. Six failure modes and they’re all mundane, which is the point — agents don’t fail dramatically, they fail by picking the wrong tool, malforming JSON, looping forever because nothing told them when to stop, blowing out the context with a giant tool output, calling a tool that doesn’t exist, or making an early mistake that every later step builds on. Cascading errors are the expensive one, because at 95% reliability per step, twenty steps gets you to about 36% end-to-end. That compounding number is the single best thing to say about why agents are still brittle.

Mitigations

• Strict tool schemas: validate JSON; retry on error.
• Step limits: max iterations.
• Output truncation: summarize long tool outputs.
• Tool hints in prompt: clear when to use each.
• Human-in-loop: confirm risky actions.

Saying it out loud. The fixes are engineering discipline, not model quality. Validate every tool call against a strict schema and retry with the error message attached, which recovers most malformed calls in one round. Cap the iteration count so a loop is bounded, not infinite. Truncate or summarize long tool outputs before they hit the context. And put a human in front of anything irreversible. The framing that scores: you’re not making the model more reliable, you’re making its failures cheap and recoverable — that’s what production agent engineering actually is.

6. Multi-turn conversations

Memory management

• Append everything: simple, but context fills up.
• Sliding window: keep last $K$ turns; drop earlier.
• Summarization: periodically summarize older turns.
• External memory: store key facts in retrievable database.

Saying it out loud. Four options on one tradeoff axis. Append everything is perfect fidelity until you run out of context and cost. A sliding window is cheap and forgets whatever fell off the edge, which is usually the thing the user set up in turn one. Summarization keeps the gist and drops the specifics — names, numbers, exact preferences, which are exactly what people expect you to remember. External memory in a retrievable store is the scalable answer and introduces a retrieval problem of its own. In practice: recent turns verbatim, older turns summarized, and durable facts written to a store.

Context coherence

• Models can forget facts mentioned 10+ turns ago.
• Style drift: response style changes over conversation.
• Preference drift: model “agrees” with user’s last opinion.

Saying it out loud. Long conversations degrade in three specific ways worth naming. The model forgets things from ten turns back — partly context length, partly lost-in-the-middle. Style drifts, so the persona you set in the system prompt erodes turn by turn. And preference drift, which is sycophancy: the model starts agreeing with whatever opinion the user expressed most recently. The last one is the most dangerous in advice products, because the model isn’t wrong in an obvious way, it’s just telling the user what they said they wanted to hear.

Personalization

• User preferences as system prompt context.
• User-specific embeddings / fine-tuning.
• Retrieval over user’s history.

Saying it out loud. Three levers, cheapest first: stuff known preferences into the system prompt, retrieve over the user’s own history, or actually fine-tune per user or per segment. Almost everyone should stop at the first two, because fine-tuning per user is an operational nightmare and retrieval over history gets you most of the benefit. The tradeoff to name is privacy and staleness — personalization means storing user data, which brings retention and deletion obligations, and a stale preference is worse than no preference because the model will confidently act on something the user changed their mind about.

7. Latency and cost

Latency sources

• TTFT (Time to first token): prefill phase (compute-bound).
• ITL (Inter-token latency): per-decoded-token latency (memory-bound).
• Network: typically 50-200ms RTT.

Saying it out loud. Split latency into two phases because they’re bottlenecked on different things. Time to first token is prefill — it processes the whole prompt at once and it’s compute-bound, so long prompts make the user wait before anything appears. Then inter-token latency is decode, which is memory-bandwidth-bound because you’re reading the whole model’s weights to produce one token. Network round trip adds 50 to 200 milliseconds on top. The practical implication: if your time-to-first-token is bad, shorten the prompt or cache the prefix; if your streaming is slow, that’s a serving and hardware problem, and shortening the prompt won’t help.

Cost factors

• Per-token cost (input vs output rates).
• Prefill is compute-cheap per token but bursty.
• Long context inflates input cost.
• Retries on tool errors / hallucinations.

Saying it out loud. Output tokens cost several times more than input tokens on most providers, so an answer that rambles is disproportionately expensive. Long context inflates the input side on every single turn of a conversation, which is why chat costs grow superlinearly with conversation length if you just append. And the hidden line item is retries — every malformed tool call, every regenerate after a bad answer, doubles that request. The metric to track is cost per completed conversation, not cost per request, because the per-request number hides exactly the failures you’re paying for twice.

Optimizations

• Prompt caching: providers cache long prompt prefixes (Anthropic, OpenAI).
• Smaller model fallback: route easy queries to small model.
• Batching: aggregate requests in serving layer.
• Streaming: deliver tokens as generated for perceived latency.
• Speculative decoding: as covered in inference deep dive.

Saying it out loud. Biggest wins first: prompt caching, because a long static system prompt and a stable RAG prefix get charged once instead of every turn, and both Anthropic and OpenAI support it. Then routing — send the easy majority of queries to a small model and reserve the frontier model for the hard tail, which is usually the single largest cost reduction available. Batching helps throughput if you own the serving layer, and streaming doesn’t change latency at all but transforms how it feels. Speculative decoding is a real speedup on the decode side with no quality loss, which is rare enough to be worth mentioning by name.

8. Evaluation challenges

Why LLM eval is hard

• Open-ended outputs (no single right answer).
• Subjective quality.
• Many valid responses to same prompt.
• Benchmarks contaminated quickly.
• Capabilities are cross-cutting (factual + reasoning + style).

Saying it out loud. Because there’s no single right answer, so you can’t just check equality, and every metric you can compute is a proxy with its own biases. Quality is partly subjective, many different responses are equally good, and the benchmarks get contaminated within months of publication. On top of that, capability is cross-cutting — one response is simultaneously testing factual recall, reasoning, and tone, so a single scalar hides which of them regressed. The framing that lands is that an eval is a measurement instrument with bias and variance, and most teams under-invest in the instrument.

Methods

• Standard benchmarks: MMLU, GSM8K, MATH, HumanEval, etc.
• LLM-as-judge: stronger LLM grades responses.
• Pairwise preference: human / LLM judges chooses A vs B.
• Capability-specific: faithfulness for RAG, code execution for code.
• A/B test: real users in production.

Saying it out loud. Four tiers, and you want a mix. Standard benchmarks tell you the model isn’t broken and nothing about your product. LLM-as-judge scales to open-ended quality and carries real biases — length, position, self-preference — so it needs calibrating against human labels. Pairwise preference is more reliable than absolute scoring because comparison is an easier judgment than rating. Capability-specific checks are the cheap gold: code that runs, JSON that parses, claims entailed by the source. And A/B is the only one that measures what the business cares about, at the cost of needing real traffic and weeks of it.

Common pitfalls

• Test set contamination.
• Prompt format sensitivity.
• Cherry-picked examples.
• Single-seed sampling.
• Self-preference bias in LLM-as-judge (model rates own outputs higher).

Saying it out loud. The pitfalls are mostly ways of fooling yourself. Contamination, so you’re measuring memorization. Prompt-format sensitivity, so your five-point win was a template choice. Cherry-picked examples. Single-seed sampling, so you’re reporting one draw as if it were the distribution. And self-preference bias, where a model judging its own family’s outputs rates them higher — which is why you use a judge from a different family than either contestant, or an ensemble of three. If I had to pick the one that does the most damage, it’s the uncalibrated judge, because it produces a confident number nobody can check.

9. Common interview gotchas

10. Eight most-asked interview questions

1. What’s the lost-in-the-middle problem and how do you mitigate? (U-shaped recall; place critical info at edges; train on long-context data.)
2. How do you reduce hallucinations in production? (RAG, self-consistency, calibration, tool use, refusal training.)
3. Why does prompt engineering work? (LLMs are sensitive to format / wording; few-shot priming; CoT for reasoning.)
4. Walk through how an agent calls a tool. (LLM outputs JSON tool call; runtime executes; result back in context; loop.)
5. What’s a jailbreak and why do they keep working? (Bypass safety; adversarial co-evolution; helpful-harmless tension.)
6. Multi-turn memory — what’s the trade-off? (Full history bloats context; sliding window forgets; summarization loses detail.)
7. Why is LLM eval hard? (Open-ended; subjective; benchmark contamination; cross-cutting capabilities.)
8. When use long context vs RAG? (Both — RAG for huge corpora; long context for in-context reasoning over retrieved chunks.)

Saying it out loud. These are the eight to be able to answer cold in about sixty seconds each, and the practice that helps is answering them aloud with a punchline first. Lost in the middle becomes “recall is U-shaped, so put the important thing at the top or the bottom.” Long context versus RAG becomes “both — RAG to find it, long context to reason over it.” Agents become “it’s a loop, and the failure is that 95% per step over twenty steps is 36% end to end.” If you can’t finish one in a minute, you’re leading with mechanism instead of with the answer.

11. Drill plan

• Recite lost-in-the-middle U-shape and 3 mitigations.
• For each hallucination type (factual/faithfulness/logical/source), recite cause + fix.
• Sketch a ReAct agent loop with tool call.
• Recite 5 jailbreak patterns + 1 defense each.
• For each prompting technique (zero-shot, few-shot, CoT, self-consistency, ToT), recite when used.
• Walk through latency vs cost trade-offs in a serving system.

12. Further reading

• Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts.
• Wei et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
• Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Models.
• Zou et al. (2023), Universal and Transferable Adversarial Attacks on Aligned Language Models.
• Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback.
• Gao et al. (2023), Retrieval-Augmented Generation for Large Language Models: A Survey.
• Schick et al. (2023), Toolformer: Language Models Can Teach Themselves to Use Tools.