Whiteboard Derivations — Deep Dive

Frontier-lab interview prep. Pair with INTERVIEW_GRILL.md.

This deep dive is the catalog of derivations you should be able to do on a whiteboard cold. Frontier-lab interviews routinely ask "derive X" — backprop, attention, OLS gradient, KL, EM, DPO. Knowing the shape of these derivations beats memorizing the answer.

This is a meta-document that points to the relevant deep dive for each derivation while listing the key steps you need to recite.

1. Backpropagation for a 2-layer MLP

Setup: $z_1=W_{1}x+b_1$, $h_1=\sigma (z_1)$, $z_2=W_2{h}_1+b_2$, $\hat{y}=\mathrm{softmax}(z_2)$, $\mathcal{L}=-\sum y\log \hat{y}$.

Steps:

1. Cross-entropy + softmax simplification (the magic step — derive it, don't just assert):

   Softmax Jacobian: $\partial {\hat{y}}_i/\partial z_{2,j}={\hat{y}}_i({\delta }_{ij}-{\hat{y}}_j)$.

   $\partial \mathcal{L}/\partial {\hat{y}}_i=-y_i/{\hat{y}}_i$.

   $\partial \mathcal{L}/\partial z_{2,j}={\sum }_i\frac{\partial \mathcal{L}}{\partial {\hat{y}}_i}\frac{\partial {\hat{y}}_i}{\partial z_{2,j}}=-{\sum }_i\frac{y_i}{{\hat{y}}_i}{\hat{y}}_i({\delta }_{ij}-{\hat{y}}_j)=-y_j+{\hat{y}}_j{\sum }_i{y}_i={\hat{y}}_j-y_j$ (using $\sum y_i=1$).

   So ${\delta }_2=\hat{y}-y$.

2. ${\nabla }_{W_2}\mathcal{L}={\delta }_2{h}_1^{\top }$.

3. ${\nabla }_{b_2}\mathcal{L}={\delta }_2$.

4. ${\delta }_1=W_2^{\top }{\delta }_2\odot {\sigma }^{\prime }(z_1)$.

5. ${\nabla }_{W_1}\mathcal{L}={\delta }_1{x}^{\top }$.

6. ${\nabla }_{b_1}\mathcal{L}={\delta }_1$.

Key insights:

• Cross-entropy + softmax simplifies dramatically: gradient is just $\hat{y}-y$. The mess from softmax's Jacobian and CE's $1/\hat{y}$ cancel.
• Chain rule: each layer multiplies by $W^{\top }$ (transpose) and ${\sigma }^{\prime }$.

See 31_neural_networks/.

2. Scaled dot-product attention

Setup: $Q,K,V\in {\mathbb{R}}^{L\times d}$.

Steps:

1. $\mathrm{scores}=Q{K}^{\top }/\sqrt{d}$.
2. $\mathrm{attn}=\mathrm{softmax}(\mathrm{scores})$.
3. $\mathrm{output}=\mathrm{attn}\cdot V$.

Why $\sqrt{d}$: variance of $Q{K}^{\top }$ entries scales with $d$ if $Q,K$ have unit-variance entries. Divide by $\sqrt{d}$ to keep variance at 1 → softmax doesn't saturate.

Multi-head: project to $h$ heads of dim $d/h$; do attention per head; concatenate; project back.

See 04_transformers/, 05_attention_mechanisms/.

3. OLS closed form

Setup: $\mathcal{L}(w)=\frac{1}{2}\Vert y-Xw{\Vert }^2$.

Steps:

1. ${\nabla }_w\mathcal{L}=-X^{\top }(y-Xw)=X^{\top }Xw-X^{\top }y$.
2. Set to zero: $X^{\top }Xw=X^{\top }y$.
3. Solve: $\hat{w}=(X^{\top }X{)}^{-1}{X}^{\top }y$ (assuming $X^{\top }X$ invertible).

Hessian: ${\nabla }^2\mathcal{L}=X^{\top }X$ — PSD always; PD if $X$ has full column rank.

Geometric: $\hat{y}=Py$ where $P=X(X^{\top }X{)}^{-1}{X}^{\top }$ is the projection onto $\mathrm{Col}(X)$.

See 24_linear_algebra_qa/, 48_optimization_and_matrix_calculus/.

4. Logistic regression gradient

Setup: $p=\sigma (w^{\top }x)$, $\mathcal{L}=-[y\log p+(1-y)\log (1-p)]$.

Steps:

1. $\partial \mathcal{L}/\partial p=-y/p+(1-y)/(1-p)=(p-y)/(p(1-p))$ (combine fractions).
2. $\partial p/\partial z=\sigma (z)(1-\sigma (z))=p(1-p)$ (sigmoid derivative).
3. Chain rule — the magic cancellation: $\partial \mathcal{L}/\partial z=\frac{p-y}{p(1-p)}\cdot p(1-p)=p-y$. The $p(1-p)$ from sigmoid derivative kills the $p(1-p)$ in the denominator from CE — that's the GLM canonical-link beauty.
4. ${\nabla }_w\mathcal{L}=(p-y)x$ (since $z=w^{\top }x$, $\partial z/\partial w=x$).

Key insight: same gradient form as linear regression (residual times input) — that's why these models feel the same. Hessian is $\sum p(1-p)x{x}^{\top }$, always PSD → loss convex.

See 01_classical_ml/, 37_mle_map_estimation/.

5. KL divergence

Definition: $\mathrm{KL}(p\Vert q)={\sum }_{x}p(x)\log \frac{p(x)}{q(x)}$.

Properties:

• $\ge 0$, with equality iff $p=q$ (Gibbs' inequality). Proof via Jensen (memorize this — most-asked):

  $-\mathrm{KL}(p\Vert q)={\sum }_{x}p(x)\log \frac{q(x)}{p(x)}$. Since $\log$ is concave, Jensen's inequality gives $\sum p(x)\log \frac{q}{p}\le \log \sum p(x)\cdot \frac{q(x)}{p(x)}=\log \sum q(x)=\log 1=0$. So $-\mathrm{KL}\le 0$, i.e. $\mathrm{KL}\ge 0$. Equality iff $q/p$ is constant, i.e. $p=q$ (since both are distributions).

• Asymmetric: $\mathrm{KL}(p\Vert q)\ne \mathrm{KL}(q\Vert p)$.

• Forward KL ($\mathrm{KL}(p^{\ast }\Vert q)$): mean-seeking. MLE.

• Reverse KL ($\mathrm{KL}(q\Vert p^{\ast })$): mode-seeking. Variational inference.

MLE = forward KL minimization: $\arg {\max }_{\theta }{\mathbb{E}}_{p^{\ast }}[\log q_{\theta }(x)]=\arg {\min }_{\theta }\mathrm{KL}(p^{\ast }\Vert q_{\theta })+H(p^{\ast })$ — the entropy term is constant.

See 33_information_theory/, 37_mle_map_estimation/.

6. EM for GMM

Setup: $p(x)={\sum }_k{\pi }_k\mathcal{N}(x|{\mu }_k,{\Sigma }_k)$.

E-step: posterior responsibilities

$${\gamma }_{ik}=\frac{{\pi }_k\mathcal{N}(x_i|{\mu }_k,{\Sigma }_k)}{{\sum }_j{\pi }_j\mathcal{N}(x_i|{\mu }_j,{\Sigma }_j)}$$

M-step: weighted MLE updates

$${\mu }_k=\frac{{\sum }_i{\gamma }_{ik}{x}_i}{{\sum }_i{\gamma }_{ik}}$$

$${\Sigma }_k=\frac{{\sum }_i{\gamma }_{ik}(x_i-{\mu }_k)(x_i-{\mu }_k{)}^{\top }}{{\sum }_i{\gamma }_{ik}}$$

$${\pi }_k=\frac{{\sum }_i{\gamma }_{ik}}{N}$$

Why EM converges (the key identity to memorize):

For any distribution $q(z)$: $$\log p_{\theta }(x)=\underset{\mathcal{L}(q,\theta )\text{ — ELBO}}{\underbrace{{\mathbb{E}}_q[\log \frac{p_{\theta }(x,z)}{q(z)}]}}+\underset{\ge 0}{\underbrace{\mathrm{KL}(q(z)\Vert p_{\theta }(z|x))}}$$

So $\log p_{\theta }(x)\ge \mathcal{L}(q,\theta )$ always, with equality iff $q=p_{\theta }(z|x)$.

• E-step: set $q=p_{\theta }(z|x)$ (the posterior responsibilities ${\gamma }_{ik}$). KL = 0 → bound is tight: $\log p_{\theta }(x)=\mathcal{L}(q,{\theta }_t)$.
• M-step: maximize $\mathcal{L}(q,\theta )$ over $\theta$ (since $q$ is fixed, this is just weighted MLE). ${\theta }_{t+1}$ raises the bound.
• Net: $\log p_{\theta }(x_{t+1})\ge \mathcal{L}(q,{\theta }_{t+1})\ge \mathcal{L}(q,{\theta }_t)=\log p_{\theta }(x_t)$. Likelihood non-decreasing → bounded above → converges.

See 19_advanced_clustering/.

7. PCA via SVD

Setup: centered $X\in {\mathbb{R}}^{n\times d}$.

Steps:

1. Center the data, compute covariance: $\Sigma =X^{\top }X/n$.
2. SVD of centered $X$: $X=US{V}^{\top }$ with $U^{\top }U=I$, $V^{\top }V=I$.
3. Substitute and simplify: $X^{\top }X=(US{V}^{\top }{)}^{\top }(US{V}^{\top })=VS{U}^{\top }US{V}^{\top }=V{S}^2{V}^{\top }$ (using $U^{\top }U=I$ — that's the load-bearing step). So $\Sigma =V(S^2/n)V^{\top }$ — this is the eigendecomposition of $\Sigma$.
4. Top-$k$ principal directions: columns of $V$. Variances along them: $S^2/n$.
5. Reduced data: $X{V}_k=U_k{S}_k$ (project data onto top-$k$ directions).

Eckart-Young: truncated SVD $X_k=U_k{S}_k{V}_k^{\top }$ minimizes $\Vert X-X{\Vert }_F^2$ over rank-$k$ $X$.

See 21_dimensionality_reduction/.

8. SVM dual

Primal: ${\min }_w\frac{1}{2}\Vert w{\Vert }^2$ s.t. $y_i(w^{\top }{x}_i+b)\ge 1$.

Lagrangian: $\mathcal{L}=\frac{1}{2}\Vert w{\Vert }^2-{\sum }_i{\alpha }_i[y_i(w^{\top }{x}_i+b)-1]$.

Steps:

1. $\partial \mathcal{L}/\partial w=w-{\sum }_i{\alpha }_i{y}_i{x}_i=0⟹w^{\ast }={\sum }_i{\alpha }_i{y}_i{x}_i$.
2. $\partial \mathcal{L}/\partial b=-{\sum }_i{\alpha }_i{y}_i=0⟹{\sum }_i{\alpha }_i{y}_i=0$ (constraint on $\alpha$).
3. Substitute $w^{\ast }$ back into $\mathcal{L}$ — this is the load-bearing step:
   • $\frac{1}{2}\Vert w^{\ast }{\Vert }^2=\frac{1}{2}{\sum }_{i,j}{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i^{\top }{x}_j$.
   • ${\sum }_i{\alpha }_i{y}_i(w^{\ast \top }{x}_i)={\sum }_i{\alpha }_i{y}_i{\sum }_j{\alpha }_j{y}_j{x}_j^{\top }{x}_i={\sum }_{i,j}{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i^{\top }{x}_j$ (the full quadratic).
   • ${\sum }_i{\alpha }_i{y}_{i}b=b\cdot 0=0$ (using $\sum {\alpha }_i{y}_i=0$).
   • ${\sum }_i{\alpha }_i$ stays.
   • Combining: $\mathcal{L}(w^{\ast },b,\alpha )=\frac{1}{2}{\sum }_{ij}{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i^{\top }{x}_j-{\sum }_{ij}{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i^{\top }{x}_j+{\sum }_i{\alpha }_i={\sum }_i{\alpha }_i-\frac{1}{2}{\sum }_{i,j}{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i^{\top }{x}_j$.

Dual: ${\max }_{\alpha }{\sum }_i{\alpha }_i-\frac{1}{2}{\sum }_{i,j}{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i^{\top }{x}_j$ s.t. $\alpha \ge 0,{\sum }_i{\alpha }_i{y}_i=0$.

Kernel trick: replace $x_i^{\top }{x}_j$ with $K(x_i,x_j)$. The dual is the only place data enters as inner products — perfect for kernels.

KKT — support vectors: complementary slackness gives ${\alpha }_i>0$ only for points where $y_i(w^{\top }{x}_i+b)=1$ (on margin); for soft-margin with $0\le {\alpha }_i\le C$, ${\alpha }_i=C$ for margin violators.

See 35_kernel_functions/, 48_optimization_and_matrix_calculus/.

9. RoPE rotation

Goal: encode relative position via rotation in 2D subspaces.

Setup: pair up dimensions; for pair $(2i,2i+1)$, apply rotation by $m{\theta }_i$ to position $m$:

$$R_m=\left(\begin{array}{cc}\cos m{\theta }_i& -\sin m{\theta }_i\\ \sin m{\theta }_i& \cos m{\theta }_i\end{array}\right)$$

with ${\theta }_i=10000^{-2i/d}$.

Property: $⟨R_{m}q,R_{n}k⟩=⟨q,R_{n-m}k⟩$. Inner product depends only on the relative position $n-m$.

Why this works (the algebra to memorize):

• $⟨R_{m}q,R_{n}k⟩=(R_{m}q{)}^{\top }(R_{n}k)=q^{\top }{R}_m^{\top }{R}_{n}k$.
• Rotations are orthogonal, so $R_m^{\top }=R_m^{-1}=R_{-m}$.
• Rotations also compose by adding angles: $R_{-m}{R}_n=R_{n-m}$.
• Therefore $q^{\top }{R}_{n-m}k=⟨q,R_{n-m}k⟩$ — a function of $n-m$ only.

This is what makes attention self-positionally-aware in a relative way without any added position embeddings to the input.

See 14_advanced_positional_embeddings/.

10. DPO (direct preference optimization)

Starting point: RLHF objective with KL regularization to a reference policy:

$$\underset{\pi }{\max }{\mathbb{E}}_{x,y\sim \pi }[r(x,y)]-\beta \mathrm{KL}(\pi (\cdot |x)\Vert {\pi }_{\mathrm{ref}}(\cdot |x))$$

Step 1 — derive the closed-form optimal policy. Set up Lagrangian on the constrained max (with ${\sum }_y\pi (y|x)=1$). Setting $\partial /\partial \pi (y|x)=0$ gives $\log \pi (y|x)=\log {\pi }_{\mathrm{ref}}(y|x)+r(x,y)/\beta -\log Z(x)-1$, where $Z$ is from the normalization Lagrange multiplier. Cleaning up:

$${\pi }^{\ast }(y|x)=\frac{1}{Z(x)}{\pi }_{\mathrm{ref}}(y|x)\exp (r(x,y)/\beta )$$

with $Z(x)={\sum }_y{\pi }_{\mathrm{ref}}(y|x)\exp (r(x,y)/\beta )$ — depends only on prompt $x$, not on $y$.

Step 2 — invert for $r$:

$$r(x,y)=\beta \log \frac{{\pi }^{\ast }(y|x)}{{\pi }_{\mathrm{ref}}(y|x)}+\beta \log Z(x)$$

Step 3 — substitute into Bradley-Terry: $p(y_w\succ y_l|x)=\sigma (r(x,y_w)-r(x,y_l))$. Critically, $\beta \log Z(x)$ depends on $x$ only — it appears identically in both reward terms and cancels in the subtraction.

Step 4 — final DPO loss (NLL of preferences):

$${\mathcal{L}}_{\mathrm{DPO}}=-\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{\mathrm{ref}}(y_w|x)}-\beta \log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{\mathrm{ref}}(y_l|x)}\right)$$

Key insight: closed-form optimal policy + $Z(x)$ depending only on prompt = reward model eliminates itself. No RL loop, no rollouts, just a supervised classification loss on preferences.

See 08_training_techniques/.

11. Variational lower bound (ELBO)

Setup: latent-variable model $p_{\theta }(x,z)$. Want to maximize $\log p_{\theta }(x)$.

Trick: introduce variational distribution $q(z|x)$ and use Jensen's:

$$\log p_{\theta }(x)=\log \int p_{\theta }(x,z)dz=\log {\mathbb{E}}_{q(z|x)}\left[\frac{p_{\theta }(x,z)}{q(z|x)}\right]$$

Jensen's inequality for concave $\log$: $\log \mathbb{E}[X]\ge \mathbb{E}[\log X]$. Apply it:

$$\log p_{\theta }(x)=\log {\mathbb{E}}_q\left[\frac{p_{\theta }(x,z)}{q(z|x)}\right]\ge {\mathbb{E}}_q\left[\log \frac{p_{\theta }(x,z)}{q(z|x)}\right]={\mathbb{E}}_q[\log p_{\theta }(x,z)]+H(q)$$

This is the ELBO.

Equivalent form (split $\log p_{\theta }(x,z)=\log p_{\theta }(x|z)+\log p(z)$):

$$\mathrm{ELBO}={\mathbb{E}}_q[\log p_{\theta }(x|z)]+{\mathbb{E}}_q[\log p(z)]-{\mathbb{E}}_q[\log q(z|x)]={\mathbb{E}}_q[\log p_{\theta }(x|z)]-\mathrm{KL}(q(z|x)\Vert p(z))$$

The gap to true log-likelihood: $\log p_{\theta }(x)-\mathrm{ELBO}=\mathrm{KL}(q(z|x)\Vert p_{\theta }(z|x))$ — exactly the KL between approximate and true posterior. ELBO is tight when $q$ matches the true posterior.

Reconstruction term + KL-to-prior term. The VAE objective.

See 21_dimensionality_reduction/ (autoencoders), 33_information_theory/.

12. Bias-variance decomposition

Setup: estimate $f^{\ast }(x)$ from random training set $D$. Evaluate at fixed $x$.

Steps:

1. Let $\bar{f}(x)={\mathbb{E}}_D[{\hat{f}}_D(x)]$.
2. Add and subtract: $(y-{\hat{f}}_D{)}^2=(y-\bar{f}+\bar{f}-{\hat{f}}_D{)}^2=(y-\bar{f}{)}^2+2(y-\bar{f})(\bar{f}-{\hat{f}}_D)+(\bar{f}-{\hat{f}}_D{)}^2$.
3. Cross-term vanishes: take ${\mathbb{E}}_D$. $y$ and $\bar{f}$ are constants w.r.t. $D$, so ${\mathbb{E}}_D[2(y-\bar{f})(\bar{f}-{\hat{f}}_D)]=2(y-\bar{f}){\mathbb{E}}_D[\bar{f}-{\hat{f}}_D]=2(y-\bar{f})\cdot 0=0$ (by definition of $\bar{f}$).
4. ${\mathbb{E}}_D[(y-{\hat{f}}_D{)}^2]=(y-\bar{f}{)}^2+{\mathbb{E}}_D[(\bar{f}-{\hat{f}}_D{)}^2]$.
5. Now take $\mathbb{E}$ over the noise in $y=f^{\ast }(x)+ϵ$: first term becomes $(\bar{f}-f^{\ast }{)}^2+{\sigma }^2={\mathrm{Bias}}^2+{\sigma }^2$. Second term is $\mathrm{Var}$.

See 27_advanced_theory/, 52_statistical_learning_theory/.

13. Information gain (decision tree split)

Setup: dataset $S$ with class labels.

Entropy: $H(S)=-{\sum }_c{p}_c\log p_c$.

After split on feature $A$ into $\{S_v\}$:

$$H(S|A)=\sum _v\frac{|S_v|}{|S|}H(S_v)$$

Information gain: $\mathrm{IG}=H(S)-H(S|A)$.

Key identity: $\mathrm{IG}=I(S;A)$ — IG is exactly the mutual information between class label and feature $A$. That makes it intuitive: pick the feature that's most informative about the label.

Why $\mathrm{IG}\ge 0$: conditioning never increases entropy (Jensen on concave $H$, applied to $H(S|A)\le H(S)$). Equality iff $S\perp A$.

Tree picks the split that maximizes IG (or Gini decrease in CART).

Gini: $G(S)=1-{\sum }_c{p}_c^2$. Computationally cheaper (no log); similar selection.

See 26_tree_based_methods/.

14. Common interview gotchas

15. Eight derivations to drill cold

1. 2-layer MLP backprop with cross-entropy + softmax.
2. Scaled dot-product attention with multi-head + masking.
3. OLS gradient + closed form with PSD Hessian.
4. Logistic regression gradient showing convexity.
5. EM for GMM: E-step posterior, M-step updates.
6. DPO loss from RLHF + Bradley-Terry.
7. ELBO derivation via Jensen's inequality.
8. Bias-variance decomposition.

For each: 5 minutes on a whiteboard. Until automatic.

16. Drill plan

• 1 derivation per day for 8 days. Then cycle.
• Time yourself: 5 min per derivation cold; 3 min after a week of practice.
• Practice teaching each: explain to an imaginary interviewer.
• Pair the derivation with the relevant deep dive's "8 most-asked interview questions" to make sure you can recite both proof and intuition.

17. Further reading

This deep dive is a meta-collection. The full derivations live in:

• 31_neural_networks for backprop.
• 04_transformers and 05_attention_mechanisms for attention.
• 01_classical_ml for OLS and logistic.
• 19_advanced_clustering for EM.
• 08_training_techniques for DPO.
• 21_dimensionality_reduction for ELBO/VAE.
• 27_advanced_theory for bias-variance.

Drill the derivations in those locations and you'll be ready for the whiteboard rounds.