MLE and MAP Estimation — Deep Dive

Frontier-lab interview prep. Pair with INTERVIEW_GRILL.md.

This topic underpins almost everything in classical and modern ML. Cross-entropy, ridge/lasso, Bayesian deep learning, RLHF reward modeling — all are MLE/MAP under specific likelihoods and priors. Senior interviews probe whether you can derive these cleanly, not just recognize them.

1. The likelihood function

Given iid data $X_1,\dots ,X_n\sim p(\cdot |\theta )$ and a parametric family $\{p(\cdot |\theta ):\theta \in \Theta \}$:

$$L(\theta )=\prod _{i=1}^{n}p(x_i|\theta ),\ell (\theta )=\log L(\theta )=\sum _{i=1}^n\log p(x_i|\theta )$$

The likelihood treats $\theta$ as the variable and the data as fixed — opposite of how $p$ is usually written.

Why log? Sums are easier than products, numerically stable (no underflow), and convex programming on $\ell$ is often tractable.

MLE: ${\hat{\theta }}_{\mathrm{MLE}}=\arg {\max }_{\theta }\ell (\theta )$.

2. Worked MLE derivations

Bernoulli

$x_i\in \{0,1\}$, $p(x|\theta )={\theta }^x(1-\theta {)}^{1-x}$.

$$\ell (\theta )=\sum _i[x_i\log \theta +(1-x_i)\log (1-\theta )]=s\log \theta +(n-s)\log (1-\theta )$$

where $s=\sum x_i$. Setting $\partial \ell /\partial \theta =0$:

$$\frac{s}{\theta }-\frac{n-s}{1-\theta }=0⟹{\hat{\theta }}_{\mathrm{MLE}}=\frac{s}{n}=\bar{x}$$

Pure intuition: MLE for Bernoulli is the empirical frequency.

Gaussian (mean and variance unknown)

$$\ell (\mu ,{\sigma }^2)=-\frac{n}{2}\log (2\pi )-\frac{n}{2}\log {\sigma }^2-\frac{1}{2{\sigma }^2}\sum _i(x_i-\mu {)}^2$$

$\partial \ell /\partial \mu =0$: $\hat{\mu }=\bar{x}$.

$\partial \ell /\partial {\sigma }^2=0$: ${\hat{\sigma }}^2=\frac{1}{n}{\sum }_i(x_i-\bar{x}{)}^2$.

The variance MLE has a $1/n$, not $1/(n-1)$. It's biased (too small) — Bessel's correction unbiases it.

Multinomial

$x_i$ is a one-hot category among $K$ classes. Parameters ${\theta }_k$, ${\sum }_k{\theta }_k=1$.

$$\ell (\theta )=\sum _i\sum _k{x}_{i,k}\log {\theta }_k=\sum _k{n}_k\log {\theta }_k$$

where $n_k={\sum }_i{x}_{i,k}$. With Lagrangian for the simplex constraint:

$${\hat{\theta }}_k=n_k/n$$

Empirical frequency of each category.

Poisson

$p(x|\lambda )=e^{-\lambda }{\lambda }^x/x!$. $\ell (\lambda )={\sum }_i[-\lambda +x_i\log \lambda -\log x_i!]$.

${\hat{\lambda }}_{\mathrm{MLE}}=\bar{x}$.

Linear regression

$y_i=w^{\top }{x}_i+{ϵ}_i$, ${ϵ}_i\sim \mathcal{N}(0,{\sigma }^2)$. Likelihood is Gaussian:

$$\ell (w)=-\frac{1}{2{\sigma }^2}\sum _i(y_i-w^{\top }{x}_i{)}^2+\mathrm{const}$$

Maximizing $\ell$ = minimizing squared error = OLS:

$${\hat{w}}_{\mathrm{MLE}}=(X^{\top }X{)}^{-1}{X}^{\top }y$$

Key insight: OLS is MLE under Gaussian noise. The choice of squared loss isn't arbitrary — it's the negative log-likelihood of a Gaussian.

Logistic regression

$y_i\in \{0,1\}$, $p(y=1|x)=\sigma (w^{\top }x)$.

$$\ell (w)=\sum _i[y_i\log \sigma (w^{\top }{x}_i)+(1-y_i)\log (1-\sigma (w^{\top }{x}_i))]$$

This is the negative cross-entropy loss (up to sign). MLE = minimize cross-entropy. No closed form; use iteratively reweighted least squares (IRLS) or gradient descent.

3. Asymptotic theory of MLE

Under regularity conditions (smooth likelihood, identifiable, true ${\theta }_0$ in interior of parameter space):

Consistency: ${\hat{\theta }}_n{\to }_p{\theta }_0$.

Asymptotic normality:

$$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\to \mathcal{N}(0,I({\theta }_0{)}^{-1})$$

where $I(\theta )=-{\mathbb{E}}_x[{\partial }^2\log p(x|\theta )/\partial {\theta }^2]$ is the Fisher information per observation. (Defining it on the joint log-likelihood $\ell ={\sum }_i\log p(x_i|\theta )$ would scale with $n$ and contradict the formula above; per-observation is correct.)

Asymptotic efficiency: variance achieves the Cramér-Rao bound.

Invariance: if ${\hat{\theta }}_{\mathrm{MLE}}$ estimates $\theta$, then $g({\hat{\theta }}_{\mathrm{MLE}})$ estimates $g(\theta )$. So the MLE of standard deviation is the square root of MLE of variance.

These properties make MLE the default estimator in classical ML — but only asymptotically. Finite-sample MLE can be biased, can overfit, and can be unbounded.

4. Bayesian setup and MAP

Bayes' theorem applied to a parameter:

$$p(\theta |x)=\frac{p(x|\theta )p(\theta )}{p(x)}$$

The MAP estimate maximizes the posterior:

$${\hat{\theta }}_{\mathrm{MAP}}=\arg \underset{\theta }{\max }p(\theta |x)=\arg \underset{\theta }{\max }[\log p(x|\theta )+\log p(\theta )]$$

Equivalent to MLE plus a regularizer that comes from the log-prior. Several important consequences:

Gaussian prior → Ridge

$p(w)=\mathcal{N}(0,{\tau }^{2}I)$. Then $\log p(w)=-\frac{1}{2{\tau }^2}\Vert w{\Vert }^2+\mathrm{const}$.

For linear regression with Gaussian likelihood:

$${\hat{w}}_{\mathrm{MAP}}=\arg \underset{w}{\min }[\frac{1}{2{\sigma }^2}\Vert y-Xw{\Vert }^2+\frac{1}{2{\tau }^2}\Vert w{\Vert }^2]$$

Multiply through by ${\sigma }^2$: ridge regression with $\lambda ={\sigma }^2/{\tau }^2$.

Laplace prior → Lasso

$p(w_j)\propto \exp (-|w_j|/b)$. Log-prior is $-|w|/b$ → ${\ell }_1$ penalty. Lasso = MAP under Laplace prior.

Beta prior + Bernoulli → Smoothed estimate

Prior $\theta \sim \mathrm{Beta}(\alpha ,\beta )$. Posterior $\theta |x\sim \mathrm{Beta}(\alpha +s,\beta +n-s)$.

MAP: mode of Beta = $\frac{\alpha +s-1}{\alpha +\beta +n-2}$ (when $\alpha ,\beta >1$).

Posterior mean: $\frac{\alpha +s}{\alpha +\beta +n}$ — gives a smoothed estimate. With $\alpha =\beta =1$ (uniform prior), posterior mean is $(s+1)/(n+2)$ — Laplace smoothing.

This is exactly what NLP people call add-one smoothing.

5. Conjugate priors — the catalog

Conjugate priors give closed-form posteriors. They also yield clean intuition: hyperparameters of the prior look like pseudo-counts — the prior acts as if you'd seen some imaginary data before.

6. MLE vs MAP vs Bayesian — the spectrum

Modern deep learning is almost entirely MLE (cross-entropy, MSE) plus MAP (weight decay, dropout). True Bayesian deep learning is a research area (Bayesian NNs, dropout-as-Bayes-approx, deep ensembles for posterior approximation).

7. Why MLE = minimum cross-entropy = minimum forward KL

For data drawn from true distribution $p^{\ast }$:

$$\arg \underset{\theta }{\max }{\mathbb{E}}_{x\sim p^{\ast }}[\log p(x|\theta )]=\arg \underset{\theta }{\min }{\mathbb{E}}_{x\sim p^{\ast }}[-\log p(x|\theta )]$$

The right-hand expression is the cross-entropy of $p^{\ast }$ relative to $p_{\theta }$. Equivalent:

$$=\arg \underset{\theta }{\min }\mathrm{KL}(p^{\ast }\Vert p_{\theta })+H(p^{\ast })$$

The $H(p^{\ast })$ term doesn't depend on $\theta$, so MLE = minimize forward KL from $p^{\ast }$ to model. This is the "mode-covering" KL — penalizes putting low probability on regions where $p^{\ast }$ is high. (Reverse KL would be mode-seeking; that's what variational inference uses.)

8. Common interview gotchas

9. Eight most-asked interview questions

1. Derive MLE for a Gaussian (both parameters). (Set partials to zero; recognize that variance MLE is biased.)
2. Show that OLS equals MLE under Gaussian noise. (Write log-likelihood, drop constants, recognize squared loss.)
3. Show that ridge equals MAP under Gaussian prior. (Write log-posterior, recognize ${\ell }_2$ penalty with $\lambda ={\sigma }^2/{\tau }^2$.)
4. What's the relationship between cross-entropy and MLE? (CE = negative log-likelihood; minimizing CE = MLE.)
5. Bayesian smoothing for Bernoulli — derive Laplace's rule of succession. (Beta(1,1) prior + observed data → posterior mean $(s+1)/(n+2)$.)
6. What's a conjugate prior and why is it useful? (Same-family posterior; closed-form updates; pseudo-count intuition.)
7. What are the asymptotic properties of MLE? (Consistent, asymp. normal, efficient, invariant.)
8. MAP vs Bayesian inference? (MAP gives a point estimate (mode); Bayesian gives full posterior + uncertainty; cost increases.)

10. Drill plan

• Derive MLE for: Bernoulli, Gaussian, Poisson, multinomial, exponential. 5 minutes each.
• Derive MAP for: linear regression with Gaussian prior (ridge), with Laplace prior (lasso).
• Derive Beta-Bernoulli posterior. Recite mean and mode.
• Recognize: OLS = MLE Gaussian, ridge = MAP Gaussian, lasso = MAP Laplace, cross-entropy = MLE general.
• Practice writing log-likelihoods cleanly without dropping constants until the end.

11. Further reading

• Murphy, Machine Learning: A Probabilistic Perspective, ch. 4–5.
• Bishop, Pattern Recognition and Machine Learning, ch. 1, 2.
• Wasserman, All of Statistics, ch. 9, 11.
• MacKay, Information Theory, Inference, and Learning Algorithms — beautiful Bayesian framing.