MLE and MAP Estimation

Hello! Welcome to the final lesson in our module on Mathematical and Statistical Foundations.

Introduction

In our last lesson, we explored the Law of Large Numbers and the Central Limit Theorem. These powerful theorems give us confidence that statistics calculated from a data sample (like the sample mean) are reliable estimators of the true, underlying population parameters. The LoLN assures us that our estimate gets better with more data, and the CLT tells us about the distribution of the estimation error.

This raises a crucial question: How do we find the "best" parameters for a chosen probability distribution to explain our data? This lesson directly addresses this by covering the learning outcome: Perform maximum likelihood estimation (MLE) and maximum a posteriori (MAP) estimation.

We will explore two fundamental principles for parameter estimation:

1. Maximum Likelihood Estimation (MLE): This method asks, "What parameter values make the observed data most probable?" It is a cornerstone of frequentist statistics and implicitly underlies many machine learning loss functions.
2. Maximum a Posteriori (MAP) Estimation: This is the Bayesian counterpart. It asks, "Given the data and my prior beliefs about the parameters, what are the most probable parameter values?" This approach allows us to incorporate prior knowledge, which can be incredibly powerful, especially with limited data.

Understanding MLE and MAP is not just a statistical exercise; it provides the theoretical justification for the objective functions we aim to optimize when training machine learning models.

1. The Core Idea: What Parameter Fits Best?

Imagine you have a set of data points. You hypothesize that they were generated from a Gaussian (normal) distribution, but you don't know the distribution's mean $\mu$ or variance ${\sigma }^2$. Which Gaussian distribution is the most "likely" one to have produced your data?

MLE provides a formal way to answer this. It directs us to choose the parameters ($\mu$, ${\sigma }^2$) that maximize the probability (or more accurately, the probability density) of observing our specific data points.

Let's watch a video that builds a fantastic visual intuition for this process.

The key idea is that for each possible value of the parameter, we get a different probability distribution for the data. We then "reverse" the question: for the data we actually did observe, which parameter value makes that observation have the highest probability?

2. Maximum Likelihood Estimation (MLE)

Now, let's formalize the intuition from the video.

The Likelihood Function

Let's say we have a dataset $D=\{x_1,x_2,\dots ,x_n\}$ of $n$ data points. We assume these points are independent and identically distributed (i.i.d.) and drawn from a probability distribution $p(x|\theta )$ parameterized by $\theta$.

Because the data points are independent, the joint probability of observing the entire dataset is the product of the individual probabilities:
$P(D|\theta )=P(x_1,x_2,\dots ,x_n|\theta )={\prod }_{i=1}^{n}p(x_i|\theta )$
This expression, when viewed as a function of the parameter $\theta$ for a fixed dataset $D$, is called the Likelihood Function, $L(\theta |D)$.

The MLE principle states that the best estimate for $\theta$ is the one that maximizes this likelihood function:
${\hat{\theta }}_{MLE}=\arg {\max }_{\theta }L(\theta |D)=\arg {\max }_{\theta }{\prod }_{i=1}^{n}p(x_i|\theta )$

The Log-Likelihood Trick

Working with products is mathematically cumbersome. Since the logarithm is a monotonically increasing function, maximizing a function is equivalent to maximizing its logarithm. This gives rise to the log-likelihood function, $\ell (\theta |D)$, which is much easier to work with.

${\hat{\theta }}_{MLE}=\arg {\max }_{\theta }\ln (L(\theta |D))=\arg {\max }_{\theta }{\sum }_{i=1}^n\ln p(x_i|\theta )$
This simple trick turns a product into a sum, which is far easier to differentiate—a key step in optimization. Your computer science background will appreciate that this also avoids numerical underflow that can occur when multiplying many small probabilities together.

The following video provides a more formal introduction to the likelihood function and a clear explanation of why we use the log-likelihood.

Deriving an MLE: The Coin Flip Example

The classic example for MLE is estimating the bias of a coin. Let's say we flip a coin $n$ times and observe ${\alpha }_1$ heads (outcome 1) and ${\alpha }_0$ tails (outcome 0). We want to estimate $\theta =P(X=1)$. This is a Bernoulli distribution.

The following reading from Tom Mitchell's renowned machine learning book provides a rigorous yet clear derivation.

As the text derives, the process is:

1. Likelihood: $L(\theta |D)=P(D|\theta )={\theta }^{{\alpha }_1}(1-\theta {)}^{{\alpha }_0}$
2. Log-Likelihood: $\ell (\theta |D)={\alpha }_1\ln \theta +{\alpha }_0\ln (1-\theta )$
3. Differentiate: $\frac{\partial \ell }{\partial \theta }=\frac{{\alpha }_1}{\theta }-\frac{{\alpha }_0}{1-\theta }$
4. Set to zero and solve: $\frac{{\alpha }_1}{\theta }=\frac{{\alpha }_0}{1-\theta }⟹{\alpha }_1(1-\theta )={\alpha }_0\theta ⟹{\hat{\theta }}_{MLE}=\frac{{\alpha }_1}{{\alpha }_1+{\alpha }_0}$

The result is exactly what our intuition would suggest: the best estimate for the probability of heads is the proportion of heads observed in the data. MLE gives us a formal justification for this intuitive answer.

3. Maximum a Posteriori (MAP) Estimation

MLE works well with lots of data. But what if our data is scarce?
Imagine you flip a new coin 3 times and get 3 heads.
The MLE estimate is ${\hat{\theta }}_{MLE}=3/3=1.0$.
This suggests the coin will always land on heads, which seems like an overconfident conclusion based on such little evidence. We have a prior belief that coins are usually fair, or at least close to fair.

MAP estimation allows us to incorporate such prior beliefs.

From Likelihood to Posterior

MAP shifts the goal from maximizing the likelihood $P(D|\theta )$ to maximizing the posterior probability $P(\theta |D)$. We use Bayes' Theorem to define the posterior:

$P(\theta |D)=\frac{P(D|\theta )P(\theta )}{P(D)}$

Here:

• $P(\theta |D)$ is the posterior: what we believe about $\theta$ after seeing the data.
• $P(D|\theta )$ is the likelihood: same as in MLE.
• $P(\theta )$ is the prior: our belief about $\theta$ before seeing any data.
• $P(D)$ is the evidence: the probability of the data.

The MAP estimate is the $\theta$ that maximizes this posterior:
${\hat{\theta }}_{MAP}=\arg {\max }_{\theta }P(\theta |D)=\arg {\max }_{\theta }\frac{P(D|\theta )P(\theta )}{P(D)}$
Since $P(D)$ is constant with respect to $\theta$, it doesn't affect the location of the maximum. We can therefore ignore it for optimization:
${\hat{\theta }}_{MAP}=\arg {\max }_{\theta }P(D|\theta )P(\theta )$
In log space, this becomes:
${\hat{\theta }}_{MAP}=\arg {\max }_{\theta }\left(\ln P(D|\theta )+\ln P(\theta )\right)$

This final expression is insightful: MAP estimate = MLE term + Log-Prior term. The prior acts as a "penalty" or a "regularizer" that pulls the estimate away from the pure MLE solution towards our prior belief.

Let's return to the video for a visual explanation of how the prior influences the estimate.

Deriving a MAP Estimate: Coin Flip with a Beta Prior

To perform MAP estimation, we must first define a prior distribution $P(\theta )$. For the coin flip example, $\theta$ is a probability between 0 and 1. A flexible distribution for this is the Beta distribution.

The Beta distribution, $P(\theta )=\text{Beta}(\theta |{\beta }_1,{\beta }_0)$, is parameterized by two hyperparameters, ${\beta }_1$ and ${\beta }_0$, which can be interpreted as representing prior knowledge equivalent to having seen ${\beta }_1-1$ "imaginary" heads and ${\beta }_0-1$ "imaginary" tails.

The Beta distribution is a conjugate prior to the Bernoulli likelihood. This is a special relationship: if the prior is Beta and the likelihood is Bernoulli, the resulting posterior is also a Beta distribution. This makes the math very convenient.

The same reading from Tom Mitchell's book now walks us through the MAP derivation.

The derived MAP estimate is:
${\hat{\theta }}_{MAP}=\frac{{\alpha }_1+{\beta }_1-1}{({\alpha }_1+{\beta }_1-1)+({\alpha }_0+{\beta }_0-1)}$
Going back to our 3-flips-3-heads example:

• ${\alpha }_1=3,{\alpha }_0=0$.
• ${\hat{\theta }}_{MLE}=3/3=1$.
• Let's assume a prior belief that the coin is fair. We can represent this with ${\beta }_1=2,{\beta }_0=2$ (representing 1 imaginary head and 1 imaginary tail).
• ${\hat{\theta }}_{MAP}=\frac{3+2-1}{(3+2-1)+(0+2-1)}=\frac{4}{4+1}=0.8$.

The MAP estimate is much more reasonable than the MLE of 1.0. As we collect more data, the observed counts ${\alpha }_1$ and ${\alpha }_0$ will dwarf the small prior counts, and the MAP estimate will converge to the MLE estimate.

4. Connection to Machine Learning

The principles of MLE and MAP are central to training machine learning models.

• Loss Functions from MLE: Many common loss functions are simply the negative log-likelihood of the model.

  • Mean Squared Error (MSE) for regression is the negative log-likelihood of the data assuming the target variable is the model's prediction plus some Gaussian noise.
  • Cross-Entropy Loss for classification is the negative log-likelihood assuming the data follows a Bernoulli or Categorical distribution.
  • Therefore, minimizing these loss functions is equivalent to performing Maximum Likelihood Estimation of the model's parameters.

• Regularization from MAP: When you add a regularization term to your loss function, you are often implicitly performing MAP estimation.

  • L2 Regularization (e.g., loss + λ||w||²) is equivalent to MAP estimation where you've placed a Gaussian prior with mean 0 on the model's weights w. This penalizes large weights, assuming a priori that most weights should be small.
  • L1 Regularization (e.g., loss + λ||w||₁) is equivalent to MAP estimation with a Laplace prior on the weights. This prior is sharply peaked at zero, encouraging many weights to become exactly zero (sparsity).

This provides a powerful probabilistic interpretation for techniques you'll frequently use in practice.

Conclusion

This lesson completes our foundational module by providing the principles for fitting models to data.

Key Takeaways:

• Maximum Likelihood Estimation (MLE) finds the parameter $\theta$ that maximizes the likelihood of the observed data, $P(D|\theta )$. It's a powerful principle but can overfit on small datasets.
• Maximum a Posteriori (MAP) Estimation finds the $\theta$ that maximizes the posterior probability $P(\theta |D)$. It incorporates a prior belief $P(\theta )$ and is equivalent to adding a regularization term to the likelihood.
• The log-likelihood is a practical tool that turns products into sums, simplifying the math and improving numerical stability.
• Minimizing standard loss functions (like MSE or Cross-Entropy) is equivalent to MLE. Adding regularization (like L1 or L2) is equivalent to MAP.

Preview of the next lesson:
We have now established the statistical principles for what we want to optimize (the likelihood or the posterior). In the next module, "Core Machine Learning Concepts," we will dive into how we perform this optimization for complex models where we can't simply solve for the parameters analytically. We will begin with Gradient Descent, the fundamental algorithm used to train almost all modern neural networks by iteratively minimizing a loss function.