Large Numbers, Central Limit Theorem, and Statistical Justification

Hello! Welcome to the next lesson in our exploration of the mathematical foundations of AI.

In our previous lesson, we defined and learned to compute the expected value $E[X]$ and variance $Var(X)$ of a random variable. We drew a crucial distinction between the theoretical mean of a distribution ($E[X]$) and the sample mean ($\bar{x}$) calculated from data. You might have wondered: how can we be sure that the sample mean we calculate from our dataset is a good estimate of the true, underlying theoretical mean? And how much might that sample mean vary from the true mean?

This lesson will answer those questions by tackling the learning outcome: Apply the law of large numbers and the central limit theorem to justify statistical methods. These two theorems are cornerstones of probability and statistics, providing the essential link between theoretical distributions and real-world data samples.

• The Law of Large Numbers (LoLN) will give us confidence that as we collect more data, our sample average will converge to the true theoretical average.
• The Central Limit Theorem (CLT) will go a step further, describing the shape of the distribution of the sample average, which remarkably, is almost always a normal distribution.

Understanding these theorems is fundamental to justifying why many machine learning techniques work, such as training models on batches of data or using statistical tests to evaluate results.

1. Building Intuition: The Emergence of the Bell Curve

Before diving into formal definitions and proofs, let's build a strong visual intuition for what happens when we repeatedly sample and sum random variables. The 3Blue1Brown channel is famous for its excellent visualizations, and their video on the Central Limit Theorem is a perfect starting point.

The key insight from this video is that the process of summing independent and identically-distributed (i.i.d.) random variables has a "normalizing" effect. The specific details of the original distribution get "washed away," and a universal bell curve emerges. The video also quantitatively showed that if you sum $n$ variables, the new mean is $n\cdot \mu$ and the new standard deviation is $\sqrt{n}\cdot \sigma$. This $\sqrt{n}$ factor is a critical piece of the puzzle.

2. The Law of Large Numbers (LoLN)

The Law of Large Numbers is perhaps the more intuitive of the two theorems. It formalizes the idea that the average of results from a large number of trials should be close to the expected value.

Let's watch a segment from a Harvard statistics lecture that defines the LoLN and distinguishes between its "weak" and "strong" forms.

Formal Statement and Proof

As the video explained, the LoLN states that the sample mean ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$ converges to the true mean $\mu =E[X]$ as $n\to \infty$.

There are two main versions:

• Strong Law of Large Numbers: States that ${\bar{X}}_n$ converges to $\mu$ with probability 1. This is the pointwise convergence mentioned in the video.
• Weak Law of Large Numbers: States that for any small positive number $ϵ$, the probability that $|{\bar{X}}_n-\mu |>ϵ$ goes to 0 as $n\to \infty$. This is called convergence in probability.

The proof of the weak law is surprisingly straightforward and elegantly builds on concepts we already know. It uses Chebyshev's Inequality, which relates the probability of a random variable being far from its mean to its variance.

Let's quickly find the variance of the sample mean, ${\bar{X}}_n$:
$Var({\bar{X}}_n)=Var\left(\frac{1}{n}{\sum }_{i=1}^n{X}_i\right)$
Using the properties of variance from our last lesson, $Var(aX)=a^{2}Var(X)$ and assuming the $X_i$ are independent:
$Var({\bar{X}}_n)=\frac{1}{n^2}Var\left({\sum }_{i=1}^n{X}_i\right)=\frac{1}{n^2}{\sum }_{i=1}^{n}Var(X_i)=\frac{1}{n^2}(n{\sigma }^2)=\frac{{\sigma }^2}{n}$
The variance of the sample mean shrinks as $n$ increases. Now, applying Chebyshev's inequality to ${\bar{X}}_n$:
$P(|{\bar{X}}_n-\mu |\ge ϵ)\le \frac{Var({\bar{X}}_n)}{{ϵ}^2}=\frac{{\sigma }^2}{n{ϵ}^2}$
As $n\to \infty$, the right side of the inequality $\frac{{\sigma }^2}{n{ϵ}^2}$ goes to 0. Therefore, the probability on the left side must also go to 0. This completes the proof of the Weak LoLN.

Justification for Statistical Methods

The LoLN is the theoretical bedrock for why we can use samples to learn about populations.

• Monte Carlo Methods: When we estimate a quantity by simulating a random process many times and averaging the results, we are relying on the LoLN. For example, estimating $\pi$ by randomly throwing darts at a square.
• Machine Learning Training: When you train a neural network, you often compute the loss on a "mini-batch" of data. The LoLN gives us confidence that the average loss over a sufficiently large batch is a good approximation of the true expected loss over the entire (potentially infinite) data distribution. This justifies using the batch gradient to update our model weights.

3. The Central Limit Theorem (CLT)

The LoLN tells us that the sample mean ${\bar{X}}_n$ hones in on the true mean $\mu$. The Central Limit Theorem tells us how it hones in—it describes the distribution of the error $({\bar{X}}_n-\mu )$.

The CLT states that for a large $n$, the distribution of the sample mean ${\bar{X}}_n$ is approximately normal, with a mean of $\mu$ and a variance of ${\sigma }^2/n$.

${\bar{X}}_n\approx N(\mu ,\frac{{\sigma }^2}{n})$

This is astounding because it holds true regardless of the original distribution of the $X_i$'s, as long as they are i.i.d. and have a finite variance. This universality is why the normal distribution is so ubiquitous in nature and statistics.

To work with this in a standard way, we standardize the variable ${\bar{X}}_n$ by subtracting its mean and dividing by its standard deviation:

$Z_n=\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}$

The CLT states that as $n\to \infty$, the distribution of $Z_n$ converges to the standard normal distribution, $N(0,1)$.

Justification for Statistical Methods

The CLT is one of the most powerful tools for justifying statistical procedures.

Here are the key applications:

1. Hypothesis Testing: Many statistical tests (like the t-test) rely on the assumption that the sample means are normally distributed. The CLT provides the justification for this assumption when the sample size is large enough.
2. Confidence Intervals: The CLT allows us to construct confidence intervals for population parameters. As seen in the polling example, the $\pm 1/\sqrt{n}$ margin of error is a direct consequence of the CLT and the properties of the normal distribution (where ~95% of the data lies within ~2 standard deviations).
3. Approximations: For distributions that are difficult to work with directly (like the binomial distribution with large $n$), the CLT allows us to use the much simpler normal distribution as an approximation. This was historically critical before modern computing power, and the principle remains vital.

Conclusion

In this lesson, we bridged the gap between theoretical probability distributions and practical data analysis.

Key Takeaways:

• Law of Large Numbers (LoLN): Guarantees that the sample mean ${\bar{X}}_n$ converges to the true population mean $\mu$ as the sample size $n$ grows. This justifies using sample averages as estimates for true averages. The variance of the sample mean, ${\sigma }^2/n$, shrinks to zero.
• Central Limit Theorem (CLT): Describes the distribution of the sample mean. For large $n$, the distribution of ${\bar{X}}_n$ is approximately normal, $N(\mu ,{\sigma }^2/n)$, regardless of the underlying distribution of the data.
• The $\sqrt{n}$ Scaling: This is the key insight from the CLT. The error of the sample mean decreases proportionally to $\sigma /\sqrt{n}$. To halve the error, you need to quadruple the sample size.
• Justification: Together, these theorems justify fundamental practices in statistics and machine learning, from polling and hypothesis testing to training models on data samples.
• Assumptions: Remember the crucial assumptions for the basic CLT: the samples must be independent and identically-distributed (i.i.d.) and have a finite variance.

Preview of the next lesson:
Now that we have the LoLN and CLT, we can be confident that statistics calculated from a sample (like the mean) are meaningful estimators of the true population parameters. In the next lesson, we will explore methods to find the "best" estimates for these parameters. We will dive into Maximum Likelihood Estimation (MLE) and Maximum a Posteriori (MAP) Estimation, which are powerful techniques for fitting probability distributions to data and form the basis for how many machine learning models are trained.