Measure Theory and the Strong Law of Large Numbers

Hello! Welcome to the first lesson of Module 3.

Introduction

In our last lesson, we explored the Dominated Convergence Theorem (DCT), a major payoff from Lebesgue integration theory. We saw that it provides rigorous conditions for a crucial operation in statistics: swapping limits and expectations. This ability to handle limits of random variables is a cornerstone of modern probability.

Today, we will use this foundation to examine one of the most fundamental theorems in all of statistics: the Strong Law of Large Numbers (SLLN). You have certainly used this law's results in your econometrics and statistics work, often to justify why sample averages are good estimators of population means. Our goal today is to go deeper and address the learning outcome: Explain how the Strong Law of Large Numbers relies on measure theory for its precise statement and proof.

We will dissect the SLLN's formal statement to see why it couldn't even be articulated without the language of measure theory. Then, we'll look at the key ingredients of its proof, revealing the measure-theoretic machinery at work.

1. The Weak vs. The Strong Law of Large Numbers

The intuitive idea behind the Law of Large Numbers (LLN) is that as you collect more and more data, the average of your sample gets closer to the true average of the underlying population. However, the mathematical formalization of "gets closer" is subtle and leads to two different versions of the law.

Let's start by getting a clear statement of the SLLN and how it differs from its "weak" counterpart.

To solidify the distinction, let's watch one more short video that focuses directly on the two types of convergence.

Let's summarize the distinction, using ${\bar{X}}_n$ for the sample mean of $n$ i.i.d. random variables and $\mu$ for the population mean.

• The Weak Law of Large Numbers (WLLN): States that ${\bar{X}}_n$ converges to $\mu$ in probability.
  ${\lim }_{n\to \infty }P(|{\bar{X}}_n-\mu |>ϵ)=0\text{for any }ϵ>0$
  This means that for a large sample size $n$, it is very unlikely that ${\bar{X}}_n$ will be far from $\mu$. However, it does not rule out the possibility that for some even larger $m>n$, the sample mean ${\bar{X}}_m$ might deviate significantly. These large deviations just become increasingly rare as the sample size grows.

• The Strong Law of Large Numbers (SLLN): States that ${\bar{X}}_n$ converges to $\mu$ almost surely (or with probability 1).
  $P\left({\lim }_{n\to \infty }{\bar{X}}_n=\mu \right)=1$
  This is a much more powerful statement. It concerns the entire, infinite sequence of sample means. It says that the set of all possible outcomes (infinite sequences of random variable draws) for which the sequence of sample means does not converge to $\mu$ has a total probability of zero.

The SLLN implies the WLLN, but the reverse is not true. The "strong" law makes a statement about the long-run behavior of a single path of realizations, while the "weak" law makes a statement about the distribution of ${\bar{X}}_n$ at a specific (large) point in time $n$.

2. The Measure-Theoretic Statement

Now, let's focus on the expression $P({\lim }_{n\to \infty }{\bar{X}}_n=\mu )=1$. How does measure theory enable us to even make sense of this?

The event "${\lim }_{n\to \infty }{\bar{X}}_n=\mu$" is the set of all outcomes $\omega$ in our sample space $\Omega$ such that the sequence of real numbers ${\bar{X}}_n(\omega )$ converges to $\mu$. From real analysis, the definition of a limit is:
$\forall ϵ>0,\exists N\text{ such that }\forall n\ge N,|{\bar{X}}_n(\omega )-\mu |<ϵ$
This can be rewritten using countable values for $ϵ$ (e.g., $1/m$ for $m\in \mathbb{N}$):
${\bigcap }_{m=1}^{\infty }{\bigcup }_{N=1}^{\infty }{\bigcap }_{n=N}^{\infty }\{\omega \in \Omega :|{\bar{X}}_n(\omega )-\mu |<1/m\}$
This expression looks complicated, but it reveals something crucial. The event we are interested in is constructed from a countable number of set operations (intersections and unions) on simpler, "basic" events like $\{\omega :|{\bar{X}}_n(\omega )-\mu |<1/m\}$.

This is exactly what a σ-algebra is for. As we discussed in Module 2, a σ-algebra is a collection of subsets of $\Omega$ that is closed under countable unions, countable intersections, and complements. For the SLLN statement to be meaningful, the complex event "${\lim }_{n\to \infty }{\bar{X}}_n=\mu$" must be an element of our σ-algebra $\mathcal{F}$, so that we can assign a probability to it with our measure $P$. Without this formal structure, we would have no basis for assigning a probability to an event defined over an infinite sequence of random variables.

The following reading from Terence Tao's blog makes this connection explicit.

As Tao states, "almost sure convergence" is simply the specialization of the measure-theoretic concept of "pointwise convergence almost everywhere" to the context of probability theory. Similarly, "convergence in probability" is the specialization of "convergence in measure". This is the direct link you were looking for: the precise language needed to state the SLLN is borrowed directly from measure theory.

3. The Measure-Theoretic Proof

Having seen that the statement of the SLLN is measure-theoretic, it's no surprise that its proof also relies on tools from measure theory. The key tool is a result called the Borel-Cantelli Lemma.

Let's walk through the logic of the proof you just read:

1. Goal: We want to show that ${\bar{X}}_n\to \mu$ almost surely. This is equivalent to showing that for any $ϵ>0$, the event $|{\bar{X}}_n-\mu |>ϵ$ happens for only a finite number of $n$, with probability 1.
2. Strategy: Let's define the "bad" events $A_n=\{|{\bar{X}}_n-\mu |>ϵ\}$. The Borel-Cantelli lemma gives us a way to prove that only finitely many $A_n$ occur. We need to show that the sum of their probabilities is finite: ${\sum }_{n=1}^{\infty }P(A_n)<\infty$.
3. Execution:
   • The proof uses a clever trick. Instead of bounding $P(A_n)$ directly, it bounds the probability of a related event using the fourth moment, $E[({\bar{X}}_n-\mu {)}^4]$.
   • Using Markov's inequality, it establishes a bound on $P(A_n)$ that looks like $K/n^2$ for some constant $K$. (The specific details of the moment calculation are less important than the overall structure of the proof).
   • It then considers the sum ${\sum }_{n=1}^{\infty }P(A_n)$. Since ${\sum }_{n=1}^{\infty }K/n^2$ is a convergent p-series, we have $\sum P(A_n)<\infty$.
4. Conclusion: The condition for the Borel-Cantelli lemma is met. Therefore, $P(A_n\text{ occur infinitely often})=0$. This means, with probability 1, the "bad events" only happen a finite number of times. After the last "bad event" occurs, we have $|{\bar{X}}_n-\mu |\le ϵ$ for all subsequent $n$. Since this holds for any $ϵ>0$, it proves that ${\bar{X}}_n\to \mu$ almost surely.

This line of reasoning—defining a sequence of events and using a limit theorem like Borel-Cantelli to make a statement about their infinite behavior—is a hallmark of measure-theoretic probability.

An Aside: Connection to Ergodic Theory

Given your background in time series and stochastic systems, you might find this connection interesting. The sequence of i.i.d. random variables $(X_n)$ can be seen as a simple "stationary process." From this perspective, the SLLN is a special case of a more general theorem in measure theory called the Pointwise Ergodic Theorem. This theorem deals with the convergence of time averages to "space averages" for measure-preserving dynamical systems. This is a deep and beautiful connection, which Terence Tao mentions in his post (5c82f, part 3, Remark 4). It shows how the SLLN is one of the first entry points into the rich field of ergodic theory, which has applications everywhere from statistical mechanics to financial modeling.

Conclusion

Let's consolidate what we've learned about the SLLN's reliance on measure theory.

• Precise Statement: The SLLN's statement, $P({\lim }_{n\to \infty }{\bar{X}}_n=\mu )=1$, involves an event defined over an infinite sequence of random variables. The machinery of σ-algebras is required to ensure this event is well-defined and can be assigned a probability. The concept of almost sure convergence is the probabilistic term for the measure-theoretic idea of pointwise convergence almost everywhere.
• Rigorous Proof: The proof of the SLLN requires tools designed to handle infinite sequences of events. The Borel-Cantelli Lemma is a prime example, allowing us to conclude that something happens "almost surely" by summing the probabilities of an infinite series of "bad" events and showing the sum is finite.
• Necessary Conditions: The SLLN requires that the first moment of the random variables is finite, i.e., $\mathbb{E}[|X|]<\infty$. As we've touched on before, the very definition of expectation for variables that can take both positive and negative values relies on the Lebesgue integral and the condition of absolute integrability.

In essence, while the intuition of the SLLN is simple, making it mathematically rigorous forces us to use the full power of the measure-theoretic framework we have been building.

Preview of the Next Lesson:
We have now seen how measure theory helps formalize limits of random variables. Next, we will tackle another concept you've used extensively in econometrics: conditional expectation. We will define conditional expectation with respect to a σ-algebra. This will provide a powerful and general way to think about "the information available at a certain time," paving the way for a rigorous, measure-theoretic understanding of stochastic processes like martingales.