Bayes' Theorem: Conditional Probability in Action

Hello! Welcome to the next lesson in our journey through the mathematical foundations of AI.

In the last few lessons, we focused on calculus, culminating in our exploration of Jacobians and Hessians for advanced optimization. We saw how derivatives, in various forms, provide the machinery for models to learn and find optimal parameters. Now, we shift our focus to the second great pillar of machine learning mathematics: Probability and Statistics.

While calculus gives us the tools for optimization, probability theory provides the framework for reasoning under uncertainty, which is at the heart of nearly every AI task. Today, we'll start with one of its most essential and powerful concepts.

This lesson addresses the learning outcome: Apply Bayes' theorem to solve problems involving conditional probability. We will explore the intuition behind the theorem, formally define it, and then apply it to practical problems, seeing how it allows us to systematically update our beliefs in the face of new evidence.

1. The Intuition: Why Evidence Doesn't Tell the Whole Story

Human intuition is notoriously poor at correctly weighing new evidence against existing knowledge. We often overemphasize the new information and neglect the background context. This is precisely the problem that Bayes' theorem solves.

To build an intuition for this, let's start with a fantastic video from 3Blue1Brown that uses a classic psychology experiment to illustrate this common flaw in our reasoning.

Bayes theorem, the geometry of changing beliefs

This video, titled 'Bayes theorem, the geometry of changing beliefs', will show you why simply looking at how well evidence fits a hypothesis is not enough. It introduces the crucial role of the 'prior' probability.

Watch from 01:04 to 04:46. Pay close attention to the 'Steve the librarian vs. farmer' example and how thinking in terms of a representative sample of people helps correct our initial, flawed intuition.

The key takeaway from the video's example is that new evidence (Steve's personality description) shouldn't replace our knowledge; it should update it. The fact that there are far more farmers than librarians is a crucial piece of information—a prior belief—that must be factored into our conclusion. Bayes' theorem gives us the formal mechanism to do this correctly.

2. Defining Bayes' Theorem

Bayes' theorem is a direct consequence of the definition of conditional probability. Recall that the probability of event A happening given that event B has already happened is:
$P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )}$
Where $P (A \cap B)$ is the probability of both A and B happening.

Since $P (A \cap B)$ is the same as $P (B \cap A)$ , we can also write:
$P (B ∣ A) = \frac{P ( B \cap A )}{P ( A )} ⟹ P (B \cap A) = P (B ∣ A) P (A)$
If we substitute this back into the first equation, we get:
$P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}$
This simple but profound equation is Bayes' theorem.

It's most often used in the context of a Hypothesis (H) and Evidence (E):
$P (H ∣ E) = \frac{P ( E ∣ H ) P ( H )}{P ( E )}$

Each term has a special name:

Posterior Probability $P (H ∣ E)$ : The updated probability of the hypothesis after considering the evidence. This is what we usually want to calculate.
Likelihood $P (E ∣ H)$ : The probability of observing the evidence if the hypothesis were true.
Prior Probability $P (H)$ : The initial probability of the hypothesis before considering the evidence.
Marginal Likelihood (or Evidence) $P (E)$ : The total probability of observing the evidence, regardless of the hypothesis.

Bayes' Theorem Formula with Labeled Components — This image breaks down the components of Bayes' theorem. It clearly labels the posterior, likelihood, prior, and the evidence (marginalization) term, providing a quick reference for the structure of the formula.

Bayes' Theorem in Machine Learning — In a machine learning context, Bayes' theorem allows us to update our belief about model parameters (w) given the observed data (D). The posterior P(w|D) is proportional to the likelihood P(D|w) times the prior P(w).

3. Calculating The Evidence: The Law of Total Probability

The trickiest part of the formula is often the denominator, $P (E)$ . How do we find the overall probability of the evidence? We use the Law of Total Probability. It allows us to calculate $P (E)$ by considering all possible scenarios.

For a hypothesis $H$ that can be either true or false (noted as $H^{c}$ ), the law states:
$P (E) = P (E ∣ H) P (H) + P (E ∣ H^{c}) P (H^{c})$
This means the total probability of the evidence is the sum of:

The probability of seeing the evidence when the hypothesis is true, weighted by the prior probability of the hypothesis being true.
The probability of seeing the evidence when the hypothesis is false, weighted by the prior probability of the hypothesis being false.

Substituting this into Bayes' theorem gives us its expanded form, which is what we most often use for calculations:
$P (H ∣ E) = \frac{P ( E ∣ H ) P ( H )}{P ( E ∣ H ) P ( H ) + P ( E ∣ H ^{c} ) P ( H ^{c} )}$

To see this derivation and the terminology explained again with a clear, practical focus, the following video is very helpful.

Bayes' Theorem (with Example!)

In this video, 'Bayes' Theorem (with Example!)', Steve Brunton provides a concise derivation and clearly labels each part of the formula in the context of inverse problems, which is highly relevant for engineering and machine learning.

Watch from the beginning to 08:35. This segment covers the derivation of the theorem, defines the posterior, prior, and update terms, and explains the law of total probability to expand the denominator.

4. Application 1: Medical Diagnosis

Medical testing is the quintessential example for applying Bayes' theorem. It powerfully illustrates how a test's accuracy metrics can be misleading without considering the base rate (prior probability) of the disease.

Let's work through a detailed example.

An introduction to Bayes' rule – Chapter 1

This chapter from the book 'Bayes' rule: A tutorial introduction to Bayesian analysis' provides an excellent, in-depth walkthrough of a medical diagnosis scenario. It carefully distinguishes between the probability of symptoms given a disease and the probability of a disease given symptoms.

Please read Section 1.1, 'Example 1: Poxy diseases'. This will take you from the patient's perspective to the doctor's, introducing all the key terms (likelihood, prior, posterior, MLE, MAP) with concrete numbers.

Let's summarize the key calculation from the Smallpox vs. Chickenpox example:

Given Information:

Likelihoods:
- $P (spots ∣ smallpox) = 0.9$
- $P (spots ∣ chickenpox) = 0.8$
Priors:
- $P (smallpox) = 0.001$ (very rare)
- $P (chickenpox) = 0.1$ (common)
Evidence: $P (spots)$ was given as 0.081. Let's see how it's calculated using the law of total probability (assuming only these two diseases cause spots):
$P (spots) = P (spots ∣ smallpox) P (smallpox) + P (spots ∣ chickenpox) P (chickenpox)$
$P (spots) = (0.9 \times 0.001) + (0.8 \times 0.1) = 0.0009 + 0.08 = 0.0809$ (The text used 0.081 for neatness).

Question: What is the probability of having smallpox, given you have spots? $P (smallpox ∣ spots)$

Calculation:
$P (smallpox ∣ spots) = \frac{P ( spots ∣ smallpox ) P ( smallpox )}{P ( spots )} = \frac{0.9 \times 0.001}{0.0809} = \frac{0.0009}{0.0809} \approx 0.011$
So, there is only a 1.1% chance you have smallpox, even though the symptoms are a 90% match! Your prior belief (that smallpox is extremely rare) dominates the outcome.

Test your understanding!

Let's try a new scenario based on the cancer screening example from the Steve Brunton video.
A certain disease affects 1 in 10,000 people ( $P (D) = 0.0001$ ). A test for it has 98% accuracy: it's 98% sensitive ( $P (positive ∣ D) = 0.98$ ) and 98% specific ( $P (negative ∣ not D) = 0.98$ ).

If a randomly selected person tests positive, what is the probability they actually have the disease?

Show answer

We want to find $P (D ∣ positive)$ .

1. Identify the terms:

Likelihood $P (positive ∣ D) = 0.98$
Prior $P (D) = 0.0001$
Probability of a false positive: $P (positive ∣ not D) = 1 - P (negative ∣ not D) = 1 - 0.98 = 0.02$
Prior of not having the disease: $P (not D) = 1 - 0.0001 = 0.9999$

2. Calculate the total evidence $P (positive)$ :
$P (positive) = P (positive ∣ D) P (D) + P (positive ∣ not D) P (not D)$
$P (positive) = (0.98 \times 0.0001) + (0.02 \times 0.9999)$
$P (positive) = 0.000098 + 0.019998 = 0.020096$

3. Apply Bayes' theorem:
$P (D ∣ positive) = \frac{P ( positive ∣ D ) P ( D )}{P ( positive )} = \frac{0.000098}{0.020096} \approx 0.004877$

So, even with a positive result from a 98% accurate test, there is less than a 0.5% chance you have the disease! This happens because the disease is so rare that the vast majority of positive tests are false positives from the large pool of healthy people.

5. Application 2: Spam Email Classification

Bayes' theorem isn't just for medical diagnosis; it's the foundation of some of the earliest successful AI applications, like spam filtering. Given your background in computer science, this example will resonate. The core idea is to calculate the probability that an email is spam given that it contains certain keywords.

Spam Email Classification and Medical Diagnosis with Multiple Tests

This article, 'Probability Beginner to Advanced for Data Science Part 1', provides a concise and well-explained example of using Bayes' theorem for spam classification. It also includes an interesting extension showing how to update beliefs with multiple pieces of evidence (multiple tests).

Read 'Example 19: Spam Email Classification' and 'Example 20: Medical Diagnosis with Multiple Tests'. Focus on how the final probability is calculated in each case.

This application forms the basis of the Naive Bayes classifier, an algorithm we will implement later in the course. It's called "naive" because it assumes the keywords (the evidence) are independent of each other, which simplifies the math considerably. Despite this simplification, it works remarkably well in practice.

Conclusion

In this lesson, we have taken our first major step into probabilistic reasoning for AI. We've seen that Bayes' theorem is not just a formula, but a fundamental principle for thinking rationally under uncertainty.

Key Takeaways:

Bayes' theorem provides a rigorous method for updating our beliefs (posterior probability) based on new evidence (via the likelihood) and our initial assumptions (prior probability).
The formula is $P (H ∣ E) = \frac{P ( E ∣ H ) P ( H )}{P ( E )}$ .
Human intuition often fails by overweighting the likelihood and ignoring the prior. Rare events are still rare, even with evidence that points toward them.
The denominator $P (E)$ acts as a normalization constant and is calculated using the Law of Total Probability, summing over all possible hypotheses.
This theorem has direct applications in fields like medical diagnosis and is a foundational concept for many machine learning algorithms, such as the Naive Bayes classifier.

Preview of the next lesson:
To effectively use Bayes' theorem, we need a way to model our prior beliefs and likelihoods. What mathematical objects can we use to represent $P (H)$ or $P (E ∣ H)$ ? The answer is probability distributions. In our next lesson, we will describe common probability distributions (Gaussian, Bernoulli, Categorical) and their applications, which are the essential building blocks for constructing probabilistic models in AI.