Implementing Naive Bayes Classifiers

Hello! Welcome back to our course on AI theory and models.

Introduction

In the last lesson, we implemented logistic regression, a powerful algorithm for binary classification. We saw that it is a discriminative model: it learns a direct mapping from inputs $\mathbf{x}$ to a class label $y$ by finding a decision boundary, without trying to understand the underlying structure of the data itself.

Today, we're shifting gears to a different family of classifiers: generative models. This lesson addresses the learning outcome: Implement Naive Bayes classifiers for probabilistic classification.

Instead of just drawing a line between classes, a generative model like Naive Bayes tries to learn the probability distribution of the data for each class. It models "what does a 'spam' email look like?" and "what does a 'not spam' email look like?". Then, when it sees a new email, it uses Bayes' theorem to determine which of these models most likely generated it.

We will cover:

1. The core principles of Naive Bayes, including Bayes' theorem and the crucial "naive" assumption of feature independence.
2. Multinomial Naive Bayes, which is excellent for text classification, and how to handle practical issues like zero probabilities with Laplace smoothing.
3. Gaussian Naive Bayes, which is used when features are continuous, and how to handle numerical stability using log-probabilities.
4. Implementing both Multinomial and Gaussian variants from scratch, leveraging your Python and mathematics background.

This lesson will not only introduce a new and efficient classification algorithm but also deepen your understanding of probabilistic modeling in machine learning.

1. The Core Idea: Probabilistic Classification with Naive Bayes

Let's start with the classic example of spam filtering. How can we calculate the probability that a message is spam, given the words it contains? This is a perfect job for Bayes' theorem, which you first encountered in Module 1.

The theorem states:
$P(\text{class}|\text{features})=\frac{P(\text{features}|\text{class})\cdot P(\text{class})}{P(\text{features})}$

In our context, we want to find the class (e.g., 'spam' or 'not spam') that has the highest probability given the features (the words in the email). So, we compare $P(\text{spam}|\text{words})$ with $P(\text{not spam}|\text{words})$ and pick the larger one.

Let's break down the terms:

• $P(\text{class}|\text{features})$ is the posterior probability we want to calculate.
• $P(\text{class})$ is the prior probability. This is our initial belief, simply the frequency of each class in the training data (e.g., the overall percentage of spam emails).
• $P(\text{features}|\text{class})$ is the likelihood. This is the probability of seeing these specific features (words) given that the message belongs to a certain class.
• $P(\text{features})$ is the evidence. Since this term is the same for all classes, it acts as a normalization constant. In classification, we can ignore it because we only care about which posterior is largest, not its exact value.

This leaves us with a simpler, proportional relationship:
$P(\text{class}|\text{features})\propto P(\text{features}|\text{class})\cdot P(\text{class})$

The "Naive" Assumption

Calculating $P(\text{features}|\text{class})$ directly is difficult. For a message with words $(w_1,w_2,...,w_n)$, it means calculating $P(w_1,w_2,...,w_n|\text{class})$. The sequence and combination of words matter, making this complex.

The Naive Bayes classifier simplifies this dramatically with a "naive" assumption: all features are conditionally independent of each other, given the class.

This means we treat a message like a "bag of words," ignoring grammar and word order. The probability of seeing the word "money" is independent of seeing the word "free," as long as we know we're in a "spam" email. This allows us to break down the likelihood into a simple product:
$P(w_1,w_2,...,w_n|\text{class})\approx P(w_1|\text{class})\cdot P(w_2|\text{class})\cdot ...\cdot P(w_n|\text{textclass})$

This is a strong and often incorrect assumption (the word "San" is clearly not independent of "Francisco"!). However, this simplification is what makes Naive Bayes so efficient, and it often performs surprisingly well. This trade-off—making a simplifying assumption (increasing bias) to achieve good practical performance (lowering variance)—is a recurring theme in machine learning.

Intuition through an Example

The following video provides an exceptionally clear, step-by-step explanation of these concepts using a spam filtering example.

2. Handling Practical Issues: Smoothing and Log Probabilities

When implementing Naive Bayes, we quickly run into two major practical problems.

The Problem of Zero Probabilities

What happens if a word in our test message never appeared in the training data for a particular class?
For example, if the word "meeting" is in a new email, but was never seen in any spam emails during training, then $P(\text{"meeting"}|\text{spam})=0$.

Because we multiply the likelihoods, this single zero will cause the entire posterior probability for the "spam" class to become zero, regardless of what other spammy words are present.

The Solution: Laplace (Add-1) Smoothing
To solve this, we use a technique called Laplace smoothing. We pretend we've seen every word in our vocabulary at least one more time than we actually did. This ensures no probability is ever zero.

The formula for the likelihood of a word $w$ given class $c$ becomes:
$P(w|c)=\frac{\text{count}(w,c)+\alpha }{\text{count}(c)+\alpha \cdot |\text{Vocabulary}|}$

• $\alpha$ is the smoothing parameter. When $\alpha =1$, it's called Add-1 smoothing.
• $|\text{Vocabulary}|$ is the number of unique words in our training set.

This technique is simple but very effective.

The Problem of Numerical Underflow

The second problem is computational. Multiplying many small probabilities (numbers between 0 and 1) can result in a number that is too small for a computer to represent accurately, a problem known as numerical underflow.

The Solution: Log Probabilities
We can solve this by working with the logarithm of the probabilities. Thanks to the property $\log (a\cdot b)=\log (a)+\log (b)$, this transforms the problematic product of probabilities into a stable sum of log-probabilities.

Our decision rule was to find the class that maximizes $P(\text{class})\cdot {\prod }_{i}P(w_i|\text{class})$. Since $\log$ is a monotonically increasing function, maximizing a value is the same as maximizing its logarithm. So, we can instead maximize:
$\log (P(\text{class}))+{\sum }_i\log (P(w_i|\text{textclass}))$
This is much more numerically stable and is the standard way to implement Naive Bayes.

3. Implementing Multinomial Naive Bayes from Scratch

Now let's see how these concepts are translated into a Python implementation for a text classification task. The following article implements Multinomial Naive Bayes for sentiment analysis on the IMDB dataset.

The code in the article demonstrates a complete, practical pipeline: preprocessing text, vectorizing it, and then applying the Naive Bayes logic we've learned. Given your background, the structure of the fit and predict methods should be quite intuitive.

Test your understanding!

In the laplace_smoothing function from the AnalyticsVidhya article, why is math.log() used? And in the predict function, why are the label_scores added together (+=) instead of multiplied?

Show answer

This is the direct implementation of using log probabilities to prevent numerical underflow.

1. math.log() is used in laplace_smoothing to calculate the log-likelihood of a word, i.e., $\log (P(w|c))$.
2. In predict, the scores are added (label_scores[l] += log_w_given_l) because adding logarithms is equivalent to multiplying the original probabilities: $\log (P(c))+\sum \log (P(w_i|c))=\log (P(c)\cdot \prod P(w_i|c))$. This makes the computation numerically stable.

4. Gaussian Naive Bayes for Continuous Data

Multinomial Naive Bayes works for discrete counts, like words in a document. But what if our features are continuous, real-valued numbers, such as temperature, pixel intensity, or a person's height? We can't create a frequency table for every possible floating-point number.

This is where Gaussian Naive Bayes comes in. It makes an additional assumption: for each class, the distribution of each continuous feature is Gaussian (a normal distribution or bell curve).

The implementation process changes slightly:

1. Training: For each class and for each feature, instead of counting frequencies, we calculate the mean ($\mu$) and variance (${\sigma }^2$) from the training data.
2. Prediction: To get the likelihood $P(x_j|c)$ for a new data point's feature value $x_j$, we don't look up a frequency. Instead, we plug $x_j$, along with the pre-calculated mean ${\mu }_{j,c}$ and variance ${\sigma }_{j,c}^2$, into the Probability Density Function (PDF) of a Gaussian distribution:
   $P(x_j|c)=\frac{1}{\sqrt{2\pi {\sigma }_{j,c}^2}}\exp \left(-\frac{(x_j-{\mu }_{j,c}{)}^2}{2{\sigma }_{j,c}^2}\right)$

The rest of the logic—calculating priors, using log probabilities, and picking the class with the highest log-posterior score—remains the same.

Implementing Gaussian Naive Bayes from Scratch

The following video provides a detailed mathematical derivation and a from-scratch Python implementation of Gaussian Naive Bayes, which will be a great exercise for you.

This implementation beautifully showcases how the statistical properties (mean, variance) of the training data are used to build a probabilistic model for classification.

5. The Naive Bayes Family: A Summary

We've now seen the two most common types of Naive Bayes. A third, Bernoulli Naive Bayes, is also used for text, but it only considers the presence or absence of a word (a binary feature), not its count.

The image below provides a great summary of when to use each type.

Conclusion

In this lesson, you have implemented Naive Bayes, a simple yet powerful probabilistic classifier. You've moved from the discriminative approach of logistic regression to a generative one, learning to model the data itself.

Key Takeaways:

• Naive Bayes is a probabilistic classifier based on Bayes' theorem. It is a generative model.
• Its core power and simplicity come from the "naive" assumption of conditional independence among features.
• Different variants are suited for different data types:
  • Multinomial Naive Bayes for discrete counts (e.g., word frequencies).
  • Gaussian Naive Bayes for continuous features, assuming a normal distribution.
• Practical implementations require handling key issues: Laplace smoothing to avoid zero probabilities and log-probabilities to ensure numerical stability.

Preview of the next lesson:
We've now explored both a discriminative model (Logistic Regression) and a generative one (Naive Bayes). In the next lesson, we will return to the discriminative family to study one of the most influential classical algorithms: Support Vector Machines (SVMs). SVMs take a geometric approach, focusing not just on separating classes, but on finding the optimal decision boundary that has the maximum margin between classes.