Overfitting and Underfitting: Diagnosis and Mitigation

Hello! Welcome to our next lesson.

Introduction

In our last session, we mastered K-fold cross-validation, our primary tool for robustly assessing a model's performance on unseen data. We saw that the scores from cross-validation give us a more reliable picture of generalization than a single train-validation split.

Today, we will use that diagnostic power to tackle two of the most fundamental challenges in machine learning: underfitting and overfitting. Our learning outcome is to diagnose and mitigate overfitting and underfitting.

The ultimate goal of any supervised machine learning model is to generalize—to learn underlying patterns from the training data that allow it to make accurate predictions on new, unseen data. When a model fails to do this well, it's typically because it's either too simple (underfitting) or too complex (overfitting). In this lesson, we will learn:

• How to define and visually identify underfitting and overfitting.
• How to use learning curves to diagnose these issues.
• The common causes behind each problem.
• A suite of powerful techniques, known as regularization, to mitigate overfitting.

What are Underfitting and Overfitting?

At a high level, the challenge is to find a model that is "just right" for the complexity of the data.

• Underfitting (High Bias): An underfit model is too simple. It fails to capture the underlying structure of the data, resulting in poor performance on both the training data and new data. It's like trying to fit a straight line to a U-shaped curve; the model lacks the capacity to learn the pattern.
• Overfitting (High Variance): An overfit model is too complex. It learns the training data so well that it starts to model the noise and random fluctuations, not just the underlying signal. Consequently, it performs exceptionally well on the training data but fails to generalize to new, unseen data. It's like memorizing the answers to a practice exam instead of learning the concepts.

The image below provides a great visual summary of these concepts for different types of models.

To build a solid foundation, let's explore these definitions, their causes, and the theoretical balance between them, known as the bias-variance tradeoff.

Diagnosing Underfitting and Overfitting with Learning Curves

The most powerful tool for diagnosing these issues is the learning curve. A learning curve plots a model's performance (e.g., error or accuracy) on both the training and validation sets as a function of training progress (e.g., number of epochs).

The gap and trajectory of these two curves are highly informative:

1. Diagnosing Underfitting: Both training and validation error will be high and will quickly plateau. The model isn't powerful enough to even learn the training data well.
2. Diagnosing Overfitting: The training error will be low and continue to decrease, while the validation error will either plateau at a high value or, more commonly, start to increase after a certain point. This divergence creates a significant gap between the two curves, indicating the model is memorizing the training data and losing its ability to generalize.

Let's look at some examples of these curves.

By monitoring these curves during training, you can directly observe when your model starts to overfit and decide on a course of action.

Mitigating Overfitting: The Role of Regularization

While fixing underfitting usually involves making your model more complex (e.g., adding layers to a neural network, using a more powerful algorithm), the more common and nuanced challenge in modern AI is fighting overfitting. The collection of techniques used to combat overfitting is called regularization.

The core idea of regularization is to constrain the model's complexity, discouraging it from learning the noise in the training data. Let's explore the most important techniques.

Let's break down the most critical of these techniques in more detail.

L1 and L2 Regularization (Weight Decay)

L1 and L2 are classic forms of regularization that work by adding a penalty term to the model's loss function. This penalty discourages the model's weights from becoming too large.

The total loss becomes:

$$\text{Total Loss}=\text{Data Loss}+\lambda \times \text{Regularization Term}$$

Here, $\lambda$ (lambda) is a hyperparameter that controls the strength of the regularization.

• L2 Regularization (Ridge): The penalty is the sum of the squared values of all the weights: $\lambda \sum w_i^2$. This encourages the model to use all of its features but with small, diffuse weight values. It's the most common type of regularization.
• L1 Regularization (Lasso): The penalty is the sum of the absolute values of all the weights: $\lambda \sum |w_i|$. This has a fascinating side effect: it pushes the weights of less important features to become exactly zero, effectively performing automatic feature selection. This results in a "sparse" model.

Test your understanding!

You are building a model with thousands of features, and you suspect that many of them are irrelevant or redundant. You want to build a simpler, more interpretable model by automatically eliminating these useless features.

Which regularization technique would be more suitable for this goal, L1 or L2, and why?

Show answer

L1 regularization would be more suitable. Because its penalty term is based on the absolute value of the weights, the optimization process is encouraged to drive the weights of unimportant features to exactly zero. This "sparsity" effectively removes those features from the model, making it simpler and more interpretable. L2 regularization, in contrast, would only shrink the weights towards zero but not make them exactly zero, keeping all features in the model.

Dropout

Dropout is a brilliantly simple yet powerful regularization technique developed specifically for neural networks.

During each training step, dropout randomly "deactivates" a fraction of the neurons in a layer. This means their output is set to zero for that forward and backward pass.

Why does this work?
It prevents neurons from becoming too co-dependent on each other. Since any given neuron might disappear at any moment, each neuron is forced to learn features that are robust and useful on their own. You can think of it as training a large ensemble of smaller, slightly different networks and averaging their predictions, which is a powerful way to reduce variance.

Given your CS background, you might find the formal algorithm for dropout interesting.

Early Stopping

This is one of the most pragmatic and effective forms of regularization. As we saw from the learning curves, a model's validation error often starts to increase after a certain number of training epochs. Early stopping simply says: stop training at that point!

You monitor the validation loss at the end of each epoch and halt the training process once the loss has not improved for a specified number of consecutive epochs (a parameter called "patience"). You then save the model weights from the epoch with the best validation performance.

A Modern Insight: The Double Descent Phenomenon

The classic view, as we've discussed, is that as model complexity increases, test error follows a U-shaped curve. However, research in modern deep learning has revealed a surprising twist. For highly over-parameterized models (like the large language models and transformers we will study later), the test error can decrease again after the initial overfitting peak.

This doesn't invalidate the regularization techniques we've learned; they remain essential. But it does suggest that the relationship between complexity and generalization is more intricate than previously thought, especially in the realm of massive models.

Conclusion

You have now learned to diagnose and treat the fundamental ailments of machine learning models. Finding the right balance between bias and variance is at the heart of building effective models.

Key Takeaways:

• Underfitting (High Bias): A model that is too simple. Diagnosed by high training and validation error. Mitigated by increasing model complexity or adding features.
• Overfitting (High Variance): A model that is too complex and memorizes training data noise. Diagnosed by a large gap between low training error and high validation error on a learning curve.
• Regularization is the set of techniques to combat overfitting by constraining model complexity.
• Key Mitigation Strategies for Overfitting:
  • Get more data or use data augmentation.
  • L1/L2 Regularization: Add a penalty to the loss function to shrink weights (L2) or enforce sparsity (L1).
  • Dropout: Randomly deactivate neurons during training to build more robust features.
  • Early Stopping: Stop training when validation performance stops improving.

Preview of the next lesson:
Today's entire discussion was a practical application of a core theoretical concept in machine learning: the bias-variance tradeoff. In our next lesson, we will formalize this relationship, mathematically breaking down a model's error into its bias, variance, and irreducible error components. This will provide you with a deeper theoretical framework for understanding why the techniques we learned today work.