Eigenvalues and Eigenvectors: ML Significance

Hello! Welcome to the second lesson in our journey through the mathematical foundations of AI.

In our last session, we explored how matrices act as linear transformations, stretching, squashing, and rotating vector spaces. We concluded by posing a question: are there any vectors that are special, in that a transformation only scales them without changing their direction?

Today, we'll answer that question. This lesson is all about eigenvectors and eigenvalues. You will learn how to calculate these special vectors and their corresponding scaling factors, and, more importantly, you will understand their profound significance in machine learning. They are key to unlocking the underlying structure of data and are the foundation for powerful techniques like Principal Component Analysis (PCA).

What are Eigenvectors and Eigenvalues?

An eigenvector of a matrix is a non-zero vector whose direction remains unchanged when the linear transformation represented by the matrix is applied to it. It may be stretched, squished, or flipped, but it won't be rotated off its original line (its "span"). The factor by which the eigenvector is scaled is called the eigenvalue.

This relationship is elegantly captured by the defining equation:

$A\mathbf{v}=\lambda \mathbf{v}$

To build a strong intuition for this concept, let's watch one of the best explanations available, from 3Blue1Brown.

As you saw, eigenvectors are the "axes" of a linear transformation. They reveal the directions along which the transformation acts purely by stretching or compressing.

So why is finding these special axes useful? In physics, if you consider a 3D rotation, the eigenvector is the axis of rotation—a concept much more intuitive than the full 3x3 rotation matrix. In machine learning, they help us find the most important directions in our data.

The Calculation of Eigenvalues and Eigenvectors

Now that we have the conceptual understanding, let's move to the mechanics of how to find them. The entire process starts from the defining equation, $A\mathbf{v}=\lambda \mathbf{v}$.

Our goal is to find the values for $\lambda$ (the eigenvalues) and $\mathbf{v}$ (the eigenvectors) that satisfy this equation for a given square matrix $A$.

We can rearrange the equation as follows:

1. $A\mathbf{v}-\lambda \mathbf{v}=0$
2. To factor out $\mathbf{v}$, we need to express $\lambda \mathbf{v}$ as a matrix-vector product. We use the identity matrix $I$: $A\mathbf{v}-\lambda I\mathbf{v}=0$
3. Now, we can factor out $\mathbf{v}$: $(A-\lambda I)\mathbf{v}=0$

This equation tells us that the matrix $(A-\lambda I)$ transforms the vector $\mathbf{v}$ into the zero vector. We are looking for a non-zero eigenvector $\mathbf{v}$, which is a "non-trivial" solution. From our knowledge of linear algebra, a matrix equation $M\mathbf{x}=0$ has a non-zero solution for $\mathbf{x}$ only if the matrix $M$ is "singular" or "non-invertible". This happens precisely when the determinant of the matrix is zero.

Therefore, to find our eigenvalues, we need to find the values of $\lambda$ that make the determinant of $(A-\lambda I)$ equal to zero:
$\det (A-\lambda I)=0$

This is known as the characteristic equation. The following video segment walks through this derivation and shows how it leads to finding the eigenvalues.

Step-by-Step Guide

Let's solidify this with a step-by-step process. The following video from Professor Dave Explains provides a very clear, practical walkthrough of the calculations for a 2x2 matrix.

To summarize the process:

1. Find the Eigenvalues ($\lambda$):

   • Set up the characteristic equation: $\det (A-\lambda I)=0$.
   • Solving this equation will yield a polynomial in $\lambda$. The roots of this polynomial are the eigenvalues of matrix $A$.

2. Find the Eigenvectors ($\mathbf{v}$):

   • For each eigenvalue $\lambda$ you found, substitute it back into the equation $(A-\lambda I)\mathbf{v}=0$.
   • This will give you a system of linear equations. Solve this system to find the vector $\mathbf{v}$.
   • Note that any non-zero scalar multiple of an eigenvector is also an eigenvector. So, there isn't a single unique eigenvector for a given eigenvalue, but rather an entire line (or subspace) of them. We typically pick a simple representative vector.

Test your understanding!

Let's calculate the eigenvalues and eigenvectors for the matrix $A=\left[\begin{array}{cc}4& -2\\ 1& 1\end{array}\right]$.

1. Find the eigenvalues.
2. Find the corresponding eigenvectors.

Show answer

1. Find Eigenvalues:
   First, we set up the characteristic equation $\det (A-\lambda I)=0$.

   $$\det \left(\left[\begin{array}{cc}4& -2\\ 1& 1\end{array}\right]-\lambda \left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\right)=0$$ $$\det \left[\begin{array}{cc}4-\lambda & -2\\ 1& 1-\lambda \end{array}\right]=0$$ $$(4-\lambda )(1-\lambda )-(-2)(1)=0$$ $$4-4\lambda -\lambda +{\lambda }^2+2=0$$ $${\lambda }^2-5\lambda +6=0$$

   Factoring the quadratic equation:

   $$(\lambda -2)(\lambda -3)=0$$

   So, the eigenvalues are ${\lambda }_1=2$ and ${\lambda }_2=3$.

2. Find Eigenvectors:

   • For ${\lambda }_1=2$:
     We solve $(A-2I)\mathbf{v}=0$.

     $$\left[\begin{array}{cc}4-2& -2\\ 1& 1-2\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]⟹\left[\begin{array}{cc}2& -2\\ 1& -1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

     Both rows give the same equation: $2x-2y=0$ (or $x-y=0$), which means $x=y$. A simple choice for the eigenvector is ${\mathbf{v}}_1=\left[\begin{array}{c}1\\ 1\end{array}\right]$.

   • For ${\lambda }_2=3$:
     We solve $(A-3I)\mathbf{v}=0$.

     $$\left[\begin{array}{cc}4-3& -2\\ 1& 1-3\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]⟹\left[\begin{array}{cc}1& -2\\ 1& -2\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right]$$

     This gives the equation $x-2y=0$, which means $x=2y$. A simple choice for the eigenvector is ${\mathbf{v}}_2=\left[\begin{array}{c}2\\ 1\end{array}\right]$.

Significance in Machine Learning

This might seem like a purely mathematical exercise, but it has profound implications in data science. In ML, we often aren't interested in an arbitrary matrix $A$, but in a special one: the covariance matrix.

For a given dataset, the covariance matrix is a square matrix that describes the variance of each feature and the covariance between pairs of features.

• The eigenvectors of the covariance matrix point in the directions of maximum variance in the data. These are called the principal components.
• The eigenvalue corresponding to each eigenvector tells you the amount of variance captured by that direction. A large eigenvalue means that the data is very spread out along that eigenvector.

This is the core idea behind Principal Component Analysis (PCA), one of the most widely used techniques for dimensionality reduction.

By finding the eigenvectors and eigenvalues of the covariance matrix, we can identify the directions that contain the most information (variance) and discard the ones that contain the least, reducing the number of dimensions while preserving the essential structure of the data.

Implementation in Python

Given your background in software engineering, you know that we rarely perform these calculations by hand. Scientific computing libraries like NumPy make this trivial. Let's see how to compute the eigenvalues and eigenvectors for the matrix from our exercise using Python.

Here is the code for our example matrix $A=\left[\begin{array}{cc}4& -2\\ 1& 1\end{array}\right]$:

import numpy as np




# Define the matrix
A = np.array([[4, -2],
              [1,  1]])




# Calculate eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues:", eigenvalues)
print("Eigenvectors:")
print(eigenvectors)




# Note: NumPy returns eigenvectors as columns of the matrix.
# The first column is the eigenvector for the first eigenvalue.
# Let's verify for the first eigenvalue (3) and its eigenvector
lambda_1 = eigenvalues[0]
v_1 = eigenvectors[:, 0]
print("\nVerifying A*v_1 = lambda_1*v_1:")
print("A @ v_1:", A @ v_1)
print("lambda_1 * v_1:", lambda_1 * v_1)

The output confirms that our manual calculations were correct (up to a scaling factor and normalization for the eigenvectors).

Eigenbasis and Diagonalization

Finally, let's connect back to the idea of transformations and prepare for our next lesson. What happens if a matrix has enough eigenvectors to form a basis for the entire space? Such a basis is called an eigenbasis.

If we change our coordinate system to use this eigenbasis, the transformation becomes incredibly simple. In the eigenbasis, the transformation matrix is a diagonal matrix with the eigenvalues on the diagonal. All it does is scale the new basis vectors.

This process of simplifying a matrix by changing to its eigenbasis is fundamental to eigendecomposition, a topic we'll explore next.

Not all matrices can be diagonalized (e.g., a shear transformation doesn't have enough eigenvectors to span the space), but for many matrices we encounter in machine learning (like symmetric covariance matrices), this is possible and extremely useful.

Conclusion

In this lesson, we've gone from the intuitive idea of "special vectors" to the practical calculations and their significance in machine learning.

Key Takeaways:

• An eigenvector of a matrix $A$ is a vector $\mathbf{v}$ that satisfies $A\mathbf{v}=\lambda \mathbf{v}$, where $\lambda$ is its corresponding eigenvalue. It represents a direction that is only scaled by the transformation.
• Eigenvalues are found by solving the characteristic equation, $\det (A-\lambda I)=0$.
• Eigenvectors are found by substituting each eigenvalue back into $(A-\lambda I)\mathbf{v}=0$ and solving for $\mathbf{v}$.
• In machine learning, the eigenvectors of a covariance matrix are the principal components of the data, representing directions of maximum variance. The eigenvalues quantify this variance.
• An eigenbasis (a basis of eigenvectors) simplifies a linear transformation into a diagonal matrix of eigenvalues, making computations easier.

Preview of the next lesson:

We've just seen that we can simplify a matrix by changing to its eigenbasis. In the next lesson, we will formalize this into a powerful technique called eigendecomposition. We will then explore an even more general and widely applicable method, Singular Value Decomposition (SVD), which can decompose any matrix, not just square ones.