Interpreting Covariance Matrix Eigenvectors and Eigenvalues

Good to see you again. In the previous lesson, gradient descent made the geometry of a loss surface concrete: feature scales influence curvature, and curvature determines how readily one learning rate can make progress. We now use the same linear-algebra vocabulary to describe the geometry of the data itself.

This lesson explains what a covariance matrix says about a multivariate dataset. By the end, you should be able to read its eigenvectors as mutually perpendicular directions of variation and its eigenvalues as the variances along those directions. This is the mathematical core that will later make principal component analysis (PCA) feel like a change of coordinate system rather than a mysterious algorithm.

---

From raw columns to a geometric object

Suppose each observation has $p$ numerical features. Write the centered data matrix as

$${\mathbf{X}}_c=\left[\begin{array}{c}{\mathbf{x}}_1^{\top }\\ ⋮\\ {\mathbf{x}}_n^{\top }\end{array}\right]$$

where each row ${\mathbf{x}}_i$ is an observation after subtracting the feature-wise mean. Centering matters: we want to describe variation around the data cloud's centre, not its location in the original coordinate system.

A sample covariance matrix can be written as

$$\mathbf{S}=\frac{1}{n-1}{\mathbf{X}}_c^{\top }{\mathbf{X}}_c$$

The denominator is sometimes $n$ rather than $n-1$, depending on whether one is treating the data as a sample or an entire population. That choice rescales every eigenvalue slightly but does not change the eigenvector directions.

For two features,

$$\mathbf{S}=\left[\begin{array}{cc}{s}_{11}& s_{12}\\ s_{21}& s_{22}\end{array}\right]$$

has a straightforward interpretation:

• $s_{11}$ and $s_{22}$ are the variances of the individual features.
• $s_{12}=s_{21}$ is their covariance: positive when deviations from their means tend to share a sign, negative when they tend to have opposing signs.
• The matrix is symmetric, a property that gives it especially useful eigenvalue and eigenvector behavior.

The covariance matrix records second-order geometry: overall spread and linear co-movement. It does not tell us whether the data cloud is Gaussian, multimodal, crescent-shaped, or otherwise structured in more complicated ways. Distinct distributions can share the same covariance matrix.

---

Variation along an arbitrary direction

The original feature axes are not necessarily meaningful directions for describing variation. If customer activity and spend both tend to rise together, for example, the cloud may be elongated diagonally rather than horizontally or vertically. We need a way to ask: how variable are the data when viewed along a chosen direction?

Let $\mathbf{u}$ be a unit vector, so that

$${\mathbf{u}}^{\top }\mathbf{u}=1$$

Projecting centered observation ${\mathbf{x}}_i$ onto this direction gives the scalar coordinate

$$z_i={\mathbf{u}}^{\top }{\mathbf{x}}_i$$

The variance of all projected coordinates is

$$\mathrm{Var}(z)={\mathbf{u}}^{\top }\mathbf{Su}$$

This quadratic expression is called a Rayleigh quotient when $\mathbf{u}$ is constrained to have unit length. It is the precise mathematical version of rotating an axis through a data cloud and measuring how widely the projections spread along it.

We want the direction that maximizes this quantity:

$$\underset{\mathbf{u}}{\max }{\mathbf{u}}^{\top }\mathbf{Su}\text{subject to}{\mathbf{u}}^{\top }\mathbf{u}=1$$

The length constraint is essential. Without it, multiplying any candidate direction by $100$ would multiply the apparent variance by $100^2$, which tells us nothing about orientation.

Using a Lagrange multiplier $\lambda$, define

$$\mathcal{L}(\mathbf{u},\lambda )={\mathbf{u}}^{\top }\mathbf{Su}-\lambda \left({\mathbf{u}}^{\top }\mathbf{u}-1\right)$$

Differentiating with respect to $\mathbf{u}$ and setting the derivative to zero gives

$$2\mathbf{Su}-2\lambda \mathbf{u}=0$$

or equivalently,

$$\boxed{\mathbf{Su}=\lambda \mathbf{u}}$$

This is the eigenvalue equation. It appears not because eigenvectors are an arbitrary linear-algebra trick, but because they solve the meaningful optimization problem: find a direction along which the centered observations have greatest variance.

---

The central interpretation

For a covariance matrix:

$$\boxed{\text{eigenvector}=\text{a principal direction of variation}}$$ $$\boxed{\text{eigenvalue}=\text{the variance along that direction}}$$

Order the eigenvalues from largest to smallest:

$${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_p\ge 0$$

Their corresponding unit eigenvectors are

$${\mathbf{u}}_1,{\mathbf{u}}_2,\dots ,{\mathbf{u}}_p$$

Then:

1. ${\mathbf{u}}_1$ is the direction along which the data vary most.
2. ${\mathbf{u}}_2$ is the direction of greatest remaining variation subject to being perpendicular to ${\mathbf{u}}_1$.
3. The later eigenvectors describe progressively smaller orthogonal sources of variation.

Because $\mathbf{S}$ is symmetric, eigenvectors associated with distinct eigenvalues can be chosen to be orthonormal:

$${\mathbf{u}}_i^{\top }{\mathbf{u}}_j=\{\begin{array}{ll}1& i=j\\ 0& i\ne j\end{array}$$

This provides a rotated coordinate system in which the coordinates are uncorrelated. The covariance matrix becomes diagonal:

$$\mathbf{S}=\mathbf{U\Lambda }{\mathbf{U}}^{\top }$$

where $\mathbf{U}$ has the eigenvectors as its columns and

$$\mathbf{\Lambda }=\left[\begin{array}{cccc}{\lambda }_1& 0& \cdots & 0\\ 0& {\lambda }_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\lambda }_p\end{array}\right]$$

contains the eigenvalues.

---

Reading the shape of a two-feature dataset

The image shows the geometric situation to keep in mind. A cloud that is elongated along a diagonal cannot be adequately summarized by saying “both $x$ and $y$ have high variance.” The key pattern is that they vary together. The first eigendirection captures this joint movement.

Consider the covariance matrix

$$\mathbf{S}=\left[\begin{array}{cc}4& 3\\ 3& 4\end{array}\right]$$

The equal diagonal entries say that the two original variables have the same individual variance, $4$. The positive off-diagonal covariance says that above-average values of one variable tend to occur with above-average values of the other.

Its eigendecomposition is particularly simple:

$${\lambda }_1=7,{\mathbf{u}}_1=\frac{1}{\sqrt{2}}\left[\begin{array}{c}1\\ 1\end{array}\right]$$ $${\lambda }_2=1,{\mathbf{u}}_2=\frac{1}{\sqrt{2}}\left[\begin{array}{c}1\\ -1\end{array}\right]$$

Interpret these, rather than treating them as calculation outputs:

Quantity	Interpretation
${\mathbf{u}}_1$	The dominant direction increases both features together, in equal proportion.
${\lambda }_1=7$	The projected data have variance $7$ in that joint-increase direction.
${\mathbf{u}}_2$	The perpendicular direction contrasts the two features: one rises while the other falls.
${\lambda }_2=1$	There is relatively little variation in that contrast.

The total variance is

$$4+4=8$$

It also equals the sum of the eigenvalues:

$$7+1=8$$

Thus, the first direction accounts for

$$\frac{7}{8}=0.875$$

or $87.5\%$ of the total variance. Most of the variation can be described as a shared movement of the two variables; the independent contrast is small.

Two details are worth keeping straight:

• The sign of an eigenvector is arbitrary. Both ${\mathbf{u}}_1$ and $-{\mathbf{u}}_1$ describe the same undirected axis. A software library may choose either.
• The eigenvalue is a variance, so the standard deviation along the direction is $\sqrt{\lambda }$. In an ellipse picture, axis lengths are proportional to $\sqrt{\lambda }$, not $\lambda$ itself.

---

Useful boundary cases

These cases make covariance eigenstructure easier to diagnose in real outputs.

No covariance

If

$$\mathbf{S}=\left[\begin{array}{cc}9& 0\\ 0& 1\end{array}\right]$$

the original axes already coincide with the principal directions. The eigenvectors are the standard coordinate vectors:

$$\left[\begin{array}{c}1\\ 0\end{array}\right]\text{and}\left[\begin{array}{c}0\\ 1\end{array}\right]$$

with eigenvalues $9$ and $1$. There is no rotation to discover; the data are simply more variable in the first feature than in the second.

Nearly perfect linear dependence

If two centered features are almost exact linear copies of one another, one eigenvalue is near zero. The data cloud is close to a line rather than filling a two-dimensional area.

A near-zero eigenvalue can therefore mean:

• strong redundancy among numerical features;
• a dataset lying close to a lower-dimensional subspace;
• or, in an engineering setting, possible duplicated, derived, or nearly deterministic columns.

It does not automatically mean that a feature may be discarded safely for a predictive task. A direction with low global variance can still contain signal for a rare but valuable target outcome.

Equal eigenvalues

When eigenvalues are equal, such as

$$\mathbf{S}=\left[\begin{array}{cc}4& 0\\ 0& 4\end{array}\right]$$

the data have equal variance in every direction. The cloud is circular in two dimensions. No direction is uniquely “first,” and any orthonormal pair can serve as eigenvectors. This is a practical reminder not to overinterpret an eigenvector when its eigenvalue is nearly tied with another.

---

Why feature scale remains a modelling decision

The covariance matrix retains the units and scales of the original features. If one column is annual revenue in dollars and another is a count of support tickets, raw covariance-based eigenvalues are likely dominated by revenue merely because its numerical scale is larger.

This connects directly to the feature-scaling lesson:

• Covariance PCA preserves absolute-scale variation. It is reasonable when the units are commensurate and magnitude itself is meaningful.
• Standardizing each feature to mean $0$ and variance $1$ before calculating covariance makes the analysis equivalent to using a correlation matrix. It gives each feature equal initial scale.

Neither choice is universally correct. It encodes a product and domain judgment about whether a unit change or a standard-deviation-sized change is the meaningful basis for comparison. In a technical proposal, asking “Was PCA fit on raw features or standardized features, and why?” is usually more informative than merely noting the number of components retained.

---

A brief numerical verification

In a notebook, np.linalg.eigh is the appropriate NumPy routine for a real symmetric covariance matrix. Unlike the general eig, it is designed for symmetric matrices and returns real eigenvalues with numerically stable orthogonal eigenvectors.

import numpy as np

S = np.array([
    [4.0, 3.0],
    [3.0, 4.0],
])

eigenvalues, eigenvectors = np.linalg.eigh(S)

# eigh returns ascending eigenvalues, so reverse both outputs.
order = np.argsort(eigenvalues)[::-1]
eigenvalues = eigenvalues[order]
eigenvectors = eigenvectors[:, order]

print("eigenvalues:\n", eigenvalues)
print("\neigenvectors as columns:\n", eigenvectors)

# Confirm S u_k = lambda_k u_k
print("\nEigenvalue equation holds:")
print(np.allclose(S @ eigenvectors, eigenvectors * eigenvalues))

# Confirm orthonormality and conservation of total variance.
print("\nU^T U:\n", eigenvectors.T @ eigenvectors)
print("\ntrace(S):", np.trace(S))
print("sum of eigenvalues:", eigenvalues.sum())

When interpreting output, each column of eigenvectors is an eigenvector. Expect signs that may differ from the hand calculation; for example, NumPy could return

$$-\frac{1}{\sqrt{2}}\left[\begin{array}{c}1\\ 1\end{array}\right]$$

rather than its positive version. That is identical geometrically.

---

Key takeaways

A covariance matrix describes the spread and linear co-movement of centered multivariate data. Its eigendecomposition identifies a coordinate system aligned with the data cloud:

$$\mathbf{S}{\mathbf{u}}_k={\lambda }_k{\mathbf{u}}_k$$

• Each eigenvector ${\mathbf{u}}_k$ is a direction of variation.
• The largest-eigenvalue eigenvector is the direction of greatest variance.
• Each eigenvalue ${\lambda }_k$ is the variance of the data projected onto its eigenvector.
• Covariance-matrix eigenvectors can be made perpendicular, so each direction captures variation distinct from the preceding ones.
• The sum of eigenvalues equals total feature variance:

$$\mathrm{tr}(\mathbf{S})=\sum _{k=1}^p{\lambda }_k$$

The geometry also explains why these directions later become principal components: retaining the directions with large eigenvalues preserves the greatest amount of linear variation, while low-eigenvalue directions represent little remaining spread.

This completes the linear algebra and optimization module. Next, the course moves from geometry to uncertainty, beginning with expectations, variances, and covariances as quantities computed from explicit probability models.