Eigendecomposition of Symmetric Matrices: Directions and Scales

Hello. In the previous lesson, you connected vector operations to geometry: lengths, alignment, and projections. Those ideas now become a way to understand a matrix as a transformation with certain preferred directions.

For a general vector, multiplying by a matrix changes both its length and direction. An eigenvector is exceptional: the matrix may stretch it, shrink it, flip it, or collapse it to zero, but it does not move it off its original line. For a real symmetric matrix, these special directions form an orthonormal coordinate system, giving a particularly clean decomposition. This is the version that appears throughout ML, especially with covariance matrices and later PCA.

By the end of this lesson, you should be able to find the eigenvalues and eigenvectors of a small symmetric matrix, assemble its eigendecomposition, and explain it as “change coordinates, scale independently, then change back.”

---

Eigenvectors: directions a transformation preserves

Let $A$ be a square matrix. A nonzero vector $\mathbf{v}$ is an eigenvector of $A$ if there is a scalar $\lambda$ such that

$$A\mathbf{v}=\lambda \mathbf{v}$$

The scalar $\lambda$ is the corresponding eigenvalue.

This equation says that applying $A$ to $\mathbf{v}$ has exactly the same effect as multiplying $\mathbf{v}$ by one number:

• If $\lambda >1$, the vector is stretched.
• If $0<\lambda <1$, it is shrunk.
• If $\lambda <0$, it is flipped through the origin and scaled by $|\lambda |$.
• If $\lambda =0$, it is collapsed to the zero vector.

The key word is direction. An eigenvector is not a unique arrow of a particular length. If $\mathbf{v}$ is an eigenvector, then every nonzero scalar multiple of it is also an eigenvector with the same eigenvalue:

$$A(c\mathbf{v})=cA\mathbf{v}=c\lambda \mathbf{v}=\lambda (c\mathbf{v})$$

So eigenvectors represent invariant lines through the origin.

For this lesson, we focus on a real symmetric matrix:

$$A=A^{\top }$$

For example,

$$\left[\begin{array}{cc}3& 1\\ 1& 3\end{array}\right]$$

is symmetric because its entries mirror across the main diagonal.

Symmetry provides powerful guarantees:

1. Every eigenvalue is real.
2. Eigenvectors belonging to distinct eigenvalues are orthogonal.
3. We can choose unit-length eigenvectors that form an orthonormal basis.

Those guarantees are what make eigendecomposition geometrically transparent and numerically useful in ML.

---

Finding eigenvalues, then eigenvectors

Consider the symmetric matrix

$$A=\left[\begin{array}{cc}3& 1\\ 1& 3\end{array}\right]$$

We want nonzero vectors $\mathbf{v}$ and scalars $\lambda$ satisfying

$$A\mathbf{v}=\lambda \mathbf{v}$$

Move everything to the left:

$$A\mathbf{v}-\lambda \mathbf{v}=0$$

Because $\lambda \mathbf{v}=\lambda I\mathbf{v}$, where $I$ is the identity matrix, this becomes

$$(A-\lambda I)\mathbf{v}=0$$

For a nonzero solution $\mathbf{v}$ to exist, the matrix $A-\lambda I$ must be singular. Therefore,

$$\det (A-\lambda I)=0$$

This is the characteristic equation.

Step 1: Find the eigenvalues

For the example matrix,

$$A-\lambda I=\left[\begin{array}{cc}3-\lambda & 1\\ 1& 3-\lambda \end{array}\right]$$

Set its determinant to zero:

$$\det (A-\lambda I)=(3-\lambda {)}^2-1=0$$

Expanding and factoring gives:

$${\lambda }^2-6\lambda +8=0$$ $$(\lambda -4)(\lambda -2)=0$$

Therefore, the two eigenvalues are

$${\lambda }_1=4,{\lambda }_2=2$$

A useful check for a $2\times 2$ matrix is:

$${\lambda }_1+{\lambda }_2=\mathrm{trace}(A)$$ $$4+2=3+3=6$$

Also,

$${\lambda }_1{\lambda }_2=\det (A)$$ $$4\cdot 2=3\cdot 3-1\cdot 1=8$$

These checks do not replace the calculation, but they can catch an arithmetic mistake during an interview or implementation review.

Step 2: Find the eigenvector for ${\lambda }_1=4$

Substitute $\lambda =4$ into

$$(A-\lambda I)\mathbf{v}=0$$ $$A-4I=\left[\begin{array}{cc}-1& 1\\ 1& -1\end{array}\right]$$

Let

$$\mathbf{v}=\left[\begin{array}{c}x\\ y\end{array}\right]$$

Then the first row gives

$$-x+y=0$$

so $y=x$. One valid eigenvector is

$${\mathbf{v}}_1=\left[\begin{array}{c}1\\ 1\end{array}\right]$$

Verify it directly:

$$A{\mathbf{v}}_1=\left[\begin{array}{cc}3& 1\\ 1& 3\end{array}\right]\left[\begin{array}{c}1\\ 1\end{array}\right]=\left[\begin{array}{c}4\\ 4\end{array}\right]=4\left[\begin{array}{c}1\\ 1\end{array}\right]$$

So the direction along the line $y=x$ is stretched by a factor of $4$.

Step 3: Find the eigenvector for ${\lambda }_2=2$

Now use $\lambda =2$:

$$A-2I=\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right]$$

The equation is

$$x+y=0$$

so $y=-x$. One valid eigenvector is

$${\mathbf{v}}_2=\left[\begin{array}{c}1\\ -1\end{array}\right]$$

Again, verify:

$$A{\mathbf{v}}_2=\left[\begin{array}{cc}3& 1\\ 1& 3\end{array}\right]\left[\begin{array}{c}1\\ -1\end{array}\right]=\left[\begin{array}{c}2\\ -2\end{array}\right]=2\left[\begin{array}{c}1\\ -1\end{array}\right]$$

So the direction along $y=-x$ is stretched by a factor of $2$.

Notice the two eigenvectors are orthogonal:

$${\mathbf{v}}_1^{\top }{\mathbf{v}}_2=\left[\begin{array}{cc}1& 1\end{array}\right]\left[\begin{array}{c}1\\ -1\end{array}\right]=0$$

That is not an accident. Distinct eigenvectors of a real symmetric matrix are orthogonal.

---

From eigenpairs to eigendecomposition

To form the standard symmetric eigendecomposition, normalize the eigenvectors to have unit length:

$${\mathbf{q}}_1=\frac{1}{\sqrt{2}}\left[\begin{array}{c}1\\ 1\end{array}\right],{\mathbf{q}}_2=\frac{1}{\sqrt{2}}\left[\begin{array}{c}1\\ -1\end{array}\right]$$

Put these unit eigenvectors into the columns of a matrix:

$$Q=\left[\begin{array}{cc}{\mathbf{q}}_1& {\mathbf{q}}_2\end{array}\right]=\frac{1}{\sqrt{2}}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]$$

Place their corresponding eigenvalues on the diagonal:

$$\Lambda =\left[\begin{array}{cc}4& 0\\ 0& 2\end{array}\right]$$

Because the columns of $Q$ are orthonormal,

$$Q^{\top }Q=Q{Q}^{\top }=I$$

Thus $Q^{-1}=Q^{\top }$, and the eigendecomposition is

$$A=Q\Lambda Q^{\top }$$

For this example,

$$\left[\begin{array}{cc}3& 1\\ 1& 3\end{array}\right]=\left(\frac{1}{\sqrt{2}}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]\right)\left[\begin{array}{cc}4& 0\\ 0& 2\end{array}\right]{\left(\frac{1}{\sqrt{2}}\left[\begin{array}{cc}1& 1\\ 1& -1\end{array}\right]\right)}^{\top }$$

The order matters: the first column of $Q$ must correspond to the first diagonal entry of $\Lambda$. Reordering eigenvectors is valid only if you make the identical reordering of eigenvalues.

---

The geometry: rotate into eigenvector coordinates, scale, rotate back

The decomposition

$$A=Q\Lambda Q^{\top }$$

is best read right to left when applying $A$ to a vector $\mathbf{x}$:

$$A\mathbf{x}=Q\Lambda Q^{\top }\mathbf{x}$$

1. $Q^{\top }\mathbf{x}$: express $\mathbf{x}$ in the orthonormal eigenvector coordinate system.
2. $\Lambda$: scale each eigenvector coordinate independently.
3. $Q$: express the result back in the original coordinate system.

For the example, take

$$\mathbf{x}=\left[\begin{array}{c}2\\ 0\end{array}\right]$$

First, express it in the eigenvector basis:

$$Q^{\top }\mathbf{x}=\left[\begin{array}{c}\sqrt{2}\\ \sqrt{2}\end{array}\right]$$

Then scale those coordinates:

$$\Lambda Q^{\top }\mathbf{x}=\left[\begin{array}{cc}4& 0\\ 0& 2\end{array}\right]\left[\begin{array}{c}\sqrt{2}\\ \sqrt{2}\end{array}\right]=\left[\begin{array}{c}4\sqrt{2}\\ 2\sqrt{2}\end{array}\right]$$

Finally, transform back:

$$Q\left[\begin{array}{c}4\sqrt{2}\\ 2\sqrt{2}\end{array}\right]=\left[\begin{array}{c}6\\ 2\end{array}\right]$$

This agrees with direct multiplication:

$$A\mathbf{x}=\left[\begin{array}{cc}3& 1\\ 1& 3\end{array}\right]\left[\begin{array}{c}2\\ 0\end{array}\right]=\left[\begin{array}{c}6\\ 2\end{array}\right]$$

More generally, if a vector is written as a combination of orthonormal eigenvectors,

$$\mathbf{x}=c_1{\mathbf{q}}_1+c_2{\mathbf{q}}_2$$

then

$$A\mathbf{x}={\lambda }_1{c}_1{\mathbf{q}}_1+{\lambda }_2{c}_2{\mathbf{q}}_2$$

The matrix does not mix the eigenvector components. It simply rescales each one.

For the unit circle, the transformed shape is an ellipse when both eigenvalues are nonzero. Its axis directions are the eigenvectors, and its semiaxis lengths are the absolute values of the eigenvalues. The provided diagram depicts the common positive-eigenvalue case. If an eigenvalue were negative, the corresponding axis would also flip; if it were zero, the circle would collapse onto a lower-dimensional shape.

---

Why symmetric eigendecomposition matters in ML

Many central ML objects are symmetric. The most important early example is a feature covariance matrix:

$$\Sigma =\frac{1}{n}\sum _{i=1}^n({\mathbf{x}}_i-\mathbf{\mu })({\mathbf{x}}_i-\mathbf{\mu }{)}^{\top }$$

A covariance matrix is symmetric and positive semidefinite. Therefore, its eigenvalues satisfy

$${\lambda }_i\ge 0$$

For a unit eigenvector ${\mathbf{q}}_i$,

$${\mathbf{q}}_i^{\top }\Sigma {\mathbf{q}}_i={\lambda }_i$$

So the eigenvalue measures the variance of the centered data along its associated eigenvector direction. Large eigenvalues identify directions of large spread; small eigenvalues identify directions with comparatively little variation. This is the basis of PCA, which you will study later as a dimensionality-reduction method.

Two caveats are worth retaining:

• Symmetry is doing real work. An arbitrary square matrix may have complex eigenvalues or may not have enough independent eigenvectors to form a basis.
• Eigenvalues are not automatically positive. They are nonnegative for covariance matrices and other positive semidefinite matrices, but a general symmetric matrix can have negative eigenvalues.
• Repeated eigenvalues make individual eigenvectors non-unique. If $A=2I$, every nonzero vector is an eigenvector with eigenvalue $2$. In that situation, the eigenspace is well-defined, but no single direction within it is special.

---

PyTorch verification

In code, use torch.linalg.eigh for real symmetric matrices. The h refers to Hermitian, the complex-number generalization of symmetric. It returns eigenvalues in ascending order and returns eigenvectors as the columns of $Q$.

import torch

A = torch.tensor(
    [
        [3.0, 1.0],
        [1.0, 3.0],
    ]
)

eigenvalues, Q = torch.linalg.eigh(A)
Lambda = torch.diag(eigenvalues)

print(eigenvalues)
# tensor([2., 4.])

# Eigendecomposition reconstruction
A_reconstructed = Q @ Lambda @ Q.T

# Column-wise eigenvector equation: A q_i = lambda_i q_i
assert torch.allclose(A, A_reconstructed)
assert torch.allclose(A @ Q, Q @ Lambda)

# Orthonormality of the eigenvector matrix
assert torch.allclose(Q.T @ Q, torch.eye(2))

Do not expect PyTorch to return the exact signs used in the handwritten derivation. If $\mathbf{q}$ is an eigenvector, then $-\mathbf{q}$ is equally valid, so an implementation may return either sign. The reconstruction and eigenvector equation are the reliable checks.

For an interview-level explanation, a concise and accurate version is:

For a real symmetric matrix, eigendecomposition writes $A=Q\Lambda Q^{\top }$, where $Q$'s columns are orthonormal eigenvectors and $\Lambda$ contains eigenvalues. Geometrically, the matrix changes into its eigenvector coordinates, scales each independent direction by its eigenvalue, and changes back. For a covariance matrix, the eigenvectors are principal directions of variation and the eigenvalues are variances along those directions.

---

You can now compute eigenpairs for a small symmetric matrix and interpret them correctly: eigenvectors are invariant directions, while eigenvalues are the signed scaling along those directions. Symmetry ensures these directions can be chosen orthonormal, yielding the clean form

$$A=Q\Lambda Q^{\top }$$

Next, you will extend this “preferred directions and scales” idea to matrices that may be rectangular or nonsymmetric using singular value decomposition.