Vector and Matrix Operations for Neural Networks

Hello, and welcome to the first lesson of your course on AI theory and architecture! We're kicking things off by building the essential mathematical toolkit you'll need throughout your journey.

This lesson focuses on the fundamental vector and matrix operations that are the computational heart of neural networks. By the end of our 60 minutes, you'll be able to perform these operations and, more importantly, understand why they are so crucial for transforming data within an AI model. We'll see how concepts you may have encountered in your computer science studies, like arrays and transformations, are given a powerful mathematical structure that allows us to build complex systems like the ones you're aiming to master.

Let's begin by exploring how we can represent transformations mathematically.

From Linear Transformations to Matrices

At its core, a neural network layer performs a series of geometric transformations on its input data. It might stretch, squash, or rotate the data to make it easier to understand. The most fundamental of these are linear transformations.

A key property of a linear transformation is that it keeps grid lines parallel and evenly spaced. For example, if you apply a linear transformation to a set of evenly spaced points on a line, they remain evenly spaced after the transformation.

Matrices give us a compact and powerful way to describe these linear transformations. To get a feel for this, please watch the first part of the following video. It uses a fun, practical example to show how a set of linear equations that transform coordinates can be neatly packaged into a matrix.

Essential Matrix Algebra for Neural Networks, Clearly Explained!!!

This section from StatQuest's 'Essential Matrix Algebra for Neural Networks' introduces linear transformations and how they are represented using matrix notation.

Starting around 2 minutes and 50 seconds in, watch the linear transformation. Focus on how the two equations for rotating the stage are converted into a row vector (for the coordinates) and a 2x2 matrix (for the transformation coefficients).

As you saw, a vector can be thought of as a point in space (like [x, y]) or simply as a list of numbers. We often write them as row vectors (1 row, n columns) or column vectors (n rows, 1 column). A matrix is a rectangular grid of numbers, arranged in rows and columns, that defines a specific linear transformation.

The Core Operation: Matrix Multiplication

Now that we can represent inputs as vectors and transformations as matrices, how do we apply the transformation? The answer is matrix multiplication. The way it's calculated might seem unusual at first, but it's specifically designed to correctly apply the transformation represented by the matrix to the vector.

Let's continue with the StatQuest video to understand the mechanics and, crucially, the logic behind matrix multiplication.

Essential Matrix Algebra for Neural Networks, Clearly Explained!!!

This part of the video explains the 'row-by-column' mechanics of matrix multiplication. Pay close attention to the reason why it's done this way: to combine sequential transformations.

About seven minutes in, watch matrix multiplication mechanics. The key insight here is that multiplying two transformation matrices results in a new matrix that represents the combined effect of both transformations. This is a fundamental property that allows neural networks to stack layers.

Key Rules of Matrix Multiplication

Let's formalize what you just saw. When we multiply matrix $A$ by matrix $B$ to get $C = A B$ , the element at row $i$ and column $j$ of $C$ , denoted $C_{ij}$ , is calculated by taking the dot product of the $i$ -th row of $A$ and the $j$ -th column of $B$ .

The dot product of two vectors is the sum of the products of their corresponding elements. For vectors $u = [u_{1}, u_{2}, ..., u_{n}]$ and $v = [v_{1}, v_{2}, ..., v_{n}]$ , their dot product is:
$u \cdot v = u_{1} v_{1} + u_{2} v_{2} + \dots + u_{n} v_{n}$

Dimensionality Rule:
For matrix multiplication $A B$ to be possible, the number of columns in matrix $A$ must equal the number of rows in matrix $B$ . If $A$ has dimensions $(m \times n)$ and $B$ has dimensions $(n \times p)$ , the resulting matrix $C$ will have dimensions $(m \times p)$ .

This diagram illustrates the dimensionality rule for matrix multiplication. An (m x n) matrix multiplied by an (n x p) matrix yields an (m x p) matrix. The inner dimensions (n) must match.

Notice that matrix multiplication is not commutative, meaning in general, $A B \neq = B A$ .

Test your understanding!

Given a vector $v = [2 - 1]$ and a matrix $W = [30 1 - 2]$ .

Calculate the product $W v$ .
Can you calculate the product $v W$ ? Why or why not?

Show answer

To calculate $W v$ , we treat $v$ as a $2 \times 1$ matrix. The matrix $W$ is $2 \times 2$ . The inner dimensions match (2 and 2), so the multiplication is valid. The result will be a $2 \times 1$ matrix.
$W v = [30 1 - 2] [2 - 1] = [(3 \times 2) + (1 \times - 1) (0 \times 2) + (- 2 \times - 1)] = [6 - 1 0 + 2] = [52]$
You cannot calculate $v W$ . The dimensions of $v$ are $2 \times 1$ and the dimensions of $W$ are $2 \times 2$ . The inner dimensions (1 and 2) do not match. To make this work, you would first need to transpose the vector $v$ .

The Matrix Transpose

As hinted at in the exercise, sometimes we need to reshape our matrices to make multiplication possible. The transpose operation, denoted by a superscript $T$ , flips a matrix over its main diagonal. It switches the row and column indices. The transpose of an $m \times n$ matrix is an $n \times m$ matrix.

For example, if $v = [2 - 1]$ , then its transpose is $v^{T} = [2 - 1]$ . Now, the product $v^{T} W$ is valid.

The next short video segment visually explains the transpose and introduces common notation you will see in research papers and deep learning framework documentation.

Essential Matrix Algebra for Neural Networks, Clearly Explained!!!

This section explains the matrix transpose and its notation.

Starting about 14 and a half minutes in, watch the transposition section. Pay attention to how transposing changes the dimensions of a matrix and enables different orderings in multiplication.

Application: A Simple Neural Network Layer

Now, let's put it all together. How is this directly used in a neural network? A basic "fully connected" or "dense" layer in a neural network does exactly what we've been discussing: it performs a linear transformation on its input, followed by adding a bias term.

The calculation for a single layer looks like this:
$output = activation (x W + b)$

$x$ is the input vector.
$W$ is the weight matrix, containing the coefficients of the linear transformation. These are the parameters the network learns during training.
$b$ is the bias vector, which is added element-wise to the result of $x W$ . This allows the transformation to include a translation, making it more flexible.
activation(...) is a non-linear function (like ReLU) applied at the end. We'll cover activations in detail in Module 5. For now, just focus on the $x W + b$ part.

This next segment shows you this exact process in action.

Essential Matrix Algebra for Neural Networks, Clearly Explained!!!

Here, you'll see how matrix multiplication and addition are used to represent the computations within a simple neural network.

About 18 and a half minutes in, watch the network demonstration. This is the most important part of the lesson. Trace how the input data (petal and sepal width) is transformed by multiplying it with the weight matrix and then adding the bias vector. This is the 'forward pass' for a single layer.

This sequence—matrix multiplication followed by vector addition—is the fundamental computation that happens over and over again, in every layer of a deep neural network. The vast computational power of GPUs is harnessed to perform these simple operations on massive matrices at incredible speeds.

A Deeper Look: The Dot Product and Duality

We've seen that the dot product is the "atomic" operation inside matrix multiplication. But it has a beautiful geometric meaning of its own that is crucial for building intuition.

The dot product between two vectors $v$ and $w$ can be interpreted as projecting one vector onto the other, and then multiplying their lengths.

Dot products and duality | Chapter 9, Essence of linear algebra

This video from 3Blue1Brown provides a wonderful geometric intuition for the dot product.

Starting 43 seconds in, watch the dot product explanation. Focus on the concept of 'projection' and how the sign of the dot product tells you whether the vectors point in similar or opposite directions.

This geometric view is useful, but there's an even deeper concept at play called duality. Duality, in this context, reveals a profound connection: a vector is not just an arrow in space; it can also be seen as the physical embodiment of a linear transformation that takes space onto a 1D number line.

Applying this transformation is computationally identical to taking a dot product with that vector. This is a subtle but powerful idea. When you see a neuron's weight vector, you can think of it as defining a specific projection—a transformation that it applies to its input data.

The next segment from the 3Blue1Brown video explains this concept of duality. It's more abstract, but given your background, it should provide a deeper appreciation for the role of these operations.

Dot products and duality | Chapter 9, Essence of linear algebra

This part of the video introduces the concept of duality, linking vectors to linear transformations. This explains why the numerical dot product operation is equivalent to geometric projection.

Watch the duality explanation. The core idea is that any linear transformation from 2D space to a 1D number line can be uniquely associated with a 2D vector, and applying that transformation is the same as taking a dot product with that vector.

Conclusion

In this lesson, we've laid the first stone in our foundation for understanding AI.

Key Takeaways:

Neural networks use matrices to perform linear transformations on input data.
Matrix multiplication is the core operation for applying these transformations. It's designed to compose sequential transformations, which is what happens when data flows through multiple network layers. The dimensions must align: $(m \times n) \times (n \times p) \to (m \times p)$ .
The computation within a basic neural network layer is $output = activation (input \times weights + bias)$ .
The dot product is the building block of matrix multiplication. It has a dual nature: it's both a numerical calculation and a geometric projection, which can be understood as a linear transformation from a vector space to the number line.

You've now seen how the elegant language of linear algebra provides the tools to build and understand neural networks.

Preview of the next lesson:

Today we saw how matrices transform vectors. In the next lesson, we'll explore a fascinating question: are there any vectors that a given matrix doesn't change the direction of, only its length? These special vectors are called eigenvectors, and their corresponding scaling factors are eigenvalues. They reveal deep properties of transformations and are fundamental to many machine learning techniques, including dimensionality reduction methods like PCA, which we'll cover later in the course.