Vector and Matrix Operations for Machine Learning Models

Welcome. This first module builds the mathematical vocabulary behind nearly every ML model: vectors represent features or activations; matrices represent datasets, weight layers, and linear transformations; tensor operations carry computations across batches.

In this lesson, you will compute the essential vector and matrix operations and, just as importantly, track their shapes. For ML interviews and production debugging alike, most linear-algebra errors reduce to one question: what is the shape of every input and output?

---

Shapes are the contract

A scalar is one number, such as a learning rate $\eta$. A vector is an ordered list of numbers, such as one example’s features:

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_d\end{array}\right]\in {\mathbb{R}}^d$$

A matrix is a rectangular array. If $A\in {\mathbb{R}}^{m\times n}$, it has $m$ rows and $n$ columns. Its entry in row $i$, column $j$ is $a_{ij}$.

In an ML setting, rows commonly represent examples and columns represent features:

$$X\in {\mathbb{R}}^{B\times d}$$

Here, $B$ is the batch size and $d$ is the number of input features. For example, $X\in {\mathbb{R}}^{32\times 128}$ means a batch of 32 examples, each represented by 128 features.

Mathematical orientation versus PyTorch shapes

Mathematically, vectors are often treated as column vectors. In PyTorch, a one-dimensional tensor has shape (d,); it is neither explicitly a row nor a column. The operation determines its role.

That distinction matters when reading equations:

ML object	Mathematical shape	Typical PyTorch shape	Meaning
One input example	$\mathbf{x}\in {\mathbb{R}}^d$	(d,)	$d$ features
Batch of inputs	$X\in {\mathbb{R}}^{B\times d}$	(B, d)	$B$ examples
Layer weights	$W\in {\mathbb{R}}^{d\times h}$	(d, h)	maps $d$ inputs to $h$ hidden units
Bias	$\mathbf{b}\in {\mathbb{R}}^h$	(h,)	one offset per hidden unit
Layer output	$H\in {\mathbb{R}}^{B\times h}$	(B, h)	hidden activations for a batch

A useful habit is to read each shape aloud:

$$X\in {\mathbb{R}}^{B\times d}$$

means “$B$ rows of examples, each with $d$ features.”

---

Elementwise operations and reductions

Some operations act independently on each corresponding entry. They do not mix information across positions.

For same-shaped vectors $\mathbf{u},\mathbf{v}\in {\mathbb{R}}^d$:

$$\mathbf{u}+\mathbf{v}=\left[\begin{array}{c}{u}_1+v_1\\ ⋮\\ u_d+v_d\end{array}\right]$$

For a scalar $c$:

$$c\mathbf{u}=\left[\begin{array}{c}c{u}_1\\ ⋮\\ c{u}_d\end{array}\right]$$

For same-shaped matrices $A,B\in {\mathbb{R}}^{m\times n}$, addition is also elementwise:

$$(A+B{)}_{ij}=a_{ij}+b_{ij}$$

The Hadamard product, written $A\odot B$, is elementwise multiplication:

$$(A\odot B{)}_{ij}=a_{ij}{b}_{ij}$$

This is not ordinary matrix multiplication. In PyTorch, A * B denotes elementwise multiplication.

For example,

$$\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]\odot \left[\begin{array}{cc}10& 1\\ 2& 5\end{array}\right]=\left[\begin{array}{cc}10& 2\\ 6& 20\end{array}\right]$$

Elementwise operations preserve the shape: two $m\times n$ matrices yield an $m\times n$ matrix.

A reduction combines values along an axis. Given a batch matrix $X\in {\mathbb{R}}^{B\times d}$:

• Summing with dim=0 collapses the batch axis, yielding one total per feature: shape (d,).
• Summing with dim=1 collapses the feature axis, yielding one total per example: shape (B,).
• Summing all entries yields one scalar.

This is fundamental in training code. For example, a loss may be computed per example, then reduced with a mean over the batch to obtain one scalar for backpropagation.

Broadcasted bias: a controlled exception

In a neural-network layer, you will often see:

$$H=XW+\mathbf{b}$$

Suppose $XW$ has shape $(B,h)$ and $\mathbf{b}$ has shape $(h,)$. Strictly speaking, these shapes differ. PyTorch applies broadcasting: it treats $\mathbf{b}$ as if it were copied across all $B$ rows.

Conceptually:

$$\left[\begin{array}{ccc}-& \text{example 1 output}& -\\ -& \text{example 2 output}& -\\ ⋮& & \\ -& \text{example B output}& -\end{array}\right]+\left[\begin{array}{c}{\mathbf{b}}^{\top }\\ {\mathbf{b}}^{\top }\\ ⋮\\ {\mathbf{b}}^{\top }\end{array}\right]$$

Broadcasting is convenient, but it should never replace a shape check. Know which dimensions are being aligned and which are being repeated.

---

Dot products: turning two vectors into one number

The dot product takes two equal-length vectors and returns a scalar:

$${\mathbf{x}}^{\top }\mathbf{w}=\sum _{j=1}^d{x}_j{w}_j$$

For example:

$${\left[\begin{array}{c}2\\ -1\\ 3\end{array}\right]}^{\top }\left[\begin{array}{c}0.5\\ 4\\ -2\end{array}\right]=(2)(0.5)+(-1)(4)+(3)(-2)=1-4-6=-9$$

The key shape rule is:

$${\mathbb{R}}^d\cdot {\mathbb{R}}^d\to \mathbb{R}$$

In ML, a dot product is a weighted sum. A linear model with input $\mathbf{x}$, weights $\mathbf{w}$, and bias $b$ computes:

$$z={\mathbf{x}}^{\top }\mathbf{w}+b$$

The input contributes one value per feature; the model weights determine how strongly each feature contributes to the score.

The dot product has geometric meaning as well, but for now treat it operationally: multiply matching entries, then sum. In the next lesson, you will interpret it through lengths, angles, similarity, and projection.

Do not confuse three products

These are frequent interview and implementation traps:

Operation	Inputs	Output	Meaning
Dot product ${\mathbf{u}}^{\top }\mathbf{v}$	two length-$d$ vectors	scalar	multiply matching entries, then sum
Outer product $\mathbf{u}{\mathbf{v}}^{\top }$	length-$m$, length-$n$ vectors	$m\times n$ matrix	every entry $u_i{v}_j$
Hadamard product $A\odot B$	same-shaped matrices	same-shaped matrix	multiply matching entries
Matrix product $AB$	$m\times n$, $n\times p$ matrices	$m\times p$ matrix	row-column dot products

For instance, if

$$\mathbf{u}=\left[\begin{array}{c}1\\ 2\end{array}\right],\mathbf{v}=\left[\begin{array}{c}3\\ 4\\ 5\end{array}\right]$$

then the outer product is:

$$\mathbf{u}{\mathbf{v}}^{\top }=\left[\begin{array}{ccc}3& 4& 5\\ 6& 8& 10\end{array}\right]$$

Its shape is $2\times 3$. Unlike the dot product, the input vectors do not need the same length.

---

Matrix-vector and matrix-matrix multiplication

A matrix-vector product applies one linear transformation to one input vector:

$$A\mathbf{x}$$

If $A\in {\mathbb{R}}^{m\times n}$, then $\mathbf{x}$ must have $n$ entries:

$$A\in {\mathbb{R}}^{m\times n},\mathbf{x}\in {\mathbb{R}}^n⟹A\mathbf{x}\in {\mathbb{R}}^m$$

Each output entry is the dot product of one row of $A$ with $\mathbf{x}$.

Let

$$A=\left[\begin{array}{ccc}2& 1& -1\\ 0& 3& 2\end{array}\right],\mathbf{x}=\left[\begin{array}{c}0.5\\ -1\\ 2\end{array}\right]$$

Then:

$$A\mathbf{x}=\left[\begin{array}{c}(2)(0.5)+(1)(-1)+(-1)(2)\\ (0)(0.5)+(3)(-1)+(2)(2)\end{array}\right]=\left[\begin{array}{c}-2\\ 1\end{array}\right]$$

The input has three values, matching the three columns of $A$. The output has two values, matching the two rows of $A$.

This column-based view is worth retaining:

$$A\mathbf{x}=x_1{\mathbf{a}}_1+x_2{\mathbf{a}}_2+\cdots +x_n{\mathbf{a}}_n$$

where ${\mathbf{a}}_j$ is column $j$ of $A$. Thus, matrix-vector multiplication can be read in two complementary ways:

• Row view: each output value is a row-vector dot product.
• Column view: the output is a weighted combination of the matrix columns.

The row view is typically fastest for hand calculation. The column view is often better for understanding a matrix as a transformation.

Matrix-matrix multiplication

For:

$$A\in {\mathbb{R}}^{m\times n},B\in {\mathbb{R}}^{n\times p}$$

the product is:

$$AB\in {\mathbb{R}}^{m\times p}$$

The inner dimensions must match: $n$ columns of $A$, $n$ rows of $B$. The output keeps the outer dimensions: $m$ rows and $p$ columns.

Each entry is a row-column dot product:

$$(AB{)}_{ij}=\sum _{k=1}^n{a}_{ik}{b}_{kj}$$

Let:

$$A=\left[\begin{array}{ccc}3& 1& 0\\ 0& 2& 1\end{array}\right],B=\left[\begin{array}{cc}0& 4\\ 2& 2\\ 0& 1\end{array}\right]$$

The shapes are:

$$(2\times 3)(3\times 2)\to (2\times 2)$$

Computing all four entries:

$$AB=\left[\begin{array}{cc}3(0)+1(2)+0(0)& 3(4)+1(2)+0(1)\\ 0(0)+2(2)+1(0)& 0(4)+2(2)+1(1)\end{array}\right]=\left[\begin{array}{cc}2& 14\\ 4& 5\end{array}\right]$$

Matrix multiplication is generally not commutative:

$$AB\ne BA$$

In this example, $AB$ has shape $2\times 2$, while $BA$ would have shape $3\times 3$. They cannot be equal even before computing their entries.

---

The batch computation behind a dense layer

A dense neural-network layer applies the same transformation to every example in a batch:

$$H=XW+\mathbf{b}$$

with:

$$X\in {\mathbb{R}}^{B\times d},W\in {\mathbb{R}}^{d\times h},\mathbf{b}\in {\mathbb{R}}^h$$

The matrix product is:

$$(B\times d)(d\times h)\to (B\times h)$$

So the layer maps a batch of $B$ examples with $d$ input features into $h$ output features per example.

The following PyTorch code makes each operation explicit:

import torch

# Two examples, each with three input features
X = torch.tensor([
    [2.0, 1.0, -1.0],
    [0.0, 3.0,  2.0],
])                           # shape: (2, 3)

# One linear model's weights
w = torch.tensor([0.5, -1.0, 2.0])  # shape: (3,)

# One score per example: two row-vector dot products
scores = X @ w                       # shape: (2,)
# tensor([-2., 1.])

# A layer with four output features
W = torch.randn(3, 4)                # shape: (3, 4)
b = torch.zeros(4)                   # shape: (4,)

H = X @ W + b                        # shape: (2, 4)

assert X.shape[1] == W.shape[0]      # inner dimensions match
assert H.shape == (2, 4)

feature_totals = X.sum(dim=0)        # shape: (3,)
example_totals = X.sum(dim=1)        # shape: (2,)

Three operational rules are especially worth memorizing:

1. X * W means elementwise multiplication and requires compatible elementwise shapes.
2. X @ W means matrix multiplication and requires matching inner dimensions.
3. X.sum(dim=0) and X.sum(dim=1) mean different things because they reduce different axes.

When debugging a model, write the expected shape beside each intermediate tensor before changing code. This catches incorrect transposes, accidental reductions, and wrong weight layouts faster than inspecting numerical values.

---

The central ideas are now in place:

• Shapes describe the valid inputs and outputs of every operation.
• Elementwise operations preserve structure; reductions collapse chosen axes.
• Dot products produce weighted sums.
• Matrix-vector multiplication applies a transformation to one vector.
• Matrix-matrix multiplication composes transformations or processes many vectors at once.
• A dense layer is the compact expression $H=XW+\mathbf{b}$, with shape flow $(B\times d)(d\times h)\to (B\times h)$.

Next, you will go beyond calculation and interpret dot products geometrically through vector length, angle, cosine similarity, and projection—concepts that reappear in embeddings, retrieval, attention, and optimization.