Dot and Matrix Products: Computing and Checking Dimensions

Welcome back. Previously, you learned to read tensor shapes as contracts: a vector of shape (d,), a matrix of shape (m, n), and speech features often organized as (batch, frames, features).

This lesson turns those contracts into computations. You will learn to calculate dot products, matrix-vector products, and matrix-matrix products, while predicting their output shapes before doing any arithmetic. That habit is essential when connecting feature extractors, neural layers, and decoders in speech systems.

---

Dot products: combining two equal-length vectors

A dot product takes two vectors of the same length and produces one scalar. If

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_d\end{array}\right]\in {\mathbb{R}}^d$$

and

$$\mathbf{w}=\left[\begin{array}{c}{w}_1\\ w_2\\ ⋮\\ w_d\end{array}\right]\in {\mathbb{R}}^d,$$

then their dot product is

$${\mathbf{x}}^{\mathsf{T}}\mathbf{w}=\mathbf{x}\cdot \mathbf{w}=\sum _{i=1}^d{x}_i{w}_i.$$

In plain language:

1. Multiply corresponding entries.
2. Add all of those products.
3. The result is one number.

For example,

$$\mathbf{x}=\left[\begin{array}{c}0.2\\ -0.1\\ 0.4\end{array}\right],\mathbf{w}=\left[\begin{array}{c}1.5\\ 2.0\\ -0.5\end{array}\right].$$

Both vectors have length three, so the dot product is valid:

$$\mathbf{x}\cdot \mathbf{w}=(0.2)(1.5)+(-0.1)(2.0)+(0.4)(-0.5).$$ $$\mathbf{x}\cdot \mathbf{w}=0.3-0.2-0.2=-\mathrm{0.1.}$$

The output has shape (): it is a scalar.

A dot product is often a weighted score. The entries of $\mathbf{x}$ are inputs or features; the entries of $\mathbf{w}$ state how each feature contributes to the score. Positive weights increase it, negative weights decrease it, and larger magnitudes matter more.

Compatibility rule for dot products

The dot product requires vectors with the same number of components:

$$\mathbf{x}\in {\mathbb{R}}^d,\mathbf{w}\in {\mathbb{R}}^d.$$

For example:

First vector	Second vector	Valid?	Output
(80,)	(80,)	Yes	scalar ()
(3,)	(4,)	No	undefined
(160,)	(160,)	Yes	scalar ()

This rule is not a superficial programming constraint. If one vector has 80 log-mel features and the other has 64 learned weights, there is no meaningful way to pair every input feature with exactly one weight.

In NumPy, use @ for the linear-algebra operation:

import numpy as np

x = np.array([0.2, -0.1, 0.4])
w = np.array([1.5, 2.0, -0.5])

score = x @ w

print(score)        # -0.1
print(score.shape)  # ()

You can also write np.dot(x, w) for two vectors. Prefer @ as a default in machine-learning code: it clearly signals that you intend a linear-algebra product.

Do not confuse this with elementwise multiplication:

print(x * w)  # [ 0.3 -0.2 -0.2], shape (3,)
print(x @ w)  # -0.1, shape ()

x * w keeps one value per feature. x @ w combines the features into one score.

---

Matrix-vector products: one score per matrix row

A matrix-vector product generalizes the dot product. Let

$$\mathbf{A}\in {\mathbb{R}}^{m\times n}$$

and

$$\mathbf{x}\in {\mathbb{R}}^n.$$

Then

$$\mathbf{Ax}\in {\mathbb{R}}^m.$$

The crucial compatibility condition is that the number of columns in $\mathbf{A}$ equals the number of entries in $\mathbf{x}$:

$$(m\times n)(n\times 1)$$

is valid, and the output has $m$ entries.

Consider

$$\mathbf{A}=\left[\begin{array}{ccc}1& -1& 2\\ 0& -3& 1\end{array}\right]\in {\mathbb{R}}^{2\times 3},\mathbf{x}=\left[\begin{array}{c}2\\ 1\\ 0\end{array}\right]\in {\mathbb{R}}^3.$$

The product is valid because the inner dimensions both equal three. Each output entry is the dot product of one row of $\mathbf{A}$ with $\mathbf{x}$:

$$\mathbf{Ax}=\left[\begin{array}{c}(1)(2)+(-1)(1)+(2)(0)\\ (0)(2)+(-3)(1)+(1)(0)\end{array}\right].$$ $$\mathbf{Ax}=\left[\begin{array}{c}1\\ -3\end{array}\right].$$

There are two rows in $\mathbf{A}$, so the result has two entries.

In NumPy, the corresponding shapes are:

A = np.array([
    [1, -1, 2],
    [0, -3, 1],
])

x = np.array([2, 1, 0])

y = A @ x

print(A.shape)  # (2, 3)
print(x.shape)  # (3,)
print(y.shape)  # (2,)
print(y)        # [ 1 -3]

Mathematics conventionally treats $\mathbf{x}$ as a column vector. NumPy represents it as a one-axis array with shape (3,), but the compatibility reasoning is the same: the vector must supply one value for every column of the matrix.

In a speech model, a single frame might contain $M$ acoustic features:

$$\mathbf{x}\in {\mathbb{R}}^M.$$

A learned weight matrix could contain $D$ feature detectors:

$$\mathbf{W}\in {\mathbb{R}}^{D\times M}.$$

Then

$$\mathbf{Wx}\in {\mathbb{R}}^D.$$

Each row of $\mathbf{W}$ computes a different weighted score from the same input frame. Later, neural-network layers will add a bias and a nonlinear activation, but this matrix-vector product is the core computation.

---

Matrix products: rows meet columns

Matrix multiplication uses the same idea repeatedly. If

$$\mathbf{A}\in {\mathbb{R}}^{m\times n}$$

and

$$\mathbf{B}\in {\mathbb{R}}^{n\times p},$$

then the product $\mathbf{AB}$ is defined and has shape

$$\mathbf{AB}\in {\mathbb{R}}^{m\times p}.$$

The shape rule is:

$$(m\times n)(n\times p)\text{has output shape}m\times p.$$

The two matching $n$ values are the inner dimensions. They must match. The remaining $m$ and $p$ values are the outer dimensions, and they determine the shape of the output.

A useful memory rule is:

Inner dimensions must agree; outer dimensions remain.

For every position $(i,j)$ in the product matrix, take:

• row $i$ of the first matrix;
• column $j$ of the second matrix;
• their dot product.

Formally, if

$$\mathbf{C}=\mathbf{AB},$$

then

$$c_{ij}=\sum _{k=1}^n{a}_{ik}{b}_{kj}.$$

The index $k$ ranges over the shared inner dimension.

A complete matrix-product calculation

Let

$$\mathbf{A}=\left[\begin{array}{ccc}1& 2& 0\\ 0& 1& 3\end{array}\right]\in {\mathbb{R}}^{2\times 3}$$

and

$$\mathbf{B}=\left[\begin{array}{cc}1& -1\\ 0.5& 2\\ 2& 0\end{array}\right]\in {\mathbb{R}}^{3\times 2}.$$

The product is valid because the shared inner dimension is three. Its output shape must be

$$(2\times 3)(3\times 2)=2\times 2.$$

Compute each entry:

$$\mathbf{AB}=\left[\begin{array}{cc}(1)(1)+(2)(0.5)+(0)(2)& (1)(-1)+(2)(2)+(0)(0)\\ (0)(1)+(1)(0.5)+(3)(2)& (0)(-1)+(1)(2)+(3)(0)\end{array}\right].$$ $$\mathbf{AB}=\left[\begin{array}{cc}2& 3\\ 6.5& 2\end{array}\right].$$

Notice the structure:

• The first row of the output came from the first row of $\mathbf{A}$.
• The second column of the output came from the second column of $\mathbf{B}$.
• Every output entry required a dot product of length three.

In NumPy:

A = np.array([
    [1, 2, 0],
    [0, 1, 3],
])

B = np.array([
    [1.0, -1.0],
    [0.5, 2.0],
    [2.0, 0.0],
])

C = A @ B

print(C.shape)  # (2, 2)
print(C)
# [[2.  3. ]
#  [6.5 2. ]]

---

Shape checking before computation

Do the shape check before calculating a single entry. This prevents most multiplication mistakes.

For a proposed product $\mathbf{AB}$:

1. Write the shape of $\mathbf{A}$: $(m,n)$.
2. Write the shape of $\mathbf{B}$: $(r,p)$.
3. Compare $n$ and $r$, the inner dimensions.
4. If they match, the result has shape $(m,p)$.
5. If they do not match, matrix multiplication is undefined.

For example:

Proposed product	Compatible?	Output shape
(2, 3) @ (3, 5)	Yes	(2, 5)
(4, 80) @ (80, 256)	Yes	(4, 256)
(2, 3) @ (4, 2)	No	none
(12, 64) @ (64, 128)	Yes	(12, 128)

The total number of elements does not determine compatibility. A matrix of shape (2, 6) and one of shape (3, 4) both contain 12 values, but

$$(2\times 6)(3\times 4)$$

is invalid because six does not equal three.

You can make shape validation explicit in code:

def matrix_product_checked(a, b):
    if a.ndim != 2 or b.ndim != 2:
        raise ValueError("Expected two matrices.")

    m, n = a.shape
    r, p = b.shape

    if n != r:
        raise ValueError(
            f"Incompatible shapes: {a.shape} and {b.shape}. "
            f"Expected {n} columns in the first matrix "
            f"to equal {r} rows in the second matrix."
        )

    return a @ b

This is a simplified version of a useful engineering habit: validate a tensor contract at boundaries, rather than waiting for a distant failure or silently incorrect output.

---

Order matters

Ordinary number multiplication is commutative:

$$2\cdot 3=3\cdot 2.$$

Matrix multiplication generally is not commutative:

$$\mathbf{AB}\ne \mathbf{BA}.$$

Sometimes reversing the order is invalid. For example, if

$$\mathbf{A}\in {\mathbb{R}}^{2\times 3}$$

and

$$\mathbf{B}\in {\mathbb{R}}^{3\times 4},$$

then

$$\mathbf{AB}\in {\mathbb{R}}^{2\times 4}$$

is valid, but

$$\mathbf{BA}$$

is not valid because the inner dimensions would be four and two.

Even when both orders are valid, the outputs may have different shapes and values. If $\mathbf{A}$ has shape (2, 3) and $\mathbf{B}$ has shape (3, 2), then:

$$\mathbf{AB}\in {\mathbb{R}}^{2\times 2},$$

while

$$\mathbf{BA}\in {\mathbb{R}}^{3\times 3}.$$

They cannot be equal because they do not even have the same shape.

---

Why this appears constantly in speech ML

Suppose one utterance has been converted into a frame-level feature matrix:

$$\mathbf{X}\in {\mathbb{R}}^{F\times M},$$

where $F$ is the number of frames and $M$ is the number of features per frame. A typical early speech representation might have $M=80$ features per frame.

A learned projection matrix can have shape

$$\mathbf{W}\in {\mathbb{R}}^{M\times D},$$

where $D$ is a model’s internal representation width. The product is

$$\mathbf{Y}=\mathbf{XW}\in {\mathbb{R}}^{F\times D}.$$

The number of time frames $F$ remains unchanged; every frame has been transformed from $M$ input features into $D$ learned features.

For example:

$$(200\times 80)(80\times 256)=200\times 256.$$

This one shape calculation tells you several important things:

• Each of the 200 frames supplies 80 feature values.
• The learned matrix expects exactly 80 input features.
• The output preserves the 200-frame time structure.
• Each frame now has 256 learned values.

If a feature extractor produces (frames, 80) but a layer has weights shaped (64, 256), the product is invalid. That failure reflects a real interface mismatch: the model expects 64 features but receives 80.

For a batch of speech examples, libraries extend the same principle across an additional batch axis. The core operation remains unchanged: the feature width must match the appropriate weight dimension.

---

Key takeaways and next step

• A dot product takes two equal-length vectors and produces one scalar.
• A matrix-vector product has the form

$$(m\times n)(n\times 1)=m\times 1.$$

• A matrix product has the form

$$(m\times n)(n\times p)=m\times p.$$

• Each output element in a matrix product is a dot product between one row of the left matrix and one column of the right matrix.
• The inner dimensions must match; the outer dimensions determine the output shape.
• In NumPy, use @ for matrix and vector products. Do not confuse it with *, which performs elementwise multiplication.
• Matrix order matters: $\mathbf{AB}$ and $\mathbf{BA}$ can differ in validity, shape, and values.

Next, we will move from computing values to understanding how values change: derivatives, partial derivatives, and gradients.