Calculating Gradients of Multivariable Functions

Welcome back. Last time, you interpreted a gradient as the complete first-order description of local change: it collects partial derivatives, points toward steepest increase, and has one component per input. Now we make that object concrete by calculating it.

For speech ML, this is the manual version of what PyTorch will later do automatically. A scalar training objective depends on many parameters; the gradient tells us the sensitivity of that objective to each one. The calculations here are deliberately small, but the pattern is the same at model scale.

---

The calculation rule: freeze all other variables

For a function of two variables,

$$f(x,y),$$

the gradient is

$$\nabla f(x,y)=\left[\begin{array}{c}\frac{\partial f}{\partial x}\\ \frac{\partial f}{\partial y}\end{array}\right].$$

To calculate $\frac{\partial f}{\partial x}$, treat $y$ as a constant and differentiate normally with respect to $x$. To calculate $\frac{\partial f}{\partial y}$, reverse those roles: treat $x$ as constant.

This is the essential operational rule:

Differentiate one variable at a time; temporarily regard every other variable as a constant.

For example, in the term

$$x^{2}y,$$

$y$ is a constant when differentiating with respect to $x$, so

$$\frac{\partial }{\partial x}(x^{2}y)=2xy.$$

But $x^2$ is a constant when differentiating with respect to $y$, so

$$\frac{\partial }{\partial y}(x^{2}y)=x^2.$$

You do not need the product rule for this term: in each partial derivative, only one factor is allowed to vary.

A small derivative reference is enough for the polynomial functions in this lesson:

Expression	Derivative with respect to its variable
constant	$0$
$x$	$1$
$x^n$	$n{x}^{n-1}$
$c{x}^n$	$cn{x}^{n-1}$
$\sin (x)$	$\cos (x)$
$\cos (x)$	$-\sin (x)$

Here, $c$ means a constant. During a partial derivative, another variable can play the role of $c$.

---

A full two-variable gradient calculation

Consider

$$f(x,y)=x^{2}y+3x{y}^2-5x+4.$$

We will calculate the gradient symbolically first, then evaluate it at a particular point. That order matters: the symbolic gradient tells us how sensitivity varies across the whole input space.

First component: differentiate with respect to $x$

Hold $y$ fixed.

$$\frac{\partial f}{\partial x}=\frac{\partial }{\partial x}\left(x^{2}y+3x{y}^2-5x+4\right).$$

Term by term:

$$\frac{\partial }{\partial x}(x^{2}y)=2xy,$$ $$\frac{\partial }{\partial x}(3x{y}^2)=3y^2,$$ $$\frac{\partial }{\partial x}(-5x)=-5,$$ $$\frac{\partial }{\partial x}(4)=0.$$

Therefore,

$$\frac{\partial f}{\partial x}=2xy+3y^2-5.$$

Notice that $3x{y}^2$ becomes $3y^2$, not $6xy$. While differentiating with respect to $x$, the entire quantity $3y^2$ is just a constant coefficient multiplying $x$.

Second component: differentiate with respect to $y$

Now hold $x$ fixed.

$$\frac{\partial f}{\partial y}=\frac{\partial }{\partial y}\left(x^{2}y+3x{y}^2-5x+4\right).$$

This time,

$$\frac{\partial }{\partial y}(x^{2}y)=x^2,$$ $$\frac{\partial }{\partial y}(3x{y}^2)=6xy,$$

and the two terms containing no $y$ vanish:

$$\frac{\partial }{\partial y}(-5x)=0,\frac{\partial }{\partial y}(4)=0.$$

So,

$$\frac{\partial f}{\partial y}=x^2+6xy.$$

Assemble the gradient

Place the partial derivatives in the same order as the inputs:

$$\nabla f(x,y)=\left[\begin{array}{c}2xy+3y^2-5\\ x^2+6xy\end{array}\right].$$

At the point $(2,-1)$,

$$\nabla f(2,-1)=\left[\begin{array}{c}2(2)(-1)+3(-1{)}^2-5\\ (2{)}^2+6(2)(-1)\end{array}\right].$$ $$\nabla f(2,-1)=\left[\begin{array}{c}-6\\ -8\end{array}\right].$$

This says that near $(2,-1)$:

• Increasing $x$, while keeping $y$ fixed, decreases $f$ at a local rate of about $6$.
• Increasing $y$, while keeping $x$ fixed, decreases $f$ at a local rate of about $8$.

The local direction of steepest increase is the gradient itself. The local direction of steepest decrease is its negative:

$$-\nabla f(2,-1)=\left[\begin{array}{c}6\\ 8\end{array}\right].$$

---

Three variables: one partial derivative per input

The procedure scales directly. For a scalar-valued function

$$g(x,y,z),$$

the gradient is

$$\nabla g(x,y,z)=\left[\begin{array}{c}\frac{\partial g}{\partial x}\\ \frac{\partial g}{\partial y}\\ \frac{\partial g}{\partial z}\end{array}\right].$$

Consider the three-parameter scalar objective

$$g(w_1,w_2,w_3)=w_1^2{w}_2-4w_2{w}_3^3+6w_3.$$

Although this is a toy expression, treat $w_1,w_2,w_3$ as three model parameters. Each partial derivative measures sensitivity to one parameter while freezing the other two.

With respect to $w_1$:

$$\frac{\partial g}{\partial w_1}=2w_1{w}_2.$$

The second and third terms contain no $w_1$, so they are constants and disappear.

With respect to $w_2$:

$$\frac{\partial g}{\partial w_2}=w_1^2-4w_3^3.$$

With respect to $w_3$:

$$\frac{\partial g}{\partial w_3}=-12w_2{w}_3^2+6.$$

Therefore,

$$\nabla g(w_1,w_2,w_3)=\left[\begin{array}{c}2w_1{w}_2\\ w_1^2-4w_3^3\\ -12w_2{w}_3^2+6\end{array}\right].$$

At

$$(w_1,w_2,w_3)=(1,2,-1),$$

we obtain

$$\nabla g(1,2,-1)=\left[\begin{array}{c}4\\ 5\\ -18\end{array}\right].$$

The gradient has three entries because the input has three entries. More generally, if

$$h(\mathbf{w})\in \mathbb{R},\mathbf{w}\in {\mathbb{R}}^d,$$

then

$${\nabla }_{\mathbf{w}}h\in {\mathbb{R}}^d.$$

A modern speech model has far more than three parameters, often organized into matrices and higher-dimensional tensors. Nevertheless, the principle remains unchanged: every parameter has a corresponding gradient entry, and the gradient tensor has the same shape as the parameter tensor.

---

A reliable workflow for manual gradients

When calculating a gradient by hand, use this routine:

1. Confirm the output is scalar. A gradient in this introductory sense is associated with a scalar-valued function, such as a loss or objective.

2. List the input variables in a fixed order. For example, $(x,y,z)$ or $(w_1,w_2,w_3)$. This is the order of gradient components.

3. Take one partial derivative at a time. Circle the active variable mentally; freeze every other variable.

4. Differentiate term by term. Any term without the active variable becomes zero.

5. Assemble the results into a vector. Do not accidentally place $\frac{\partial f}{\partial y}$ in the first slot.

6. Evaluate at a point only after forming the symbolic gradient. This avoids losing the function-level view of the calculation.

A quick numerical check can help catch sign and coefficient errors. For a small number $h$,

$$\frac{\partial f}{\partial x}(x,y)\approx \frac{f(x+h,y)-f(x,y)}{h}.$$

For the earlier function at $(2,-1)$, the exact first partial derivative is

$$\frac{\partial f}{\partial x}(2,-1)=-6.$$

Using a small positive $h$, the finite-difference estimate should be close to $-6$. This is only a sanity check; symbolic differentiation is exact for the functions studied here. Later, automatic differentiation will compute exact derivatives of the operations in a model’s computation graph rather than estimate them numerically.

---

Common mistakes to eliminate early

Mistake	Why it is wrong	Correction
Differentiating every variable at once	A partial derivative permits only one active variable.	Freeze all non-active variables.
Keeping a term that lacks the active variable	It is a constant with respect to that variable.	Its derivative is $0$.
Dropping another variable entirely	A frozen variable is still a coefficient.	Keep it in the result, such as $x^{2}y\mapsto 2xy$ with respect to $x$.
Using the wrong gradient order	Components must align with the original input order.	Write the variable ordering explicitly before assembling the vector.
Confusing $\nabla f$ with $\nabla f({\mathbf{x}}_0)$	The first is a vector-valued function; the second is a specific vector.	Form the symbolic gradient, then substitute the point.

In ML code, these are not merely notation issues. A gradient assigned to the wrong parameter, reshaped incorrectly, or interpreted with the wrong sign can make optimization fail even when the model architecture is otherwise sound.

---

Key takeaways

• To calculate a partial derivative, differentiate with respect to one variable and treat all others as constants.
• The gradient of a scalar function is the vector of its partial derivatives, listed in input order.
• For

$$f(x,y),$$

the gradient has two components; for

$$g(x,y,z),$$

it has three; in general it has one component per input dimension.

• A term that does not contain the active variable contributes zero to that partial derivative.
• Build the symbolic gradient first, then evaluate it at a particular point.
• This manual procedure is the foundation for interpreting the parameter gradients that PyTorch will eventually produce.

Next, you will handle functions built from several nested operations. The chain rule will show how gradients pass through those compositions—the essential mechanism behind backpropagation in neural networks.