Using the Chain Rule for Gradient Descent

Hello. In the previous lesson, you learned that the gradient of a scalar loss points uphill and that $-\nabla L$ gives the local downhill direction. The remaining practical question is: how do we compute the loss gradient when a model is a sequence of many operations?

This lesson answers that question with the chain rule and a computational graph. You will trace a forward pass through a one-neuron model, propagate derivatives backward to its parameters, and apply a gradient-descent update. This is the small-scale version of backpropagation used in every differentiable neural network.

---

From a nested expression to a computational graph

Consider a simple model for one training example:

$$z=wx+b,$$ $$a=\sigma (z),$$ $$L=\frac{1}{2}(a-y{)}^2.$$

Here:

• $x$ is an input feature and $y$ is its target;
• $w$ and $b$ are trainable weight and bias;
• $z$ is the pre-activation;
• $a$ is the prediction after the sigmoid activation;
• $L$ is a scalar loss.

Written as one expression, the loss is:

$$L(w,b)=\frac{1}{2}{\left(\sigma (wx+b)-y\right)}^2.$$

You could differentiate this directly. But neural networks make such expressions far too large to manipulate reliably as one formula. Instead, break the computation into local operations:

$$w,x\to u=wx\to z=u+b\to a=\sigma (z)\to L.$$

A computational graph represents exactly this decomposition:

• nodes are values, such as $u$, $z$, and $a$;
• edges/operations show how one value is computed from earlier values;
• the graph is evaluated left to right during the forward pass;
• gradients are computed right to left during the backward pass.

The image’s central idea is worth stating precisely:

A parameter influences the loss through the intermediate values between them. The chain rule multiplies the local sensitivities along that path.

For the weight $w$, the dependency path is:

$$w\to z\to a\to L.$$

Therefore,

$$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial a}\frac{\partial a}{\partial z}\frac{\partial z}{\partial w}.$$

For the bias,

$$\frac{\partial L}{\partial b}=\frac{\partial L}{\partial a}\frac{\partial a}{\partial z}\frac{\partial z}{\partial b}.$$

The loss-to-activation and activation-to-pre-activation factors are shared. Backpropagation exploits this reuse rather than deriving a separate expanded expression for every parameter.

---

The backward-pass rule: upstream gradient times local gradient

Suppose a graph node computes

$$r=g(q).$$

During backpropagation, imagine you already know how the final loss changes with $r$:

$$\frac{\partial L}{\partial r}.$$

This is called the upstream gradient arriving at $r$. The local operation tells you

$$\frac{\partial r}{\partial q}.$$

The chain rule gives the gradient passed back to $q$:

$$\frac{\partial L}{\partial q}=\frac{\partial L}{\partial r}\frac{\partial r}{\partial q}.$$

In words:

$$\text{gradient passed backward}=\text{upstream gradient}\times \text{local derivative}.$$

A convenient notation in implementations is

$$\bar{q}=\frac{\partial L}{\partial q}.$$

Then each node receives $\bar{r}$ and computes $\bar{q}$. Since the final output is the loss itself,

$$\bar{L}=\frac{\partial L}{\partial L}=1.$$

That is the seed of the backward pass.

Local derivative rules you should know

For scalar values, these common operations are enough to understand a basic network:

Forward operation	Local derivatives	Backward implication
$r=p+q$	$\frac{\partial r}{\partial p}=1,\frac{\partial r}{\partial q}=1$	Copy the upstream gradient to both inputs
$r=pq$	$\frac{\partial r}{\partial p}=q,\frac{\partial r}{\partial q}=p$	Multiply the upstream gradient by the other input
$r=p^2$	$\frac{\partial r}{\partial p}=2p$	Multiply by $2p$
$r=\sigma (p)$	$\frac{\partial r}{\partial p}=\sigma (p)(1-\sigma (p))$	Multiply by $r(1-r)$
$r=\frac{1}{2}(p-y{)}^2$	$\frac{\partial r}{\partial p}=p-y$	Prediction error is the upstream signal

Two details often cause errors in interviews and implementation:

1. The local derivative is evaluated using forward-pass values. This is why frameworks cache selected intermediate results.
2. A multiply gate sends different gradients to its two inputs. For $r=pq$, the gradient with respect to $p$ contains $q$, not $p$.

---

A full worked backward pass through one neuron

Let us calculate every relevant value for a one-feature classifier-like neuron. Use:

$$x=2,y=1,w=0.5,b=-1.$$

The model is:

$$u=wx,$$ $$z=u+b,$$ $$a=\sigma (z),$$ $$L=\frac{1}{2}(a-y{)}^2.$$

Forward pass

Compute each intermediate value in order:

$$u=wx=(0.5)(2)=1,$$ $$z=u+b=1+(-1)=0,$$ $$a=\sigma (0)=\frac{1}{1+e^0}=0.5,$$ $$L=\frac{1}{2}(0.5-1{)}^2=\mathrm{0.125.}$$

The prediction is $0.5$, below the target $1$, producing a loss of $0.125$.

Backward pass

Start at the scalar loss. Since $L$ is the output,

$$\frac{\partial L}{\partial L}=1.$$

Now work right to left.

1. Loss to prediction

Because

$$L=\frac{1}{2}(a-y{)}^2,$$ $$\frac{\partial L}{\partial a}=a-y=0.5-1=-\mathrm{0.5.}$$

The negative sign has a meaningful interpretation: increasing $a$ slightly would decrease the loss, which makes sense because the target is $1$.

2. Prediction to pre-activation

For sigmoid,

$$\frac{\partial a}{\partial z}=a(1-a).$$

At $a=0.5$,

$$\frac{\partial a}{\partial z}=0.5(1-0.5)=\mathrm{0.25.}$$

Apply the chain rule:

$$\frac{\partial L}{\partial z}=\frac{\partial L}{\partial a}\frac{\partial a}{\partial z}=(-0.5)(0.25)=-\mathrm{0.125.}$$

It is common to call this quantity the neuron’s error signal or backpropagated signal for the pre-activation.

3. Pre-activation to weight and bias

Recall:

$$z=wx+b.$$

For the weight,

$$\frac{\partial z}{\partial w}=x=2.$$

Thus,

$$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial z}\frac{\partial z}{\partial w}=(-0.125)(2)=-\mathrm{0.25.}$$

For the bias,

$$\frac{\partial z}{\partial b}=1,$$

so

$$\frac{\partial L}{\partial b}=\frac{\partial L}{\partial z}\frac{\partial z}{\partial b}=(-0.125)(1)=-\mathrm{0.125.}$$

The parameter gradient is therefore

$${\nabla }_{(w,b)}L=\left[\begin{array}{c}-0.25\\ -0.125\end{array}\right].$$

Notice the structure:

$$\frac{\partial L}{\partial w}=(a-y)\cdot a(1-a)\cdot x,$$ $$\frac{\partial L}{\partial b}=(a-y)\cdot a(1-a).$$

The weight gradient has one additional factor, $x$, because the weight affects $z$ through multiplication by the input. This relationship generalizes: for a linear layer, parameter gradients depend on both an error signal and the activation/input entering that parameter.

---

Gradient descent: turn the gradient into a parameter update

The gradient tells us the direction of greatest local increase in loss. To reduce loss, move in the opposite direction:

$$\mathbf{\theta }\leftarrow \mathbf{\theta }-\eta {\nabla }_{\mathbf{\theta }}L.$$

Here:

• $\mathbf{\theta }$ is the vector of trainable parameters;
• $\eta >0$ is the learning rate;
• ${\nabla }_{\mathbf{\theta }}L$ is the gradient of the loss with respect to those parameters.

For our two parameters,

$$\left[\begin{array}{c}w\\ b\end{array}\right]\leftarrow \left[\begin{array}{c}w\\ b\end{array}\right]-\eta \left[\begin{array}{c}\frac{\partial L}{\partial w}\\ \frac{\partial L}{\partial b}\end{array}\right].$$

Choose

$$\eta =\mathrm{0.4.}$$

Then:

$$w_{\text{new}}=0.5-0.4(-0.25)=0.6,$$ $$b_{\text{new}}=-1-0.4(-0.125)=-\mathrm{0.95.}$$

Both parameters increase because both gradient components were negative. This is not a separate rule; it follows directly from subtracting a negative number.

Check the resulting forward pass:

$$z_{\text{new}}=(0.6)(2)-0.95=0.25,$$ $$a_{\text{new}}=\sigma (0.25)\approx 0.562,$$ $$L_{\text{new}}=\frac{1}{2}(0.562-1{)}^2\approx \mathrm{0.096.}$$

The loss decreased:

$$0.125\to \mathrm{0.096.}$$

That is one training step:

1. make a prediction;
2. compute a scalar loss;
3. backpropagate to obtain gradients;
4. update parameters in the negative-gradient direction.

---

What changes when a value is used more than once?

So far, each intermediate value had one route to the final output. Real graphs often branch: one value influences the loss through multiple downstream paths.

For example:

$$q=x+y,$$ $$f=qx.$$

The variable $x$ appears twice: once directly in the multiplication and once indirectly through $q$. Expanding gives

$$f=(x+y)x=x^2+xy,$$

so

$$\frac{\partial f}{\partial x}=2x+y.$$

Backpropagation obtains the same result by adding contributions from both paths:

• direct path $x\to f$:

$$\frac{\partial f}{\partial x}{|}_{\text{direct}}=q;$$

• indirect path $x\to q\to f$:

$$\frac{\partial f}{\partial x}{|}_{\text{through }q}=\frac{\partial f}{\partial q}\frac{\partial q}{\partial x}=x\cdot 1=x.$$

Thus,

$$\frac{\partial f}{\partial x}=q+x=(x+y)+x=2x+y.$$

The rule is:

Multiply gradients along a single path; add gradients across distinct paths.

In software, this is why gradient accumulation must use addition. A parameter can affect many later operations, and overwriting rather than accumulating its gradient silently produces an incorrect update.

---

A disciplined manual procedure

For small graphs, use this sequence:

1. Identify parameters and data.
   Parameters are updated; input data and targets are held fixed for that training step.

2. Write the computation in stages.
   Introduce meaningful intermediates such as $z=wx+b$, $a=\sigma (z)$, and $L$.

3. Perform a forward pass.
   Evaluate and retain the values needed by local derivatives.

4. Start the backward pass from the scalar objective.
   Set $\frac{\partial L}{\partial L}=1$.

5. Apply upstream times local derivative at each operation.
   Move in reverse topological order: loss, activation, linear computation, parameters.

6. Accumulate gradients where the graph branches.

7. Update parameters simultaneously.
   Use the old parameter values and their corresponding gradients:

$${\theta }_i\leftarrow {\theta }_i-\eta \frac{\partial L}{\partial {\theta }_i}.$$

A useful interview-level explanation is:

“Backpropagation is reverse-mode automatic differentiation on a computational graph. A forward pass computes and caches intermediate values; a backward pass propagates the scalar loss gradient backward using the chain rule. Gradients multiply through each path and accumulate at branches, producing one derivative per trainable parameter. An optimizer then uses those gradients to update the parameters.”

A final caution: gradient descent guarantees a local first-order decrease only when the learning rate is appropriately small. With too large a learning rate, the linear approximation underlying the step is poor and the loss can rise or diverge. Later lessons will compare optimization methods that alter this basic update rule, but they all begin with the gradients obtained here.

---

Key takeaways

A computational graph turns a complex model into local operations. The forward pass computes values from inputs to loss; the backward pass computes derivatives from loss back to parameters.

For a chain of dependencies, multiply local derivatives:

$$\frac{\partial L}{\partial w}=\frac{\partial L}{\partial a}\frac{\partial a}{\partial z}\frac{\partial z}{\partial w}.$$

For a graph with multiple paths from a variable to the loss, add the gradient contributions from those paths.

Once backpropagation yields the parameter gradient, gradient descent updates parameters as:

$$\mathbf{\theta }\leftarrow \mathbf{\theta }-\eta {\nabla }_{\mathbf{\theta }}L.$$

In the next lesson, you will move from optimization mechanics to probability, using conditional probability and Bayes’ rule to update beliefs from evidence.