Partial Derivatives and Gradients

Hello! Welcome to your next lesson in the "Mathematical and Statistical Foundations for AI" module.

In our last lesson, we explored how matrix decomposition techniques like Eigendecomposition and SVD allow us to understand the fundamental structure of linear transformations. We concluded with the idea that while linear algebra provides the static "scaffolding" for AI models, the process of learning requires a different set of tools from calculus.

Today, we'll dive into those tools. This lesson covers how to compute partial derivatives and the gradient of multivariate functions. Your goal is to understand how we can measure the "slope" of a function with many inputs—like a neural network's loss function, which can depend on millions or even billions of parameters. This "slope," known as the gradient, is the single most important piece of information we need to train AI models, as it tells us exactly how to adjust our parameters to improve performance.

1. From Slopes to Slices: Introducing Partial Derivatives

From single-variable calculus, you'll recall that the derivative $f^{\prime }(x)$ gives us the slope of the tangent line to a curve at any point $x$. It tells us how the function's output changes for a tiny change in its input.

But what happens when a function has multiple inputs, like $f(x,y)$? This function describes a surface in 3D space. At any point on this surface, there isn't just one "slope"; there are infinitely many, depending on the direction you move.

The core idea of multivariable calculus is to simplify this problem by asking a more constrained question: What is the slope if we only move along one axis at a time, keeping all other variables fixed? This is the concept of a partial derivative.

As the video illustrates, by holding $y$ constant, we are effectively taking a 2D slice of the 3D surface. Within that slice, we have a simple curve whose slope we can find using standard single-variable differentiation rules.

The Calculation

To compute a partial derivative, you simply treat all variables—except the one you're differentiating with respect to—as constants.

Let's use the function from the video, $f(x,y)=z=x^2-y^2$.

To find the partial derivative with respect to $x$:
We treat $y$ as a constant. The derivative of $x^2$ with respect to $x$ is $2x$. The derivative of $y^2$ (a constant) with respect to $x$ is 0.
So, the partial derivative of $f$ with respect to $x$ is:
$\frac{\partial f}{\partial x}=2x-0=2x$
The symbol $\partial$ (a curly 'd', often pronounced "del") is used to denote a partial derivative, distinguishing it from the ordinary derivative $d$.

To find the partial derivative with respect to $y$:
Now, we treat $x$ as a constant. The derivative of $x^2$ (a constant) with respect to $y$ is 0. The derivative of $-y^2$ with respect to $y$ is $-2y$.
So, the partial derivative of $f$ with respect to $y$ is:
$\frac{\partial f}{\partial y}=0-2y=-2y$

This process extends to functions with any number of variables. For $f(w,x,y,z)$, to find $\frac{\partial f}{\partial w}$, you treat $x,y,$ and $z$ as constants.

2. The Gradient: A Vector of Slopes

We can now calculate the slope of our function along each axis. But what if we want a complete picture of how the function changes at a point? We can combine all the partial derivatives into a single vector called the gradient.

For a function $f(x_1,x_2,\dots ,x_n)$, the gradient, denoted $\nabla f$ (nabla f), is the vector of its partial derivatives:
$\nabla f=\left[\frac{\partial f}{\partial x_1},\frac{\partial f}{\partial x_2},\dots ,\frac{\partial f}{\partial x_n}\right]$

The gradient is more than just a collection of partial derivatives; it has profound geometric meaning.

• It is a vector that points in the direction of the steepest ascent of the function at a given point.
• Its magnitude, $\Vert \nabla f\Vert$, represents the rate of change in that steepest direction.

This is the key insight for machine learning. If $\nabla f$ points "uphill," then $-\nabla f$ must point in the direction of steepest descent—the most efficient way to go "downhill."

This is the mathematical foundation of gradient descent, the workhorse optimization algorithm for training almost all modern AI models. We calculate the gradient of the loss function with respect to the model's parameters (weights), and then take a small step in the opposite direction of the gradient to update those parameters.

The text formalizes the gradient descent update rule:
$\mathbf{w}\to \mathbf{w}-ϵ{\nabla }_{\mathbf{w}}L(\mathbf{w})$
Here, $\mathbf{w}$ is the vector of our model's weights, $L$ is the loss function, ${\nabla }_{\mathbf{w}}L(\mathbf{w})$ is the gradient of the loss with respect to the weights, and $ϵ$ (epsilon) is a small positive number called the learning rate.

3. Visualizing Gradients in Python

Let's ground these concepts in code. Your proficiency in Python makes it the perfect tool to bridge the gap between abstract formulas and concrete results. We'll revisit the function $f(x,y)=x^2-y^2$ and use Python to not only calculate its partial derivatives but also to visualize the tangent lines whose slopes they represent.

The video by Jon Krohn includes a detailed code walkthrough. I'll summarize the key logic here, but you can refer to the video for a line-by-line explanation.

Core Logic of the Python Demo

1. Define the function and its partial derivatives:

   import numpy as np
   
   # The multivariate function
   def f(x, y):
       return x**2 - y**2
   
   # The partial derivative of f with respect to x
   def del_f_del_x(x, y):
       return 2*x
   
   # The partial derivative of f with respect to y
   def del_f_del_y(x, y):
       return -2*y
   

2. Analyze the partial derivative with respect to x:

   • To visualize the slice where $y$ is constant, we can fix y=0.
   • We can then plot $z=f(x,0)=x^2$, which is a simple parabola.
   • At any point on this parabola, say x = -1, the slope of the tangent line should be given by our partial derivative function: del_f_del_x(-1, 0) = 2*(-1) = -2.
   • We can plot this tangent line to visually confirm our calculation.

3. Analyze the partial derivative with respect to y:

   • Similarly, to visualize the slice where $x$ is constant, we fix x=0.
   • We plot $z=f(0,y)=-y^2$, which is an inverted parabola.
   • At any point, say y = 1, the slope is del_f_del_y(0, 1) = -2*1 = -2.

The full code in the video generates plots that show these curves and their tangent lines, providing a powerful visual confirmation of the math.

Conclusion

Today, we've taken a crucial first step into the calculus that powers machine learning. You now have the tools to analyze how a function with many variables changes.

Key Takeaways:

• A partial derivative measures the instantaneous rate of change of a multivariate function along one specific axis, holding all other variables constant.
• The gradient, $\nabla f$, is a vector containing all the partial derivatives. It points in the direction of the function's steepest increase.
• The negative gradient, $-\nabla f$, points in the direction of steepest decrease. This is the core principle behind the gradient descent algorithm, which is used to minimize loss functions and train AI models.

Preview of the next lesson:
The functions we dealt with today were relatively simple. Neural networks, however, are deeply nested, composite functions. For example, the loss is a function of the final activation, which is a function of the pre-activation sum, which is a function of the weights and the previous layer's activations, and so on.

To calculate the gradient of the loss with respect to a weight deep inside the network, we can't just apply simple derivative rules. We need a systematic way to handle these nested functions. This method is the multivariate chain rule, and its algorithmic implementation in deep learning is the famous backpropagation algorithm. In the next lesson, we will derive and apply the chain rule, which will prepare us to build a neural network from scratch.