Every time a neural network trains, an optimizer quietly asks one question millions of times: "If I nudge this weight a little, does the loss go up or down, and by how much?" The answer is a derivative. Derivatives, and their multi-dimensional generalization, the gradient, are the machinery that turns a single loss number into a concrete instruction for how to change every parameter. Learn them once and gradient descent, backpropagation, and learning rates stop feeling like magic.
A derivative is a slope
A derivative measures the instantaneous rate of change of a function: the slope of its graph at one exact point. Formally it is the limit of the average rate of change as the measuring interval shrinks to zero:
f′(x)=lim
Ask the tutor
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
h→0
hf(x+h)−f(x)
The sign is what matters for learning. If f′(x)>0 the function is rising; if f′(x)<0 the function is falling; if it equals zero the curve is momentarily flat, a peak, a valley, or a saddle.
Worked example: differentiate f(x)=x2
Using the power rule dxdxn=nxn−1, we get f′(x)=2x. At x=3 the slope is:
f′(3)=2×3=6
That number is a promise about the neighborhood of x=3: nudging x upward by a tiny Δx raises f by roughly 6Δx. Check it numerically, f(3)=9 and f(3.01)=9.0601, a rise of 0.0601 over a step of 0.01, which works out to 6.01, essentially the exact slope of 6.
Partial derivatives and the gradient
Real models take many inputs, not one. A partial derivative measures the slope with respect to a single variable while all the others are held fixed. For the two-input bowl f(x,y)=x2+y2:
∂x∂f=2x,∂y∂f=2y
Stack every partial derivative into a vector and you have the gradient:
∇f(x,y)=[∂x∂f,∂y∂f]=[2x,2y]
The gradient has one indispensable property: it points in the direction of steepest increase. Its negative, −∇f, therefore points in the direction of steepest decrease, exactly the direction you want when minimizing a loss.
Derivative (one variable)
Input is a single number x
Output is one slope, f'(x)
Sign says increasing or decreasing
Example: f'(x) = 2x
Gradient (many variables)
Input is a vector of parameters
Output is a vector of partial derivatives
Direction points toward steepest increase
Example: ∇f = [2x, 2y]
Which way is downhill?
The surface f(x,y)=x2+y2 is a bowl with its minimum at the origin. Evaluate the gradient at the point (3,4):
∇f(3,4)=[2×3,2×4]=[6,8]
The vector [6,8] points uphill, away from the bottom of the bowl. Downhill is the opposite direction, [−6,−8], which aims straight back toward the origin where the minimum sits. That is the whole idea behind training: repeatedly follow the negative gradient.
1Measure the lossAt (3, 4) the loss is f = 9 + 16 = 25.
2Compute the gradient∇f(3, 4) = [6, 8], the direction of steepest increase.
3Flip it for descentThe descent direction is the negative gradient: -∇f = [-6, -8].
4Scale by the learning rateWith learning rate η = 0.1, the update is 0.1 × [-6, -8] = [-0.6, -0.8].
5Take the stepNew point = (3, 4) + [-0.6, -0.8] = (2.4, 3.2).
6Confirm progressNew loss f(2.4, 3.2) = 5.76 + 10.24 = 16, down from 25.
Because f(2.4,3.2)=16 is less than f(3,4)=25, the step moved us genuinely downhill. Repeat this loop thousands of times, across millions of parameters, and the values converge toward the minimum, that is how a model learns.
Backpropagation is just the chain rule
Deep networks are functions composed of many simpler functions.
Backpropagation computes the gradient of the loss with respect to every weight
by applying the chain rule layer by layer, from the output back to the input.
Every gradient it produces obeys the same downhill logic shown above.
Common mistakes
Descending up the hill. The gradient points toward the steepest increase. To minimize loss you step along the negative gradient, dropping the minus sign sends training the wrong way.
Ignoring the learning rate. Taking the full gradient as a step can overshoot the minimum; too small a step crawls. The scalar η controls how far each step moves.
Letting other variables drift. A partial derivative holds every other variable constant. Treating them as free changes the answer.
Assuming a zero gradient means a minimum. A flat point can be a maximum or a saddle, not just a valley floor.