A neural network is a stack of simple functions that, together, can approximate very complicated ones. Training it means one thing: nudging every weight in the direction that makes the network a little less wrong. Backpropagation is the algorithm that computes that direction efficiently, and it is just the chain rule from calculus, applied with bookkeeping.
From a neuron to a network
A single artificial neuron takes inputs x1,…,xn, weights them, adds a bias, and passes the result through a nonlinear activationσ:
Answer from memory before revealing, retrieval practice is what builds durable recall.
1.
In backpropagation, what does the chain rule let you compute?
Ask the tutor
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
a=σ(i∑wixi+b)
Stack neurons into a layer, stack layers front to back, and the output of one layer is the input to the next. The nonlinearity is not optional decoration: without it, any stack of layers collapses algebraically into a single linear map, and a linear map cannot even separate the four points of XOR. That is the whole reason "deep" matters.
Why nonlinearity is the point
Two stacked linear layers W2(W1x) equal one linear layer (W2W1)x.
Only a nonlinear activation between them lets the network bend space enough to
carve curved decision boundaries. Try it in the Neural Network Playground: on
the XOR set, a network with a hidden layer separates the classes; remove the
network's capacity and it cannot.
Learning as descending a loss
To train, we need a number that says how wrong the network is. For binary classification that number is the binary cross-entropy between the predicted probability y^ and the true label y∈{0,1}:
L=−(ylogy^+(1−y)log(1−y^))
Training walks the weights downhill on this loss by gradient descent: compute the gradient ∇θL (the direction of steepest increase), then step the opposite way, scaled by a learning rate η:
θ←θ−η∇θL
The only hard part is computing ∇θL for millions of weights buried inside nested functions. That is backpropagation's job.
Backpropagation, three ways
Intuition
For a curious beginner
Imagine the network makes a mistake at the output. Backprop asks, layer by layer walking backwards, "how much did each knob contribute to this mistake?" A weight that pushed hard toward the wrong answer gets a large share of the blame; a weight that barely mattered gets little. Every weight then moves a small step to reduce its share of the blame. Repeat on many examples and the blame shrinks.
Engineering
How it is actually used
Do one forward pass, caching each layer's pre-activation z and activation a. Then run a backward pass: start with the error signal at the output, and repeatedly apply the chain rule to push it back one layer. Each layer turns the error signal arriving from above into (a) gradients for its own weights and bias, and (b) the error signal to hand to the layer below. Because the two passes reuse the same cached values, the cost of all the gradients is about the same as one forward pass, not one pass per weight.
Mathematical
The underlying mechanism
Let z(l) be a layer's pre-activations and a(l)=σ(z(l)). Define the error signal δ(l)=∂L/∂z(l). For a sigmoid output with cross-entropy loss the output error collapses to a clean form:
δ(L)=y^−y
Propagate it backwards, where ⊙ is elementwise product and σ′ the activation derivative:
δ(l)=(W(l+1)⊤δ
Then the gradients for that layer's parameters are
∂W(l)∂L=
That is the entire algorithm: one recurrence for δ, two products per layer.
For a curious beginner
Imagine the network makes a mistake at the output. Backprop asks, layer by layer walking backwards, "how much did each knob contribute to this mistake?" A weight that pushed hard toward the wrong answer gets a large share of the blame; a weight that barely mattered gets little. Every weight then moves a small step to reduce its share of the blame. Repeat on many examples and the blame shrinks.
How it is actually used
Do one forward pass, caching each layer's pre-activation z and activation a. Then run a backward pass: start with the error signal at the output, and repeatedly apply the chain rule to push it back one layer. Each layer turns the error signal arriving from above into (a) gradients for its own weights and bias, and (b) the error signal to hand to the layer below. Because the two passes reuse the same cached values, the cost of all the gradients is about the same as one forward pass, not one pass per weight.
The underlying mechanism
Let z(l) be a layer's pre-activations and a(l)=σ(z(l)). Define the error signal δ(l)=∂L/∂z(l). For a sigmoid output with cross-entropy loss the output error collapses to a clean form:
δ(L)=y^−y
Propagate it backwards, where ⊙ is elementwise product and σ′ the activation derivative:
δ(l)=(W(l+1)⊤δ
Then the gradients for that layer's parameters are
∂W(l)∂L=
That is the entire algorithm: one recurrence for δ, two products per layer.
One step, with numbers
Take the smallest possible network, a single sigmoid neuron with two inputs, and watch one weight update end to end. Inputs x=[1,2], weights w=[0.5,−0.3], bias b=0.1, and the true label is y=1.
Forward pass. The pre-activation is z=0.5(1)+(−0.3)(2)+0.1=0.0, so y^=σ(0)=0.5. The loss is L=−log(0.5)≈0.693.
Backward pass. The output error is δ=y^−y=0.5−1=−0.5. The weight gradients are ∂L/∂w=δx=[−0.5,−1.0] and ∂L/∂b=−0.5.
Update with learning rate η=0.1: w←[0.5,−0.3]−0.1[−0.5,−1.0]=[0.55,−0.2] and b←0.1−0.1(−0.5)=0.15.
Did it help? Re-run the forward pass with the new weights: z=0.55(1)−0.2(2)+0.15=0.30, so y^≈0.574 and L≈0.555. One step moved the prediction toward the label and the loss down from 0.693, exactly what gradient descent promises, now in arithmetic you can check by hand.
1Forward passRun inputs through the network, caching each layer's values
2Compute lossMeasure how wrong the output is against the label
3Backward passApply the chain rule backwards to get every weight's gradient
4Update weightsStep each weight opposite its gradient, scaled by the learning rate
5RepeatDo this over many examples until the loss stops falling
How do we know backprop is correct?
The honest test is a numerical gradient check: perturb one weight by a tiny ϵ, measure how the loss changes, and compare to the analytic gradient:
∂θi∂L≈2ϵL(θ+ϵei)−L(θ−ϵei)
If the analytic and numerical values agree to several decimals, the derivation and the code are right. The playground's math core passes exactly this check in its tests, which is why the boundary you watch bend is produced by real backprop, not an animation of one.
What can go wrong
Too large a learning rate overshoots and the loss diverges; too small and
training crawls. ReLU units can "die" (stuck outputting zero) if the rate is
high. Deep stacks can suffer vanishing or exploding gradients when the
repeated W⊤ products shrink or grow the error signal, the motivation for
careful initialization, normalization, and residual connections in real
networks.
Why this scales to everything else
Every modern architecture, convolutional networks, transformers, diffusion models, is trained by this same loop: forward pass, loss, backprop, gradient step. The layers differ; the learning principle does not. Understand backprop on a two-input toy and you understand the engine inside a frontier model. The difference is scale, data, and engineering, not the idea.