Everything a neural net does is a derivative away.

How to use this workbook

The learning loop, and the one rule that makes it work.

The chapters are ordered so that each one uses only what came before it. The dependency chain is:

derivatives → partials/gradients → chain rule → linear layers → losses → gradient descent → backprop → activations → training loop

Backpropagation (Ch. 07) is the payoff: it is nothing but chapters 1–6 glued together. If backprop ever feels like magic, the fix is always to revisit the chain rule.

The rule

For each concept you'll see

• What / Why / Where — definition, purpose, and where it appears in a real network
• A worked example on graph paper — every number shown
• Code twice — from scratch (plain Python/tensors) first, then the PyTorch API that hides it
• Exercises — with solutions collapsed until you want them

Derivatives

Goal: given a function, compute how fast its output changes when the input nudges.

What

The derivative \(f'(x)\) is the slope of \(f\) at the point \(x\): how much the output changes per tiny change of input.

Why

Training asks one question millions of times: “if I nudge this weight, does the loss go up or down, and how steeply?” That question is a derivative.

Where

Inside loss.backward() — it computes the derivative of the loss with respect to every parameter.

Definition, in numbers first

Slope = rise over run, measured over a shrinking run \(h\):

$$ f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h} $$

The five rules you actually need

Deep learning uses a tiny subset of calculus. These cover ~95% of everything:

(The sixth, the chain rule, is so important it gets its own chapter.)

def numeric_derivative(f, x, h=1e-5):
    """Estimate f'(x) with a tiny rise-over-run. Your universal answer-checker."""
    return (f(x + h) - f(x)) / h

f = lambda x: x**2
print(numeric_derivative(f, 3.0))   # ~6.00001

import torch

x = torch.tensor(3.0, requires_grad=True)  # track operations on x
y = x**2
y.backward()      # compute dy/dx and store it in x.grad
print(x.grad)     # tensor(6.)

Partial derivatives & gradients

Goal: handle functions of many inputs — because a loss depends on many weights.

What

A partial derivative \(\frac{\partial f}{\partial w_1}\) is an ordinary derivative taken while pretending every other variable is a constant. The gradient \(\nabla f\) is the vector collecting all the partials.

Why

A loss \(L(w_1, w_2, \dots, w_{61706})\) depends on every weight in LeNet. The gradient answers “which direction in weight space increases \(L\) fastest?” — so we step the opposite way.

Where

param.grad after loss.backward() — one partial derivative per entry of every weight tensor.

def numeric_partial(f, point, i, h=1e-5):
    """Partial derivative of f w.r.t. variable i: nudge only that one."""
    nudged = list(point)
    nudged[i] += h
    return (f(*nudged) - f(*point)) / h

f = lambda w1, w2: w1**2 + 3*w1*w2
print(numeric_partial(f, (2.0, 1.0), 0))  # ~7.0  (d f / d w1)
print(numeric_partial(f, (2.0, 1.0), 1))  # ~6.0  (d f / d w2)

w = torch.tensor([2.0, 1.0], requires_grad=True)
f = w[0]**2 + 3*w[0]*w[1]
f.backward()
print(w.grad)   # tensor([7., 6.])  — the gradient vector, all partials at once

The chain rule

Goal: differentiate compositions — because a network is a composition of layers.

What

If \(y = f(g(x))\), then \(\dfrac{dy}{dx} = f'(g(x)) \cdot g'(x)\): the derivative of the outer function times the derivative of the inner one.

Why

A network is \( \text{loss}(\text{layer}_3(\text{layer}_2(\text{layer}_1(x)))) \). To learn how a weight buried in layer 1 affects the loss, you multiply the local slopes along the path connecting them. That multiplication is backpropagation.

Where

Every step of loss.backward(). Also explains vanishing gradients: multiply many slopes \(< 1\) and the product dies.

Longer chains: just keep multiplying

Three functions deep: \(y = f(g(h(x)))\)

$$ \frac{dy}{dx} = \frac{dy}{dg} \cdot \frac{dg}{dh} \cdot \frac{dh}{dx} $$

Each factor is a local derivative that one layer can compute knowing nothing about the rest. That locality is why backprop is mechanical enough for a library to automate.

import torch

# By hand: y = (2x+1)^2, dy/dx = 2*(2x+1) * 2
x_val = 1.0
manual = 2 * (2*x_val + 1) * 2          # 12.0

# Autograd does the same multiplication internally
x = torch.tensor(x_val, requires_grad=True)
y = (2*x + 1)**2
y.backward()
print(manual, x.grad)                    # 12.0 tensor(12.)

Linear layers

Goal: read \(y = Wx + b\) as arithmetic you can do by hand, and count its parameters.

What

A linear layer computes weighted sums: each output is a dot product of the input with that output's own weight row, plus a bias.

Why

It is the only learnable computation in most networks. Convolutions, attention, fully-connected layers — all are structured variants of “multiply by weights and sum.”

Where

nn.Linear, nn.Conv2d (a linear layer with weight sharing), the X @ W + b in your softmax recipe.

Dot product first

$$ a \cdot b = \sum_i a_i b_i \qquad e.g.\;\; [1, 2, 3] \cdot [4, 0, 2] = 4 + 0 + 6 = 10 $$

One number that measures “how strongly does the input match this weight pattern?” — the edge-detector calculation from convolution is exactly this.

def linear(W, b, x):
    """y_i = dot(W[i], x) + b[i], written as explicit loops."""
    n_out, n_in = len(W), len(x)
    y = [0.0] * n_out
    for i in range(n_out):                # one output at a time
        for j in range(n_in):             # accumulate the dot product
            y[i] += W[i][j] * x[j]
        y[i] += b[i]
    return y

W = [[1, 0, 2], [-1, 1, 0]]
b = [1, 0]
x = [2, 3, 1]
print(linear(W, b, x))   # [5.0, 1.0]

import torch
from torch import nn

layer = nn.Linear(3, 2)                       # allocates W (2x3) and b (2), random init
with torch.no_grad():                          # setting weights manually, skip autograd
    layer.weight[:] = torch.tensor([[1., 0., 2.], [-1., 1., 0.]])
    layer.bias[:]   = torch.tensor([1., 0.])

x = torch.tensor([2., 3., 1.])
print(layer(x))          # tensor([5., 1.])

Softmax & loss functions

Goal: turn raw scores into probabilities, then into a single number measuring “how wrong.”

What

Softmax converts a vector of raw scores (logits) into probabilities that sum to 1. Cross-entropy then scores them: the negative log of the probability assigned to the true class. For regression, MSE \((\hat{y} - t)^2\) plays the same role.

Why

Gradient descent needs one scalar to minimize. The loss compresses “how good are all these predictions” into that scalar — and its shape determines what gradients flow back.

Where

Your recipe's softmax() + cross_entropy(); PyTorch's nn.CrossEntropyLoss fuses both (it takes raw logits).

import torch

def softmax(O):
    O_max = O.max(dim=1, keepdim=True).values   # subtract max: avoids exp() overflow,
    exp_O = torch.exp(O - O_max)                # doesn't change the result (why? ex 5.2)
    return exp_O / exp_O.sum(dim=1, keepdim=True)

def cross_entropy(y_hat, y):
    picked = y_hat[range(len(y_hat)), y]        # prob assigned to each true class
    return (-torch.log(picked)).mean()

O = torch.tensor([[2.0, 0.0, 1.0]])
y = torch.tensor([0])
print(softmax(O))                    # [[0.665, 0.090, 0.245]]
print(cross_entropy(softmax(O), y))  # 0.408

loss_fn = torch.nn.CrossEntropyLoss()   # = softmax + cross-entropy fused
print(loss_fn(O, y))                     # 0.408 — note: takes RAW logits, not probabilities

Gradient descent

Goal: use the gradient to actually improve the weights — and see convergence and divergence happen by hand.

What

Repeated updates \(w \leftarrow w - \eta \, \dfrac{\partial L}{\partial w}\): step every weight a little bit against its gradient. \(\eta\) is the learning rate.

Why

The gradient points uphill on the loss surface; stepping downhill reduces the loss. Small enough steps, repeated, find a valley.

Where

Your recipe's sgd(): param -= lr * param.grad. Same update inside torch.optim.SGD; Adam and friends are variations on the step size.

w, lr = 0.0, 0.1
for step in range(6):
    grad = 8*w - 24                 # dL/dw, derived by hand above
    w = w - lr * grad               # THE update. All of deep learning training is this line.
    print(f"step {step}: w={w:.4f}  loss={(2*w - 6)**2:.5f}")
# w -> 3.0, loss -> 0. Try lr = 0.3 to reproduce the divergence table.

Backpropagation

Goal: the payoff chapter — compute every gradient in a real (tiny) network entirely by hand, then watch PyTorch agree.

What

An algorithm for applying the chain rule to a whole network efficiently: run forward saving intermediate values, then sweep backward multiplying local derivatives, reusing shared sub-products.

Why

Naively computing each weight's derivative separately re-walks the whole network per weight. Backprop computes all of them in one backward pass — the reason training 61k-parameter (or 175B-parameter) models is feasible.

Where

loss.backward(). Everything in this chapter is what that one call does.

The network

The smallest network that has everything — a hidden layer, an activation, a loss:

$$ x \;\xrightarrow{\;\times w_1\;}\; z \;\xrightarrow{\;\text{ReLU}\;}\; h \;\xrightarrow{\;\times w_2\;}\; \hat{y} \;\xrightarrow{\;(\hat{y}-t)^2\;}\; L $$

Numbers: \(x = 2\), target \(t = 3\), weights \(w_1 = 0.5\), \(w_2 = 1.5\). (Biases omitted to keep the paper math short — they backprop the same way, see Ex. 7.2.)

x, t = 2.0, 3.0
w1, w2 = 0.5, 1.5

# ---- forward (save intermediates) ----
z     = w1 * x
h     = max(0.0, z)              # ReLU
y_hat = w2 * h
L     = (y_hat - t)**2

# ---- backward (chain rule, right to left) ----
dL_dyhat = 2 * (y_hat - t)       # step 1:  -3
dL_dw2   = dL_dyhat * h          # step 2:  -3
dL_dh    = dL_dyhat * w2         # step 3:  -4.5
dL_dz    = dL_dh * (1.0 if z > 0 else 0.0)   # step 4: ReLU gate
dL_dw1   = dL_dz * x             # step 5:  -9

print(L, dL_dw1, dL_dw2)         # 2.25 -9.0 -3.0

import torch

w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(1.5, requires_grad=True)
x, t = torch.tensor(2.0), torch.tensor(3.0)

y_hat = w2 * torch.relu(w1 * x)   # forward builds the computation graph
L = (y_hat - t)**2
L.backward()                       # backward walks it right-to-left, exactly as above

print(w1.grad, w2.grad)            # tensor(-9.) tensor(-3.)  — matches the paper ✓

Activations & nonlinearity

Goal: prove that layers without activations collapse, and see what the activation buys.

What

An elementwise nonlinear function \(\phi\) applied after each linear step: \(h = \phi(Wx + b)\). ReLU \(= \max(0, x)\), sigmoid \(= 1/(1+e^{-x})\), tanh, GELU…

Why

Without \(\phi\), any stack of linear layers is algebraically equal to one linear layer — depth adds nothing. The activation is what makes depth meaningful.

Where

Between every pair of layers. Last layer usually has none (logits go straight into the loss).

import torch
from torch import nn

x = torch.linspace(-3, 3, 7).reshape(-1, 1)     # 7 test inputs

no_act = nn.Sequential(nn.Linear(1, 4), nn.Linear(4, 4), nn.Linear(4, 1))
with_act = nn.Sequential(nn.Linear(1, 4), nn.ReLU(),
                         nn.Linear(4, 4), nn.ReLU(), nn.Linear(4, 1))

# no_act's output is ALWAYS a perfect straight line in x — check the
# differences between consecutive outputs (constant slope => all equal):
y = no_act(x).detach().squeeze()                 # detach: just numbers, no autograd
print(torch.diff(y))                             # all identical values
print(torch.diff(with_act(x).detach().squeeze()))  # varies — the line bends

The full training loop

Goal: assemble chapters 1–8 into the complete algorithm — and map every from-scratch piece to its PyTorch name.

Every training run — LeNet, GPT, anything — is this loop:

1. Forward (Ch. 04, 08): push a batch through linear layers + activations to get predictions
2. Loss (Ch. 05): compress predictions vs. targets into one scalar
3. Backward (Ch. 03, 07): chain rule computes every parameter's gradient
4. Update (Ch. 06): step each parameter against its gradient
5. Repeat

import torch

# Tiny 2-layer MLP on fake data, pure tensors — no nn.Module anywhere.
torch.manual_seed(0)
X = torch.randn(100, 4)                       # 100 samples, 4 features
true_W = torch.tensor([[1.], [-2.], [3.], [0.5]])
y = X @ true_W + 0.1*torch.randn(100, 1)      # targets from a secret linear rule

W1 = (torch.randn(4, 8) * 0.1).requires_grad_(True)   # layer 1 weights
b1 = torch.zeros(8, requires_grad=True)
W2 = (torch.randn(8, 1) * 0.1).requires_grad_(True)   # layer 2 weights
b2 = torch.zeros(1, requires_grad=True)
lr = 0.05

for step in range(200):
    # 1. forward                       (Ch. 04 + 08)
    h = torch.relu(X @ W1 + b1)
    y_hat = h @ W2 + b2
    # 2. loss                          (Ch. 05 — MSE here)
    loss = ((y_hat - y)**2).mean()
    # 3. backward                      (Ch. 03 + 07)
    loss.backward()
    # 4. update                        (Ch. 06)
    with torch.no_grad():              # raw arithmetic, don't record for autograd
        for p in (W1, b1, W2, b2):
            p -= lr * p.grad           # the gradient descent step
            p.grad.zero_()             # clear: .backward() ACCUMULATES by default
    if step % 50 == 0:
        print(f"step {step}: loss {loss.item():.4f}")

The translation table

Every wrapper you'll meet (in d2l, PyTorch, Lightning) is one of these rows:

from torch import nn

net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 1))
loss_fn = nn.MSELoss()
opt = torch.optim.SGD(net.parameters(), lr=0.05)

for step in range(200):
    loss = loss_fn(net(X), y)     # forward + loss
    opt.zero_grad()               # p.grad.zero_() for every param
    loss.backward()               # chain rule
    opt.step()                    # p -= lr * p.grad for every param

Resources

Curated for the paper-first style of this workbook.

• 3Blue1Brown — Neural Networks seriesThe best visual intuition for gradient descent and backprop; watch after Ch. 06–07.
• Karpathy — “The spelled-out intro to neural networks and backpropagation”Builds autograd from scratch in Python (micrograd). The perfect follow-up to Ch. 07.
• TensorFlow PlaygroundSet activation to “Linear” and watch a deep net fail; switch to ReLU and watch it bend. Ch. 08, live.
• CNN ExplainerInteractive convolution/pooling on a real image — bridges this workbook back to LeNet.
• Dive into Deep LearningYour main text. Chapters 2.4–2.5 (calculus, autograd) and 5 (MLPs) map directly onto this workbook.
• The Matrix Calculus You Need For Deep Learning (Parr & Howard)When you're ready to redo Ch. 07 with full matrices instead of scalars.