Session 4: Regularization, Normalization, Optimizers and Over-parametrization
2026-11-16
8:30–11:30 · 180 min
| Time | Block |
|---|---|
| 8:30–8:35 | Recap: where we left off |
| 8:35–8:55 | Regularization: weight decay and dropout |
| 8:55–9:25 | Normalization: BatchNorm, LayerNorm and Adaptive LayerNorm |
| 9:25–9:45 | Optimizers: SGD with momentum and Adam |
| 9:45–9:50 | Learning rate schedules |
| 9:50–10:10 | ⚡ Coding exercise: weight decay and Adam |
| 10:10–10:25 | ☕ Break |
| 10:25–10:40 | Over-parametrization and double descent |
| 10:40–10:55 | Lazy training and the lottery ticket |
| 10:55–11:05 | Scaling laws |
| 11:05–11:25 | ⚡ Coding exercise: a mini scaling law |
| 11:25–11:30 | Wrap-up |
0:05–0:25 · 20 min
Shrink every weight a little at each step, independently of the loss,
\[ w_i \leftarrow (1 - \lambda_w)\, w_i \]
Randomly zero a fraction of a layer’s input to stop a neuron from relying on one specific feature,
\[ x_{i+1} = f_i(x_i \circ a_i), \qquad a_i \sim \text{Bernoulli}(1-p) \]
*32-32-wide ReLU MLP trained on 8 noisy points: plenty of capacity to memorize the noise.
Inverted-dropout convention: zero with probability \(p\) and rescale survivors by \(1/(1-p)\) at train time, so eval() needs no change at all.
0:25–0:55 · 30 min
Normalize each feature across the batch,
\[ \text{BN}(x_i) = \gamma \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \]
Normalize each sample across its own features instead,
\[ \text{LN}(x_i) = \gamma \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \]
Same (batch x feature) grid of raw activations, normalized along a different axis. The highlighted strip is one group: everything in it shares a single mean and std.
class BatchNorm(nn.Module):
def __init__(self, n_features, eps=1e-5, eta=0.999):
super().__init__()
self.eps, self.eta = eps, eta
self.gamma = nn.Parameter(torch.ones(n_features))
self.beta = nn.Parameter(torch.zeros(n_features))
self.register_buffer("running_mean", torch.zeros(n_features))
self.register_buffer("running_var", torch.ones(n_features))
def forward(self, x):
# x: (batch, n_features)
if self.training:
mean, var = x.mean(dim=0), x.var(dim=0, unbiased=False)
with torch.no_grad():
self.running_mean = self.eta * self.running_mean + (1 - self.eta) * mean
self.running_var = self.eta * self.running_var + (1 - self.eta) * var
else:
mean, var = self.running_mean, self.running_var
x_hat = (x - mean) / torch.sqrt(var + self.eps)
return self.gamma * x_hat + self.betaclass LayerNorm(nn.Module):
def __init__(self, n_features, eps=1e-5):
super().__init__()
self.eps = eps
self.gamma = nn.Parameter(torch.ones(n_features))
self.beta = nn.Parameter(torch.zeros(n_features))
def forward(self, x):
# x: (..., n_features); statistics over the last dimension only
mean = x.mean(dim=-1, keepdim=True)
var = x.var(dim=-1, unbiased=False, keepdim=True)
x_hat = (x - mean) / torch.sqrt(var + self.eps)
return self.gamma * x_hat + self.betaNo running buffers at all: every statistic comes from the current sample.
0:55–1:15 · 20 min
Follow a newtonian analogy: the gradient is a force, it updates a velocity, and the weights move by their velocity,
\[ v_\theta \leftarrow m\, v_\theta + g, \qquad w_\theta \leftarrow w_\theta - \eta\, v_\theta \]
class SGDMomentum(torch.optim.Optimizer):
def __init__(self, params, lr=1e-2, momentum=0.9):
super().__init__(params, dict(lr=lr, momentum=momentum))
@torch.no_grad()
def step(self):
for group in self.param_groups:
lr, m = group["lr"], group["momentum"]
for p in group["params"]:
if p.grad is None:
continue
state = self.state[p]
if "velocity" not in state:
state["velocity"] = torch.zeros_like(p)
v = state["velocity"]
v.mul_(m).add_(p.grad)
p.add_(v, alpha=-lr)Momentum is slow on flat regions of the loss. Adam tracks the squared gradient too, and divides the step by its square root: small gradients get amplified while large ones get damped,
\[ m \leftarrow \beta_1 m + (1-\beta_1) g, \qquad v \leftarrow \beta_2 v + (1-\beta_2) g^2 \] \[ w \leftarrow w - \eta\, \frac{\hat m}{\sqrt{\hat v} + \epsilon} \]
with \(\hat m, \hat v\) the bias-corrected moments (\(/(1-\beta_i^t)\), needed because \(m, v\) start at zero).
class Adam(torch.optim.Optimizer):
def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8):
super().__init__(params, dict(lr=lr, betas=betas, eps=eps))
@torch.no_grad()
def step(self):
for group in self.param_groups:
lr, (beta1, beta2), eps = group["lr"], group["betas"], group["eps"]
for p in group["params"]:
if p.grad is None:
continue
state = self.state[p]
if "t" not in state:
state["t"] = 0
state["m"] = torch.zeros_like(p)
state["v"] = torch.zeros_like(p)
state["t"] += 1
t, m, v = state["t"], state["m"], state["v"]
m.mul_(beta1).add_(p.grad, alpha=1 - beta1)
v.mul_(beta2).addcmul_(p.grad, p.grad, value=1 - beta2)
m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)
p.addcdiv_(m_hat, v_hat.sqrt().add_(eps), value=-lr)Adam is the de facto standard, despite a flawed original convergence proof: it provably converges under \(0 \le \beta_1 < \beta_2 < 1\).
1:15–1:20 · 5 min
Linear warmup into a cosine decay: the learning rate ramps up over the first 100 steps, then follows a cosine curve down to a small floor by step 1000.
1:20–1:40 · 20 min
Same toy problem (8 noisy points from \(\sin(2.5x)\)), but now you train it yourself.
import torch, torch.nn as nn
torch.manual_seed(0)
def make_mlp():
return nn.Sequential(
nn.Linear(1, 64), nn.ReLU(),
nn.Linear(64, 64), nn.ReLU(),
nn.Linear(64, 1),
)
def target_fn(x):
return torch.sin(2.5 * x)
g = torch.Generator().manual_seed(16)
x_train = torch.empty(8, 1).uniform_(-1, 1, generator=g)
y_train = target_fn(x_train) + 0.25 * torch.randn(8, 1, generator=g)
x_val = torch.linspace(-1, 1, 200).unsqueeze(1)
y_val = target_fn(x_val)
def train(optimizer_fn, epochs=1500):
torch.manual_seed(0) # same init every call, only the optimizer differs
model = make_mlp()
opt = optimizer_fn(model.parameters())
for step in range(epochs):
opt.zero_grad()
loss = ((model(x_train) - y_train) ** 2).mean()
loss.backward()
opt.step()
with torch.no_grad():
train_loss = ((model(x_train) - y_train) ** 2).mean().item()
val_loss = ((model(x_val) - y_val) ** 2).mean().item()
return train_loss, val_loss
print(train(lambda p: torch.optim.SGD(p, lr=0.3), epochs=3000))
# -> roughly (0.04, 0.16): low training loss, much higher validation lossRun it. Confirm you see the same overfitting gap as in the figures.
Add weight_decay to the SGD call and sweep a few values:
weight_decay here is exactly the \(w \leftarrow (1-\lambda_w) w\) rule from the slides: for plain SGD it’s mathematically identical to adding an \(\ell_2\) penalty to the loss. You should see validation loss drop by roughly 3x around wd=0.01, with training loss barely moving. Good, regularization working as advertised.
Swap the optimizer for Adam, keeping the exact same weight_decay API, and sweep a wider range (Adam needs a smaller learning rate):
At wd=0 it overfits even harder than SGD did (training loss hits exactly 0). Small wd still helps, but keep turning it up the way you just did for SGD and, somewhere around wd=0.3-0.6, both losses collapse together to roughly the same bad value rather than training loss gracefully rising: the model has stopped fitting almost anything. torch.optim.Adam’s weight_decay adds \(\lambda_w w\) to the gradient, so it goes through Adam’s own per-parameter normalization (\(g^2\), \(\sqrt{\hat v}\)) instead of shrinking every weight by the fixed fraction the formula promises. That’s the coupling problem.
Switch torch.optim.Adam for torch.optim.AdamW, same arguments, same sweep:
No collapse this time: validation loss improves smoothly over a much wider range of wd, reaching a better minimum than coupled Adam ever did before it broke. AdamW applies the decay directly to the parameter, after the Adam step, exactly like the \(w \leftarrow
(1-\lambda_w) w\) rule: the update that goes through \(\sqrt{\hat v}\) is only the gradient term.
If you have time: drop AdamW and instead add one line after opt.step() to a plain Adam run,
and check it reproduces AdamW’s numbers. This is the exact formula from the Weight Decay slide, applied by hand.
1:40–1:55 · 15 min
Back at 10:15 for the second half: why bigger neural networks, which should overfit according to everything we proved about VC dimension so far, so often generalize better instead.
1:55–2:10 · 15 min
Train a ridge regression on an increasing number of random Fourier features, on a problem that is genuinely non-linear but has very few training samples:
A ridge regression fit with an increasing number \(D\) of random Fourier features, on a target itself built from 100 such features (so more features always means less bias). Validation loss spikes right at \(D = n_{\text{train}}\), then descends again.
2:10–2:25 · 15 min
Relative distance traveled by the parameters during training. Wide networks barely move from their initialization even as the loss converges.
Linearize the loss around the initialization,
\[ l_x(w) \approx l_x(w_0) + \nabla l_x(w_0)^\top (w - w_0) \]
A randomly-initialized dense network contains a sub-network that, trained in isolation from its original initialization, matches the full network’s accuracy:
A width-64 MLP magnitude-pruned down to each width, surviving units reset to their original initialization (winning ticket) vs. a fresh random reinitialization at the same width. Below $$3 hidden units, the ticket keeps working while random reinitialization degrades sharply.
Over-parametrization gives the initialization more draws at useful neurons (lottery ticket); NTK means the unlucky draws barely get touched and don’t hurt; redundant neurons are what double descent turns into a generalization benefit instead of overfitting.
2:25–2:35 · 10 min
Validation loss against training compute (FLOPs) for a SmallResNet at 5 widths, each curve the minimum loss over 30 seeds at a ladder of training checkpoints. The dashed line fits a power law to the compute-optimal frontier (the best loss reached by any width at a given compute budget).
Smaller models are the efficient choice at a small budget; past a point, continuing to train a small model wastes compute that a bigger model would spend better.
2:35–2:55 · 20 min
Train 3 widths of a small MLP on a toy regression problem, across a ladder of compute budgets (number of gradient steps), and find the compute-optimal frontier.
import torch, torch.nn as nn
def make_mlp(width):
return nn.Sequential(
nn.Linear(1, width), nn.ReLU(),
nn.Linear(width, width), nn.ReLU(),
nn.Linear(width, 1),
)
def target_fn(x):
return torch.sin(3 * x) + 0.3 * x
WIDTHS = [4, 16, 64]
BUDGETS = [20, 50, 100, 200, 400, 800] # gradient steps
x_train = torch.linspace(-3, 3, 64).unsqueeze(1)
y_train = target_fn(x_train) + 0.05 * torch.randn_like(x_train)
x_val = torch.linspace(-3, 3, 200).unsqueeze(1)
y_val = target_fn(x_val)
results = {} # (width, budget) -> validation MSE
for width in WIDTHS:
model = make_mlp(width)
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
for step in range(1, max(BUDGETS) + 1):
# TODO: one training step (zero_grad, forward, MSE loss,
# backward, opt.step)
if step in BUDGETS:
with torch.no_grad():
# TODO: record results[(width, step)] = validation MSE
passOnce results is filled in:
import matplotlib.pyplot as plt
flops_per_step = {4: 1, 16: 4, 64: 16} # relative, not exact
frontier = {}
for (width, budget), mse in results.items():
compute = flops_per_step[width] * budget
# TODO: keep only the best (lowest-mse) width at each compute level
# to build the compute-optimal frontier: frontier[compute] = min(...)
plt.loglog(sorted(frontier), [frontier[c] for c in sorted(frontier)], "o-")
plt.xlabel("compute (relative FLOPs)"); plt.ylabel("validation MSE")Does a wider model ever lose to a narrower one at the same compute budget? At what budget does the 64-wide model start to pull ahead?
2:55–3:00 · 5 min
Chapter 8: sequence modeling, with RNNs, attention, and the transformer.