Deep Learning

Session 4: Regularization, Normalization, Optimizers and Over-parametrization

David Picard

2026-11-16

Welcome

Session 4: Agenda

8:30–11:30 · 180 min

Time Block
8:30–8:35 Recap: where we left off
8:35–8:55 Regularization: weight decay and dropout
8:55–9:25 Normalization: BatchNorm, LayerNorm and Adaptive LayerNorm
9:25–9:45 Optimizers: SGD with momentum and Adam
9:45–9:50 Learning rate schedules
9:50–10:10 ⚡ Coding exercise: weight decay and Adam
10:10–10:25 ☕ Break
10:25–10:40 Over-parametrization and double descent
10:40–10:55 Lazy training and the lottery ticket
10:55–11:05 Scaling laws
11:05–11:25 ⚡ Coding exercise: a mini scaling law
11:25–11:30 Wrap-up

Where we left off

  • Session 3: perceptrons, backprop, MLPs, CNNs, ResNet
  • A network that is deep enough and wide enough can fit almost anything
  • Today: the tricks that make a deep network actually train well, and why making it bigger often helps instead of just overfitting

Regularization

0:05–0:25 · 20 min

Why regularize a neural network

  • A loss that never vanishes (cross-entropy on correctly classified samples still has gradient) keeps reinforcing the same correlated weights
  • Weights inflated this way can destabilize training, or crowd out correlations with smaller but still useful activations
  • Two standard fixes: shrink the weights directly, or make the network unable to rely on any single feature

Weight decay

Shrink every weight a little at each step, independently of the loss,

\[ w_i \leftarrow (1 - \lambda_w)\, w_i \]

  • Equivalent to an \(\ell_2\) penalty on the weights, the neural network analog of ridge regression
  • Watch the interaction with the optimizer: this is not the same as adding \(\lambda_w \|w\|_2^2\) to the loss once the optimizer normalizes gradients (Adam, later today)

Weight decay against overfitting

Dropout

Randomly zero a fraction of a layer’s input to stop a neuron from relying on one specific feature,

\[ x_{i+1} = f_i(x_i \circ a_i), \qquad a_i \sim \text{Bernoulli}(1-p) \]

  • Train: survivors rescaled by \(1/(1-p)\) so the input stays correct in expectation (inverted dropout)
  • Eval: dropout turned off, no rescaling needed
  • Leaving dropout on at test time turns the output into a random variable: a cheap pseudo-ensemble that flags samples the network is unsure about

Dropout against overfitting

*32-32-wide ReLU MLP trained on 8 noisy points: plenty of capacity to memorize the noise.

Dropout in PyTorch

class Dropout(nn.Module):
    def __init__(self, p=0.5):
        super().__init__()
        self.p = p  # drop probability
    def forward(self, x):
        if self.training:
            mask = torch.bernoulli(torch.full_like(x, 1 - self.p))
            return x * mask / (1 - self.p)
        return x

Inverted-dropout convention: zero with probability \(p\) and rescale survivors by \(1/(1-p)\) at train time, so eval() needs no change at all.

Normalization

0:25–0:55 · 30 min

Why normalize

  • Backprop creates a positive feedback loop: a weight that helps the decision gets reinforced, which can grow activations without bound
  • A layer’s input distribution also keeps shifting as the layers below it keep training (BatchNorm’s covariate shift)
  • Fix: force the input of a layer to have a controlled mean and variance, with a learned affine transform to put back whatever scale and offset the network actually needs

BatchNorm

Normalize each feature across the batch,

\[ \text{BN}(x_i) = \gamma \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \]

  • \(\mu, \sigma^2\): mean and variance of \(x_i\) over the batch, tracked at train time with an exponential moving average for use at test time
  • Couples samples in a batch together \(\Rightarrow\) struggles with a small batch size, and the training/test statistics gap can hurt under domain shift

LayerNorm

Normalize each sample across its own features instead,

\[ \text{LN}(x_i) = \gamma \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \]

  • \(\mu, \sigma^2\) computed over the last dimension of a single sample: no coupling across the batch, no running statistics, train and eval use the exact same computation
  • \(\gamma, \beta\) are optional (could be absorbed by the surrounding layers), but let every other layer ignore the signal’s scale entirely

Comparing BatchNorm and LayerNorm

Same (batch x feature) grid of raw activations, normalized along a different axis. The highlighted strip is one group: everything in it shares a single mean and std.

Adaptive LayerNorm

  • Instead of learning \(\gamma, \beta\) directly, make them the output of a small network fed with some auxiliary signal
  • E.g. for image generation (next chapter), conditioning on “what class to generate” steers the whole signal through every layer
  • Solves a different problem than covariate shift: controllable conditioning, not training stability

BatchNorm in PyTorch

class BatchNorm(nn.Module):
    def __init__(self, n_features, eps=1e-5, eta=0.999):
        super().__init__()
        self.eps, self.eta = eps, eta
        self.gamma = nn.Parameter(torch.ones(n_features))
        self.beta = nn.Parameter(torch.zeros(n_features))
        self.register_buffer("running_mean", torch.zeros(n_features))
        self.register_buffer("running_var", torch.ones(n_features))
    def forward(self, x):
        # x: (batch, n_features)
        if self.training:
            mean, var = x.mean(dim=0), x.var(dim=0, unbiased=False)
            with torch.no_grad():
                self.running_mean = self.eta * self.running_mean + (1 - self.eta) * mean
                self.running_var = self.eta * self.running_var + (1 - self.eta) * var
        else:
            mean, var = self.running_mean, self.running_var
        x_hat = (x - mean) / torch.sqrt(var + self.eps)
        return self.gamma * x_hat + self.beta

LayerNorm in PyTorch

class LayerNorm(nn.Module):
    def __init__(self, n_features, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.gamma = nn.Parameter(torch.ones(n_features))
        self.beta = nn.Parameter(torch.zeros(n_features))
    def forward(self, x):
        # x: (..., n_features); statistics over the last dimension only
        mean = x.mean(dim=-1, keepdim=True)
        var = x.var(dim=-1, unbiased=False, keepdim=True)
        x_hat = (x - mean) / torch.sqrt(var + self.eps)
        return self.gamma * x_hat + self.beta

No running buffers at all: every statistic comes from the current sample.

Optimizers

0:55–1:15 · 20 min

Beyond vanilla SGD

  • Deep networks have more parameters than fit in a mini-batch, so most gradients are noisy estimates of rare features
  • Gradient accumulation (summing several mini-batches before stepping) works, but is slow: updates become less frequent
  • Instead, buffer the gradient history itself and update from the buffer

SGD with momentum

Follow a newtonian analogy: the gradient is a force, it updates a velocity, and the weights move by their velocity,

\[ v_\theta \leftarrow m\, v_\theta + g, \qquad w_\theta \leftarrow w_\theta - \eta\, v_\theta \]

  • At a constant gradient, the velocity’s magnitude saturates around \(1/(1-m)\): the learning rate must account for that amplification
  • Cheap: one extra buffer per parameter

SGD with momentum in PyTorch

class SGDMomentum(torch.optim.Optimizer):
    def __init__(self, params, lr=1e-2, momentum=0.9):
        super().__init__(params, dict(lr=lr, momentum=momentum))
    @torch.no_grad()
    def step(self):
        for group in self.param_groups:
            lr, m = group["lr"], group["momentum"]
            for p in group["params"]:
                if p.grad is None:
                    continue
                state = self.state[p]
                if "velocity" not in state:
                    state["velocity"] = torch.zeros_like(p)
                v = state["velocity"]
                v.mul_(m).add_(p.grad)
                p.add_(v, alpha=-lr)

Adam

Momentum is slow on flat regions of the loss. Adam tracks the squared gradient too, and divides the step by its square root: small gradients get amplified while large ones get damped,

\[ m \leftarrow \beta_1 m + (1-\beta_1) g, \qquad v \leftarrow \beta_2 v + (1-\beta_2) g^2 \] \[ w \leftarrow w - \eta\, \frac{\hat m}{\sqrt{\hat v} + \epsilon} \]

with \(\hat m, \hat v\) the bias-corrected moments (\(/(1-\beta_i^t)\), needed because \(m, v\) start at zero).

Adam in PyTorch

class Adam(torch.optim.Optimizer):
    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8):
        super().__init__(params, dict(lr=lr, betas=betas, eps=eps))
    @torch.no_grad()
    def step(self):
        for group in self.param_groups:
            lr, (beta1, beta2), eps = group["lr"], group["betas"], group["eps"]
            for p in group["params"]:
                if p.grad is None:
                    continue
                state = self.state[p]
                if "t" not in state:
                    state["t"] = 0
                    state["m"] = torch.zeros_like(p)
                    state["v"] = torch.zeros_like(p)
                state["t"] += 1
                t, m, v = state["t"], state["m"], state["v"]
                m.mul_(beta1).add_(p.grad, alpha=1 - beta1)
                v.mul_(beta2).addcmul_(p.grad, p.grad, value=1 - beta2)
                m_hat = m / (1 - beta1 ** t)
                v_hat = v / (1 - beta2 ** t)
                p.addcdiv_(m_hat, v_hat.sqrt().add_(eps), value=-lr)

Adam is the de facto standard, despite a flawed original convergence proof: it provably converges under \(0 \le \beta_1 < \beta_2 < 1\).

Watching three optimizers race

Learning Rate Schedules

1:15–1:20 · 5 min

Warmup and cosine decay

  • Constant learning rate: fast early, but unstable by the end of training
  • Warmup: ramp the learning rate up linearly for the first few hundred steps, empirically stable even for a high peak rate
  • Then decay it, following a cosine curve down to a small floor value
  • Downside: needs the total step count fixed in advance, which is exactly what a scaling law (later today) is for

Linear warmup into a cosine decay: the learning rate ramps up over the first 100 steps, then follows a cosine curve down to a small floor by step 1000.

Racing under the schedule

Coding Exercise: Weight Decay and Adam

1:20–1:40 · 20 min

Step 1: set up the overfitting problem

Same toy problem (8 noisy points from \(\sin(2.5x)\)), but now you train it yourself.

import torch, torch.nn as nn
torch.manual_seed(0)

def make_mlp():
    return nn.Sequential(
        nn.Linear(1, 64), nn.ReLU(),
        nn.Linear(64, 64), nn.ReLU(),
        nn.Linear(64, 1),
    )

def target_fn(x):
    return torch.sin(2.5 * x)

g = torch.Generator().manual_seed(16)
x_train = torch.empty(8, 1).uniform_(-1, 1, generator=g)
y_train = target_fn(x_train) + 0.25 * torch.randn(8, 1, generator=g)
x_val = torch.linspace(-1, 1, 200).unsqueeze(1)
y_val = target_fn(x_val)

def train(optimizer_fn, epochs=1500):
    torch.manual_seed(0)  # same init every call, only the optimizer differs
    model = make_mlp()
    opt = optimizer_fn(model.parameters())
    for step in range(epochs):
        opt.zero_grad()
        loss = ((model(x_train) - y_train) ** 2).mean()
        loss.backward()
        opt.step()
    with torch.no_grad():
        train_loss = ((model(x_train) - y_train) ** 2).mean().item()
        val_loss = ((model(x_val) - y_val) ** 2).mean().item()
    return train_loss, val_loss

print(train(lambda p: torch.optim.SGD(p, lr=0.3), epochs=3000))
# -> roughly (0.04, 0.16): low training loss, much higher validation loss

Run it. Confirm you see the same overfitting gap as in the figures.

Step 2: weight decay fixes it, for SGD

Add weight_decay to the SGD call and sweep a few values:

for wd in [0.0, 0.01, 0.02, 0.05]:
    tr, va = train(lambda p: torch.optim.SGD(p, lr=0.3, weight_decay=wd), epochs=3000)
    print(f"wd={wd:<5} train={tr:.4f} val={va:.4f}")

weight_decay here is exactly the \(w \leftarrow (1-\lambda_w) w\) rule from the slides: for plain SGD it’s mathematically identical to adding an \(\ell_2\) penalty to the loss. You should see validation loss drop by roughly 3x around wd=0.01, with training loss barely moving. Good, regularization working as advertised.

Step 3: the same fix, now under Adam

Swap the optimizer for Adam, keeping the exact same weight_decay API, and sweep a wider range (Adam needs a smaller learning rate):

for wd in [0.0, 0.01, 0.1, 0.3, 0.6, 1.0]:
    tr, va = train(lambda p: torch.optim.Adam(p, lr=0.01, weight_decay=wd), epochs=1500)
    print(f"wd={wd:<5} train={tr:.4f} val={va:.4f}")

At wd=0 it overfits even harder than SGD did (training loss hits exactly 0). Small wd still helps, but keep turning it up the way you just did for SGD and, somewhere around wd=0.3-0.6, both losses collapse together to roughly the same bad value rather than training loss gracefully rising: the model has stopped fitting almost anything. torch.optim.Adam’s weight_decay adds \(\lambda_w w\) to the gradient, so it goes through Adam’s own per-parameter normalization (\(g^2\), \(\sqrt{\hat v}\)) instead of shrinking every weight by the fixed fraction the formula promises. That’s the coupling problem.

Step 4: decouple it

Switch torch.optim.Adam for torch.optim.AdamW, same arguments, same sweep:

for wd in [0.0, 0.3, 0.6, 1.0, 2.0, 5.0]:
    tr, va = train(lambda p: torch.optim.AdamW(p, lr=0.01, weight_decay=wd), epochs=1500)
    print(f"wd={wd:<5} train={tr:.4f} val={va:.4f}")

No collapse this time: validation loss improves smoothly over a much wider range of wd, reaching a better minimum than coupled Adam ever did before it broke. AdamW applies the decay directly to the parameter, after the Adam step, exactly like the \(w \leftarrow (1-\lambda_w) w\) rule: the update that goes through \(\sqrt{\hat v}\) is only the gradient term.

If you have time: drop AdamW and instead add one line after opt.step() to a plain Adam run,

with torch.no_grad():
    for p in model.parameters():
        p.mul_(1 - lr * wd)

and check it reproduces AdamW’s numbers. This is the exact formula from the Weight Decay slide, applied by hand.

Break

1:40–1:55 · 15 min

☕ 15-minute break

Back at 10:15 for the second half: why bigger neural networks, which should overfit according to everything we proved about VC dimension so far, so often generalize better instead.

Over-parametrization

1:55–2:10 · 15 min

A paradox on paper

  • Our generalization bounds only hold for low-complexity hypothesis classes, yet the empirical trend in deep learning is that bigger is better
  • A bound is a guarantee of success, not a proof of failure: violating its conditions does not mean the predictor must fail, just that we no longer have a guarantee either way
  • So what actually happens once a network is large enough to fit the training data exactly, and then some?

Double descent

Train a ridge regression on an increasing number of random Fourier features, on a problem that is genuinely non-linear but has very few training samples:

  • Loss first decreases as expected (SRM), since more features mean less bias
  • It then spikes exactly when the number of features equals the number of training samples, the interpolation threshold, where the fit is forced to be exactly right and any noise gets fully absorbed
  • Past that threshold, loss decreases again: more features means more equally-good interpolating solutions, and the minimum-norm one among them is the one SGD finds

Interpolation threshold

A ridge regression fit with an increasing number \(D\) of random Fourier features, on a target itself built from 100 such features (so more features always means less bias). Validation loss spikes right at \(D = n_{\text{train}}\), then descends again.

Lazy Training and the Lottery Ticket

2:10–2:25 · 15 min

Lazy training

  • Take the smallest network solving a toy problem (3 hidden neurons) and give it many more (say 64). How does training behave?
  • Wider networks travel a shorter relative distance from their initialization, while converging more reliably

Relative distance traveled by the parameters during training. Wide networks barely move from their initialization even as the loss converges.

The Neural Tangent Kernel

Linearize the loss around the initialization,

\[ l_x(w) \approx l_x(w_0) + \nabla l_x(w_0)^\top (w - w_0) \]

  • This is a linear model in the mapping \(\phi(x) = \nabla l_x(w_0)\): the Neural Tangent Kernel, increasingly accurate as width grows
  • Some weights are highly influential, others are near-blind-spots of the kernel and barely move: only the useful ones need to change
  • A wide network doesn’t need luck to find a good solution: it just needs enough neurons that a few of them already point the right way

The Lottery Ticket Hypothesis

A randomly-initialized dense network contains a sub-network that, trained in isolation from its original initialization, matches the full network’s accuracy:

  • Train a 64-neuron network and keep it as reference
  • Prune to the neurons with the highest converged magnitude, but reset them to their original initial values, and retrain
  • Performance stays essentially unchanged down to the $$3 neurons we already knew were the bare minimum for this problem

Pruning down to the winning ticket

A width-64 MLP magnitude-pruned down to each width, surviving units reset to their original initialization (winning ticket) vs. a fresh random reinitialization at the same width. Below $$3 hidden units, the ticket keeps working while random reinitialization degrades sharply.

Winning ticket vs. random luck

Over-parametrization gives the initialization more draws at useful neurons (lottery ticket); NTK means the unlucky draws barely get touched and don’t hurt; redundant neurons are what double descent turns into a generalization benefit instead of overfitting.

Scaling Laws

2:25–2:35 · 10 min

How big should a network be

  • Bigger costs more compute, and can cross the interpolation threshold depending on how much data is available: is the extra cost worth it?
  • Fit a power law relating validation loss to compute, model size and dataset size, and use it to predict the best trade-off before paying for the full-scale run

A scaling law on Bluebell

Validation loss against training compute (FLOPs) for a SmallResNet at 5 widths, each curve the minimum loss over 30 seeds at a ladder of training checkpoints. The dashed line fits a power law to the compute-optimal frontier (the best loss reached by any width at a given compute budget).

Smaller models are the efficient choice at a small budget; past a point, continuing to train a small model wastes compute that a bigger model would spend better.

Coding Exercise: a Mini Scaling Law

2:35–2:55 · 20 min

⚡ Fit your own scaling law

Train 3 widths of a small MLP on a toy regression problem, across a ladder of compute budgets (number of gradient steps), and find the compute-optimal frontier.

import torch, torch.nn as nn

def make_mlp(width):
    return nn.Sequential(
        nn.Linear(1, width), nn.ReLU(),
        nn.Linear(width, width), nn.ReLU(),
        nn.Linear(width, 1),
    )

def target_fn(x):
    return torch.sin(3 * x) + 0.3 * x

WIDTHS = [4, 16, 64]
BUDGETS = [20, 50, 100, 200, 400, 800]  # gradient steps

x_train = torch.linspace(-3, 3, 64).unsqueeze(1)
y_train = target_fn(x_train) + 0.05 * torch.randn_like(x_train)
x_val = torch.linspace(-3, 3, 200).unsqueeze(1)
y_val = target_fn(x_val)

results = {}  # (width, budget) -> validation MSE
for width in WIDTHS:
    model = make_mlp(width)
    opt = torch.optim.Adam(model.parameters(), lr=1e-2)
    for step in range(1, max(BUDGETS) + 1):
        # TODO: one training step (zero_grad, forward, MSE loss,
        # backward, opt.step)
        if step in BUDGETS:
            with torch.no_grad():
                # TODO: record results[(width, step)] = validation MSE
                pass

⚡ Fit your own scaling law (continued)

Once results is filled in:

import matplotlib.pyplot as plt

flops_per_step = {4: 1, 16: 4, 64: 16}  # relative, not exact
frontier = {}
for (width, budget), mse in results.items():
    compute = flops_per_step[width] * budget
    # TODO: keep only the best (lowest-mse) width at each compute level
    # to build the compute-optimal frontier: frontier[compute] = min(...)

plt.loglog(sorted(frontier), [frontier[c] for c in sorted(frontier)], "o-")
plt.xlabel("compute (relative FLOPs)"); plt.ylabel("validation MSE")

Does a wider model ever lose to a narrower one at the same compute budget? At what budget does the 64-wide model start to pull ahead?

Wrap-up

2:55–3:00 · 5 min

Recap

  • Regularization: weight decay and dropout keep a network from leaning on spurious correlations
  • Normalization: BatchNorm and LayerNorm stabilize training along different axes; adaptive LayerNorm repurposes the same mechanism for conditioning
  • Optimizers: momentum buffers the gradient, while Adam also normalizes by its own scale; both need a learning rate schedule to land well
  • Over-parametrization: double descent, lazy training and the lottery ticket all point the same way: bigger networks are not just memorizing, they are easier to optimize and more likely to contain a good solution
  • Scaling laws: predict the best size for a budget before spending it

Next session

Chapter 8: sequence modeling, with RNNs, attention, and the transformer.