Deep Learning

Why depth works, and what it takes to train it

About this course

Deep networks as an engineering discipline — gradients through depth, the techniques that make training stable, and the dominant architectures.

What you will be able to do

  • Explain backpropagation as an algorithm over a computational graph
  • Diagnose and fix gradient pathologies with initialisation and normalisation
  • Describe why residual connections changed what depth is achievable
  • Apply convolutional and recurrent architectures to their natural data
  • Train a model in practice — schedules, regularisation and transfer learning

What this course covers

  • Overfit-one-batch diagnostic and its limits
  • Error bar as prerequisite for any comparison
  • Learning-rate range test vs. full sweep
  • Augmentation as a testable invariance assertion
  • Two ways a reported number lies: gradient deletion and early-stopping bias
  • Worked arithmetic: reproducing the lecture's key numbers by hand
  • Clause thirteen: measurement of noise precedes every comparison
  • Foundational vocabulary: pieces, regions, patterns, parameter count
  • Composition multiplies linear pieces; width only adds them
  • Expressivity vs. trainability: what the construction does not prove
  • Regions at random initialisation: the gap between bound and reality
  • The shootout: deep vs. wide at equal parameter count
  • Deeper is not always better: the eight-layer penalty
  • What a failed run looks like from the inside: collapsed capacity
  • Clause one: depth raises the ceiling and lowers the floor
  • Key vocabulary: fan-in, fan-out, variance, gain, pre-activation
  • Forward recursion: where the factor of two comes from
  • Backward recursion: fan-out demand and the rectifier's half in both directions
  • Orthogonal initialisation: exactness and its limits
  • Drift: what training does to the initialisation setting
  • Practical initialisation prescription
  • Clause four of the spine and lecture summary
  • Experimental protocol: two-run memorization measurement
  • Standard error and refusing sub-threshold findings
  • What the memorisation result kills and leaves standing
  • Double descent in ridgeless least squares on random features
  • Early stopping as an honest non-result
  • Three refused sentences and the one permitted sentence
  • Clause eight: the solution, not the set
  • Convolutional layer as constrained dense layer
  • Output size formula and odd-kernel convention
  • Translation equivariance: exact interior, broken border
  • Pooling and equivariance on a coarser grid
  • Two-currency cost: parameters vs. arithmetic operations
  • Postmortem: when a spatial prior fails at small scale
  • Clause nine: a constraint is a claim about the data
  • Transfer learning vocabulary and the baseline obligation
  • The layer ladder and the missing baseline
  • Linear probe as instrument: scope and deliberate weakness
  • Specialization measured by linear reconstruction
  • Freeze, fine-tune, or something in between — and what each costs the source task
  • Data efficiency curves and where transfer wins (if at all)
  • Clause fourteen: reuse is a question about regime, not technique
  • Receptive field recursion and the role of stride
  • PCA of image patches and first-layer edges
  • Parameter and function-class cost of stacking 3×3 layers
  • The 1×1 convolution and the bottleneck block
  • Halve resolution, double channels: constant compute per stage
  • Reach vs. depth: why networks stride on purpose
  • Clause ten and the measurement-space warning
  • Computation graph and adjoint definitions
  • Reverse-mode algorithm and the sum rule
  • Forward mode vs. reverse mode: pass-count asymmetry
  • Activation memory and gradient checkpointing
  • Exact operation count for one training step
  • Gradient checking with central differences
  • Backpropagation as a graph algorithm: capstone summary
  • Batch Normalization Transform and Epsilon Constant
  • Scale Invariance and Effective Learning Rate
  • Batch Statistics as Estimates and Batch-Size Hyperparameter
  • Batch Norm vs. Layer Norm: Moving the Normalization Axis
  • Why Batch Norm Helps: Covariate Shift Story and Its Test
  • Clause Five: Normalization's Place in the Lecture Arc
  • Preview of Residual Connections as the Third Repair
  • Fixed-size state capacity and its limits
  • Soft content-addressed lookup (attention defined)
  • Worked attention example with four keys
  • Score scaling by sqrt(dimension)
  • Constant path length vs. redistributable share
  • Quadratic cost of attention
  • Scope boundary: what this course did not establish
  • Per-layer gain and its two factors
  • Telescoping identity: product equals attenuation exactly
  • Why the quarter-per-layer bound is incomplete
  • The knife-edge: critical weight scale where gain equals one
  • Why initialization alone does not solve the problem
  • Gradient clipping as a direction-preserving rescale
  • Clause three and the road ahead
  • Shared weight gradient as sum over time steps
  • Jacobian of a linear recurrence as a matrix power
  • What the spectral radius actually guarantees
  • Effect of tanh non-linearity on per-step decay
  • Memory horizon measured on a real sequence task
  • Structural contrast with feed-forward depth
  • Clause eleven and the lecture's central result
  • The Degradation Problem
  • The Identity (Skip) Path and Its Derivative
  • Measuring Gains: Naive Skip Explodes
  • The Unrolled Ensemble: Exact Path Expansion
  • What the Skip Does Not Fix: Geometric Variance Growth
  • Branch Scale as the Completing Half of the Invention
  • Clause Six and the Lecture's Closing Summary
  • Bias correction duration and peak error
  • L2 penalty vs. decoupled weight decay under Adam
  • Early step size inflation and warm-up rationale
  • Minibatch gradient noise and the finite-population scaling law
  • Sharpness as a parameterisation artefact
  • Scale management as the central function of Adam
  • Invariance as the test for valid training diagnostics
  • The Additive Cell-State Highway
  • Gradient Survival Through Forget Gates
  • GRU vs. LSTM: Gate Count and Parameter Cost
  • The Forget-Gate Bias and Memory Horizon
  • Controlled Comparison: Gated Cell vs. RNN on Lecture 11's Task
  • The Fixed-State Bottleneck: What Gating Cannot Do
  • The Central Result: Gating Makes Decay Rate a Per-Unit Choice

Syllabus

  1. Depth and the Gradient

    1. L1 · Why Depth?
    2. L2 · Backpropagation Is an Algorithm, Not a Formula
    3. L3 · The Gradient Through Depth

  2. Making Training Work

    1. L4 · Initialisation: Setting the Gains
    2. L5 · Normalisation: Keeping the Gains Set
    3. L6 · Residual Connections
    4. L7 · Optimisers at Depth

  3. Generalisation

    1. L8 · Generalisation, and Why the Old Theory Breaks

  4. Architectures

    1. L9 · Convolutional Networks I: The Operation and Its Bias
    2. L10 · Convolutional Networks II: Receptive Fields and Architectures
    3. L11 · Sequences I: Recurrence and Backprop Through Time
    4. L12 · Sequences II: Gating (LSTM and GRU)

  5. Practice and What Comes Next

    1. L13 · Training Craft
    2. L14 · Transfer Learning
    3. L15 · Attention, and Where This Course Ends