Transformers and Large Language Models

The architecture, the training objective, and the economics of inference

About this course

Transformers end to end: attention as a layer, what pretraining optimises, how scaling behaves, and what it costs to serve and adapt a model.

What you will be able to do

  • Describe self-attention and multi-head attention as operations on tensors
  • Account for a transformer's parameters and FLOPs end to end
  • Explain scaling laws and what they predict
  • Reason about inference cost — the KV cache, sampling and the context window
  • Compare fine-tuning, instruction tuning and preference optimisation

What this course covers

  • KV Cache Mechanism and Savings
  • The KV Cache Is Not an Ordinary Cache
  • KV Cache Size and the 6D Crossover
  • Arithmetic Intensity as a Diagnostic Instrument
  • Prefill vs. Decode: Two Machines in One
  • Batching: the Fix and Its Memory Bill
  • The Quadratic Attention Term and the 12D−1 Crossover
  • The loss has a true answer: entropy rate as floor
  • Comparing predictors in standard errors, not nats
  • Causal vs. masked objective: definitions and target counts
  • A count of targets is not a rate of learning
  • Value and cost of two-sided context
  • Perplexity is denominated in tokenizer units
  • Tokenization as frozen pre-processing
  • Character, word, and sub-word tokenization trade-offs
  • UTF-8 byte floor and its uneven script costs
  • Byte-pair encoding algorithm
  • Diminishing returns of additional merges
  • Fertility as a property of the tokenizer–text pair
  • Lecture synthesis: what is and is not established
  • Study design and vocabulary for fine-tuning
  • Adapter parameter cost and break-even rank
  • Whitening convention and its effect on SVD optimality
  • Rank as regularization: task spectrum determines adapter value
  • Catastrophic forgetting as direction, not distance
  • Memory bill: optimizer state, activations, and forward arithmetic
  • Adapter reach, matrix position, and regime comparison
  • Four decoding operations on one probability row
  • Renormalization cost and the two families of rule
  • Two KL divergence rulers and the temperature dial
  • Greedy decoding and inevitable periodicity
  • Truncation's compounding cost across sequences
  • Context window as a shared token budget
  • Model error shifts optimal temperature away from one
  • Capability vs. format compliance as separate axes
  • Exact match score as a product of two axes
  • What instruction-tuning demonstrations actually move
  • What the lecture declines to measure and why
  • Capability-absent control and the reversal of the ordering
  • Whose distribution? Demonstrator errors and asymmetric damage
  • Synthesis: the habit of separating axes when a score moves
  • Provenance convention for borrowed numbers
  • One prompt walked through the full stack
  • Layer ownership of observable behaviours
  • Thirteen refusals: measurement contradicting the tidy story
  • Controlled settings and the limits of transfer
  • Three questions for any claim about a language model
  • Self-attention: queries, keys, and values from one sequence
  • Row stochasticity vs. asymmetry: two kinds of truth
  • Permutation equivariance of unmasked self-attention
  • Causal mask: structure, arithmetic, and positional effect
  • What breaking equivariance does and does not establish
  • Convex-combination limit of attention output
  • Parameter count of one attention layer: 4D² with no N
  • Three honest parameter counts of one model
  • Full parameter formula for a transformer
  • Two FLOPs per parameter: the forward-pass identity
  • The approximate-equals sign: three separate jobs
  • Backward pass and the 6ND training-cost formula
  • Attention's parameter-free work and the 6D sequence threshold
  • The full training bill: 14 bytes per parameter
  • C ≈ 6ND as the course's central arithmetic equation
  • The Split and the Bill That Does Not Move
  • What Multi-Head Attention Actually Buys
  • One Head's Budget Is Conserved
  • Concatenation vs. the Output Projection (Wo)
  • Low-Rank Restriction on Scores vs. Softmax Lifting Weights
  • Head Ablation and Protocol Dependence
  • What Head Specialization Does and Does Not Mean
  • Preferences as data: Bradley-Terry and identifiability limits
  • KL-regularised objective and its closed-form optimum
  • The hat problem: why the closed form does not solve alignment
  • Reward hacking: proxy optimisation vs. true reward collapse
  • Direct Preference Optimisation (DPO) and the alignment tax
  • Three-ledger synthesis and the limits of alignment claims
  • Scaling law structure and vocabulary
  • Fitting a known law: noise sensitivity and shape mismatch
  • Range dependence of the exponent
  • The floor as the task's irreducible entropy
  • Compute-optimal allocation from exponent ratios
  • Synthesis: what an exponent is and is not
  • The mask's gap: direction not distance
  • Learned absolute positional embedding (lookup table)
  • Sinusoidal positional encoding and its dot-product behavior
  • Four-term score decomposition and position's share
  • Rotary position embedding (RoPE): exact offset-only property
  • Extrapolation horizon: the 1.285× result
  • Synthesis: three schemes, three failure modes
  • Two verbs: position mixing vs. feature mixing
  • Residual stream as an exact running sum
  • Pre- vs. post-layer norm: gradient scaling with depth
  • Step-size stability across depth and orderings
  • Parameter count: twelve D² per block
  • Depth vs. width trade-off at a fixed parameter budget
  • Closing corrections and carry-out habits

Syllabus

  1. Attention and the Block

    1. L1 · Tokenisation — The Model Never Sees Text
    2. L2 · Self-Attention as a Layer
    3. L3 · Multi-Head Attention
    4. L4 · Positional Encoding — Putting Order Back In
    5. L5 · The Transformer Block — Mix Across Positions, Then Across Features

  2. Pretraining and Scale

    1. L6 · Pretraining Objectives — What the Loss Makes It Predict
    2. L7 · The Model, End to End — Parameters and FLOPs
    3. L8 · Scaling Laws

  3. Inference

    1. L9 · Inference I — The KV Cache and the Memory Wall
    2. L10 · Inference II — Sampling and the Context Window

  4. Adaptation and Alignment

    1. L11 · Fine-Tuning and Parameter-Efficient Adaptation
    2. L12 · Instruction Tuning
    3. L13 · Preference Optimisation and Alignment

  5. Evaluation and Synthesis

    1. L15 · The Whole Stack, and Where This Course Ends